This paper is only available as a PDF. To read, Please Download here.
Abstract
Peer review is the ordinary method for judging the acceptability of scientific papers
for publication. Yet there are few prospective data defining the accuracy or reproducibility
of this method. This study was aimed at evaluating interrater consistency in reviewing
a single manuscript and at determining whether a referee's likely predisposition (inferred
from his or her own published papers in the field) influences his or her attitude
toward this manuscript. A computerized search identified 33 first authors of research
papers about transcutaneous electrical nerve stimulation. They were asked to review
part of a fictitious scientific paper on the topic; they were not made aware of being
tested. Reviewers' ratings were recorded by multiple-choice answers in a structured
questionnaire. The analyses of the answers revealed poor interrater reliability. Furthermore,
referees were clearly influenced by their own preconceptions and judged according
to their own published experience in that particular medical subject. That is, referees
who would be expected to agree with the paper's findings tended to judge it less harshly
than did referees who would be expected to disagree. These results suggest significant
impact of reviewer bias on referees' judgment and imply that the peer review system
in its present form has room for improvements in fairness and consistency.
To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to Translational ResearchAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
References
- Lancet. 1991; 1: 462-463
- Interreferee agreement on some characteristics of manuscripts submitted to the Journal of Personality and Social Psychology.J Pers Soc Psychol. 1974; 29: 698-702
- Confirmation response bias among social work journals.Sci Technol Hum Values. 1990; 15: 9-38
- Methodological errors in medical research.Blackwell, Oxford1990
- Reference bias in reports of drug trials.Br Med J. 1987; 295: 654-656
- Publication prejudices, an experimental study of confirmatory bias in the peer review system.Cogn Ther Res. 1977; 1: 161-175
- Publication bias in clinical research.Lancet. 1991; 337: 867-872
- Chauvinism.Nature. 1991; 352: 560
- Interjudgmental reliability of reviews of the Journal of Educational Psychology.J Educ Psychol. 1981; 73: 872-880
- The reliability of manuscript reviewing for the Journal of Abnormal Psychology.J Abnorm Psychol. 1979; 22: 596-600
- A statistical analysis of reviewer agreement and bias in evaluating medical abstracts.Yale J Biol Med. 1976; 49: 373-383
- Evaluation of the Personality and Social Psychology Bulletin by its readers and authors.Pers Soc Psychol Bull. 1977; 3: 583-591
- The reliability of reviews for the American Psychologist.Am Psychol. 1978; 33: 935
- Scientific publication and knowledge politics.J Soc Behav Pers. 1987; 2: 165-176
- Publication bias.Lancet. 1991; 337: 1419
- Studying journal editors, the worst heresy.IRB. 1989; 11: 7-8
- Pride and prejudice in peer review.J Clin Epidemiol. 1991; 44: 343-345
- The ethics of human experimentation.in: Human experimentation: a guided step into the unknown. Oxford University Press, New York1985: 153-172
- Construction, consent, and condemnation in research on peer review.J Clin Epidemiol. 1991; 44: 339-341
Article info
Publication history
Accepted:
February 17,
1994
Received in revised form:
February 2,
1994
Received:
October 29,
1993
Footnotes
☆An editorial relevant to this article appears on p. 146 of this issue of the Journal.
Identification
Copyright
© 1994 Published by Elsevier Inc.
