This paper is only available as a PDF. To read, Please Download here.
Peer review is the ordinary method for judging the acceptability of scientific papers for publication. Yet there are few prospective data defining the accuracy or reproducibility of this method. This study was aimed at evaluating interrater consistency in reviewing a single manuscript and at determining whether a referee's likely predisposition (inferred from his or her own published papers in the field) influences his or her attitude toward this manuscript. A computerized search identified 33 first authors of research papers about transcutaneous electrical nerve stimulation. They were asked to review part of a fictitious scientific paper on the topic; they were not made aware of being tested. Reviewers' ratings were recorded by multiple-choice answers in a structured questionnaire. The analyses of the answers revealed poor interrater reliability. Furthermore, referees were clearly influenced by their own preconceptions and judged according to their own published experience in that particular medical subject. That is, referees who would be expected to agree with the paper's findings tended to judge it less harshly than did referees who would be expected to disagree. These results suggest significant impact of reviewer bias on referees' judgment and imply that the peer review system in its present form has room for improvements in fairness and consistency.
To read this article in full you will need to make a payment
Purchase one-time access:Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
One-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:Subscribe to Translational Research
Already a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
- Lancet. 1991; 1: 462-463
- Interreferee agreement on some characteristics of manuscripts submitted to the Journal of Personality and Social Psychology.J Pers Soc Psychol. 1974; 29: 698-702
- Confirmation response bias among social work journals.Sci Technol Hum Values. 1990; 15: 9-38
- Methodological errors in medical research.Blackwell, Oxford1990
- Reference bias in reports of drug trials.Br Med J. 1987; 295: 654-656
- Publication prejudices, an experimental study of confirmatory bias in the peer review system.Cogn Ther Res. 1977; 1: 161-175
- Publication bias in clinical research.Lancet. 1991; 337: 867-872
- Chauvinism.Nature. 1991; 352: 560
- Interjudgmental reliability of reviews of the Journal of Educational Psychology.J Educ Psychol. 1981; 73: 872-880
- The reliability of manuscript reviewing for the Journal of Abnormal Psychology.J Abnorm Psychol. 1979; 22: 596-600
- A statistical analysis of reviewer agreement and bias in evaluating medical abstracts.Yale J Biol Med. 1976; 49: 373-383
- Evaluation of the Personality and Social Psychology Bulletin by its readers and authors.Pers Soc Psychol Bull. 1977; 3: 583-591
- The reliability of reviews for the American Psychologist.Am Psychol. 1978; 33: 935
- Scientific publication and knowledge politics.J Soc Behav Pers. 1987; 2: 165-176
- Publication bias.Lancet. 1991; 337: 1419
- Studying journal editors, the worst heresy.IRB. 1989; 11: 7-8
- Pride and prejudice in peer review.J Clin Epidemiol. 1991; 44: 343-345
- The ethics of human experimentation.in: Human experimentation: a guided step into the unknown. Oxford University Press, New York1985: 153-172
- Construction, consent, and condemnation in research on peer review.J Clin Epidemiol. 1991; 44: 339-341
Accepted: February 17, 1994
Received in revised form: February 2, 1994
Received: October 29, 1993
☆An editorial relevant to this article appears on p. 146 of this issue of the Journal.
© 1994 Published by Elsevier Inc.