Original article| Volume 124, ISSUE 2, P178-182, August 1994

Download started.


Reviewer bias: A blinded experimental study

      This paper is only available as a PDF. To read, Please Download here.


      Peer review is the ordinary method for judging the acceptability of scientific papers for publication. Yet there are few prospective data defining the accuracy or reproducibility of this method. This study was aimed at evaluating interrater consistency in reviewing a single manuscript and at determining whether a referee's likely predisposition (inferred from his or her own published papers in the field) influences his or her attitude toward this manuscript. A computerized search identified 33 first authors of research papers about transcutaneous electrical nerve stimulation. They were asked to review part of a fictitious scientific paper on the topic; they were not made aware of being tested. Reviewers' ratings were recorded by multiple-choice answers in a structured questionnaire. The analyses of the answers revealed poor interrater reliability. Furthermore, referees were clearly influenced by their own preconceptions and judged according to their own published experience in that particular medical subject. That is, referees who would be expected to agree with the paper's findings tended to judge it less harshly than did referees who would be expected to disagree. These results suggest significant impact of reviewer bias on referees' judgment and imply that the peer review system in its present form has room for improvements in fairness and consistency.
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Translational Research
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


      1. Lancet. 1991; 1: 462-463
        • Scott WA
        Interreferee agreement on some characteristics of manuscripts submitted to the Journal of Personality and Social Psychology.
        J Pers Soc Psychol. 1974; 29: 698-702
        • Epstein WM
        Confirmation response bias among social work journals.
        Sci Technol Hum Values. 1990; 15: 9-38
        • Andersen B
        Methodological errors in medical research.
        Blackwell, Oxford1990
        • Gotzsche P
        Reference bias in reports of drug trials.
        Br Med J. 1987; 295: 654-656
        • Mahoney M
        Publication prejudices, an experimental study of confirmatory bias in the peer review system.
        Cogn Ther Res. 1977; 1: 161-175
        • Easterbrook JP
        • Berlin JA
        • Gopalan R
        • Matthews DR
        Publication bias in clinical research.
        Lancet. 1991; 337: 867-872
        • Ernst E
        • Kienbacher T
        Nature. 1991; 352: 560
        • Marsh HW
        • Ball S
        Interjudgmental reliability of reviews of the Journal of Educational Psychology.
        J Educ Psychol. 1981; 73: 872-880
        • Cicchetti DV
        • Eron LD
        The reliability of manuscript reviewing for the Journal of Abnormal Psychology.
        J Abnorm Psychol. 1979; 22: 596-600
        • Cicchetti DV
        • Conn H
        A statistical analysis of reviewer agreement and bias in evaluating medical abstracts.
        Yale J Biol Med. 1976; 49: 373-383
        • Lindner DE
        Evaluation of the Personality and Social Psychology Bulletin by its readers and authors.
        Pers Soc Psychol Bull. 1977; 3: 583-591
        • Scarr S
        • Weber BLR
        The reliability of reviews for the American Psychologist.
        Am Psychol. 1978; 33: 935
        • Mahoney M
        Scientific publication and knowledge politics.
        J Soc Behav Pers. 1987; 2: 165-176
        • Wilmhurst P
        Publication bias.
        Lancet. 1991; 337: 1419
        • Epstein WM
        Studying journal editors, the worst heresy.
        IRB. 1989; 11: 7-8
        • Kemper KJ
        Pride and prejudice in peer review.
        J Clin Epidemiol. 1991; 44: 343-345
        • Silverman WA
        The ethics of human experimentation.
        in: Human experimentation: a guided step into the unknown. Oxford University Press, New York1985: 153-172
        • Feinstein AR
        Construction, consent, and condemnation in research on peer review.
        J Clin Epidemiol. 1991; 44: 339-341