IMPORTANT: This site is undergoing maintenance on May 28, 2024 from 2:00 AM EST to 4:00 PM EST. Individual users will not be able to log into their accounts to comment on articles or take CME. Institutional administrators will not be able to make changes. Contact [email protected] with questions.
IMPORTANT: This site is undergoing maintenance on May 28, 2024 from 2:00 AM EST to 4:00 PM EST. Individual users will not be able to log into their accounts to comment on articles or take CME. Institutional administrators will not be able to make changes. Contact [email protected] with questions.
Editorials
1 January 1998

Masking, Blinding, and Peer Review: The Blind Leading the Blinded

Publication: Annals of Internal Medicine
Volume 128, Number 1
For some 300 years, peer review, with all its strengths (there are many) and its flaws (there are also many), has been a key element in the vexing process of getting to the truth. Despite its importance, the peer review process itself has until recently not been subject to the same intense scrutiny as the content of scholarly work [1]. It is a healthy sign, therefore, that peer review is now getting the serious attention it deserves, as was abundantly apparent at the International Congress on Biomedical Peer Review and Global Communications held in Prague in September 1997.
A prominent feature of this attention, and a major focus of the Congress, was the issue of anonymity. Stated briefly, the question is whether peer review is less biased when the authors' identity is withheld from reviewers (masking) and when the reviewers' identity is not revealed to authors (blinding). Most medical journals, including Annals, have on the one hand traditionally behaved as though anonymity did not matter, because we have not masked the names and institutional affiliations of authors when sending manuscripts to peer reviewers. These same journals, including Annals, have on the other hand behaved as though anonymity mattered a great deal, because we have blinded authors to the reviewers' identity. (Annals does accept signed reviews, but few reviewers choose to sign them.) As editors we have, at the very least, been inconsistent, largely because we have accepted the efficacy of the traditional peer review process on blind faith rather than on the basis of evidence-a clear case of the blind leading the blinded.
That said, the a priori case for masking and blinding is strong. Maintaining anonymity occupies an honorable position in the canon of science; blinding is, after all, a central tenet of the methodology of clinical trials, and for good reason. Bias is the great demon of science, and the opportunity for bias is clearly reduced when experimental observations are not confounded by personal but irrelevant associations or preconceptions. Blinding has clearly proven important in making nonmedical judgments; for example, having musicians perform behind a screen is widely credited with helping to reduce gender bias in the selection of candidates for symphony positions. And if the question were still in doubt, we only need think about the effectiveness of celebrity endorsements in marketing, recognizing that the entire purpose of marketing is to create purchasing bias.
Why, then, the disconnect between principle and practice in the masking of author identity? The answer appears to be partly historical (journals have never done it), partly logistic (masking is time consuming, expensive, and technically difficult), and partly intellectual (it has remained an open question whether, in practice, masking actually makes a difference). Several previous studies had suggested that masking authors' identity improved the quality of the peer reviews done for journals in both medicine [2, 3] and economics [4]. The preliminary results of two recent large trials, one involving multiple journals, were presented in Prague; in contrast to the earlier results, neither found any detectable benefit of masking [5, 6]. Interest in these new findings has been intense, and the response seems to have been a substantial willingness to accept the null hypothesis-that is, that masking doesn't matter [7]. Lost in the excitement, unfortunately, has been the important observation that for a large proportion of manuscripts in these trials, masking was unsuccessful; that is, reviewers correctly guessed the identity of authors [2, 3, 8]. The hypothesis that masking does not improve the objectivity and quality of reviews has, therefore, simply not been adequately tested. As pointed out by the researchers in Prague, masking author identity is likely to be important for “influential” authors, precisely those whose identities are therefore difficult, if not impossible, to hide. In light of the available evidence-three published studies in which masking author identity improved review quality and several unpublished studies (hence, not peer reviewed!) in which masking had no effect or the results were inconclusive-a rush to judgment about the nonutility of masking seems premature.
As for blinding authors to the identity of reviewers, the principal argument in favor of the practice is that the signing of reviews would inhibit reviewers from being open and probing in their critiques (as has increasingly happened with letters of personal recommendation); this would clearly not be in the best interests of good science. The principal argument against blinding is that it might foster irresponsibility, particularly slanted and destructive criticism, because reviewers know that authors cannot hold them personally accountable for their opinions. The case for “opening up” peer review by identifying reviewers to authors is therefore being vigorously put forward ([7]; Rennie D. Reporting, peer review, and authorship: the future. Presentation at the International Congress on Biomedical Peer Review and Global Communications. Prague, Czech Republic, 17 to 21 September 1997).
The case for openness is bolstered by the anecdotal experience of at least one journal that found that “the sky did not fall” (that is, reviews were not noticeably different) when reviewers were asked to sign their critiques. Documentation is lacking, however, on whether the quality of reviews improved or declined as a result of unblinding. Moreover, at least one large, detailed, quantitative study has shown that reviewers who remain anonymous are at least as likely to be overly, and perhaps inappropriately, enthusiastic about manuscripts (“zealots”) as they are to be overly harsh (“assassins”) [9]. The results of two experimental trials presented in Prague also showed that although unblinding did not produce lower-quality reviews, it also did not noticeably improve them [5, 10]. More important, the deficiencies observed appeared to be largely due to a lack of critical assessment skills rather than to biased, hypercritical, or otherwise inappropriate opinions from reviewers [10]. The argument for openness thus rests less on evidence about its effect on review quality, than on concern for fairness, the felt need to “level the playing field” for authors and reviewers.
Fairness and accountability are potent considerations, to be sure. But blinding authors to reviewers' identity does not, of course, make reviewers anonymous or unaccountable: Editors know very well who they are. Relying on multiple rather than single reviewers and using reviewers' comments as advisory rather than binding opinions are further important safeguards for authors. Most Annals “blinded” reviewers are more than responsible: They spend untold hours laboring over difficult and confusing texts; they help to sort things out; they probe, identify flaws, clarify strengths. The rare reviewer who slants critiques inappropriately or oversteps the bounds of civility is simply not invited to review again. The price we pay for blinding therefore appears to be small.
If a major goal of peer review is objectivity, we should be doing everything we can to remove the personal element from the process of making intellectual judgments about biomedical papers, whether that element is the identity of authors or of reviewers. From this perspective, we think that blinding authors to reviewers' identity makes sense and intend to continue the practice at Annals. We think it would also be appropriate to mask authors' identity during peer review (double-blinding) and will remain open to that approach; however, we are reluctant to implement it until we have greater assurance that masking can be done effectively and that it actually helps to make reviews more objective.
Overall, the present condition of peer review might best be described in the words of the country and western song “I Ain't Broke, but I'm Badly Bent.” But because the evidence suggests that the major problem with peer review is lack of critical assessment skills, the best way to fix it is to get serious about training reviewers in the science (and art) of high-quality review rather than to unblind reviewers, which is only likely to bend the review process further out of shape. A formal curriculum in the fundamentals of peer reviewing, a national (or, better, international) credentialing process for peer reviewers (should every academic institution have a certified in-house expert in reviewing to serve as a resource person?), a set of broadly agreed-on criteria for peer review quality, and the development of more effective mechanisms for applying those criteria all seem to be worth serious consideration. Moreover, because a small proportion of blinded reviewers tends to be consistently either overly enthusiastic or overly critical, we also need to begin tracking reviewer performance in ways that make it possible to adjust for such generic biases [9].
The collective wisdom inherent in the peer-reviewed literature has made peer review the “gold standard” for scholarly work, and peer review can only get more important as we increasingly face the task of separating signal from noise in the tidal wave of electronic information. Improving the quality of peer review is more important than ever, and the recent peer review congress was a big step forward in this effort. The small band of editors and researchers who came together in Prague can't do the job alone, however; university faculties, professional societies, and funding agencies all need to get into the act. (The workshop on peer review being held in Berkeley, California, on 20 February 1998, sponsored by the Council of Biology Editors, is right on target.) Opening our eyes to the real limitations of peer review will be a dirty job, but somebody's got to do it. Researchers and authors, reviewers and editors, our readers, and our patients deserve no less.
Frank Davidoff, MD
Editor

References

1.
Lock S. A Difficult Balance. Editorial Peer Review in Medicine. Philadelphia: ISI Pr; 1985.
2.
McNutt RA, Evans AT, Fletcher RH, Fletcher SW. The effects of blinding on the quality of peer review. A randomized trial. JAMA. 1990; 263:1371-6.
3.
Fisher M, Friedman SB, Strauss B. The effects of blinding on acceptance of research papers by peer review. JAMA. 1994; 272:143-6.
4.
Laband DN, Piette MJ. A citation analysis of the impact of blinded peer review. JAMA. 1994; 272:147-9.
5.
van Rooyen S, Godlee F, Evans S, Smith R, Black N. The effect of blinding and unmasking on the quality of peer review: A randomized, controlled trial [Abstract]. Proceedings of the International Congress on Biomedical Peer Review and Global Communications. Prague, Czech Republic, 17 to 21 September 1997:17.
6.
Justice AC, Cho MK, Winker MA, Berlin JA, Rennie D, Berkwits M, et al. Masking author identity in peer review: does it improve peer-review quality? [Abstract] Proceedings of the International Congress on Biomedical Peer Review and Global Communications. Prague, Czech Republic, 17 to 21 September 1997:18.
7.
Smith R. Peer review: reform or revolution? Time to open up the black box of peer review [Editorial]. BMJ. 1997; 315:759-60.
8.
Cho A, Justice AC, Winker MA, Berlin JA, Waeckerle J, Rennie D, et al. Masking author identity in peer review: what factors influence masking success? [Abstract] Proceedings of the International Congress on Biomedical Peer Review and Global Communications. Prague, Czech Republic, 17 to 21 September 1997:18.
9.
Siegelman SS. Assassins and zealots: variations in peer review. Radiology. 1991; 178:637-42.
10.
Godlee F, Gale DR, Martyn CN. The effect on the quality of peer review of blinding peer reviewers and asking them to sign their reports: a randomized, controlled trial [Abstract]. Proceedings of the International Congress on Biomedical Peer Review and Global Communications. Prague, Czech Republic, 17 to 21 September 1997:17.

Comments

0 Comments
Sign In to Submit A Comment

Information & Authors

Information

Published In

cover image Annals of Internal Medicine
Annals of Internal Medicine
Volume 128Number 11 January 1998
Pages: 66 - 68

History

Published in issue: 1 January 1998
Published online: 15 August 2000

Keywords

Authors

Affiliations

Frank Davidoff, MD, Editor

Metrics & Citations

Metrics

Citations

If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. For an editable text file, please select Medlars format which will download as a .txt file. Simply select your manager software from the list below and click Download.

For more information or tips please see 'Downloading to a citation manager' in the Help menu.

Format





Download article citation data for:
Frank Davidoff. Masking, Blinding, and Peer Review: The Blind Leading the Blinded. Ann Intern Med.1998;128:66-68. doi:10.7326/0003-4819-128-1-199801010-00011

View More

View options

PDF/ePub

View PDF/ePub

Get Access

Login Options:
Purchase

You will be redirected to acponline.org to sign-in to Annals to complete your purchase.

Create your Free Account

You will be redirected to acponline.org to create an account that will provide access to Annals.

Media

Figures

Other

Tables

Share

Share

Copy the content Link

Share on social media