Objective:
To evaluate the effects of peer review and editing on manuscript quality.
Setting:
Editorial offices of Annals of Internal Medicine.
Design:
Masked before–after study.
Manuscripts:
111 consecutive original research manuscripts accepted for publication at Annals between March 1992 and March 1993.
Measurements:
We used a manuscript quality assessment tool of 34 items to evaluate the quality of the research report, not the quality of the research itself. Each item was scored on a 1 to 5 scale. Forty-four expert assessors unaware of the design or aims of the study evaluated the manuscripts, with different persons evaluating the two versions of each manuscript (before and after the editorial process).
Results:
33 of the 34 items changed in the direction of improvement, with the largest improvements seen in the discussion of study limitations, generalizations, use of confidence intervals, and the tone of conclusions. Overall, the percentage of items scored three or more increased by an absolute 7.3% (95% CI, 3.3% to 11.3%) from a baseline of 75%. The average item score improved by 0.23 points (CI, 0.07 to 0.39) from a baseline mean of 3.5. Manuscripts rated in the bottom 50% showed two- to threefold larger improvements than those in the top 50%, after correction for regression to the mean.
Conclusions:
Peer review and editing improve the quality of medical research reporting, particularly in those areas that readers rely on most heavily to decide on the importance and generalizability of the findings.
References
- 1. Relman AS, Angell M. How good is peer review? (Editorial). N Engl J Med. 1989; 321:827-9. Google Scholar
- 2. Relman AS. Peer review in scientific journals—what good is it? West J Med. 1990; 153:520-2. Google Scholar
- 3. Peters D, Ceci S. Peer-review practices of psychological journals: The fate of submitted articles, submitted again. Behavioral and Brain Sciences. 1982; 5:187-255. Google Scholar
- 4. Bower B. Peer review under fire. Science News. 1991; (Jun 22):394-5. Google Scholar
- 5. Altman L. The myth of passing peer review. In: Bailar J, Angell M, Boots S, eds. Ethics and Policy in Scientific Publication. Bethesda, MD: Council of Biology Editors, Inc.; 1990. Google Scholar
- 6. Medical Journals: slowing flow of news on life saving discoveries? Washington Post. 1991; (Jan 25):A25. Google Scholar
- 7. Horrobin DF. The philosophical basis of peer review and the suppression of innovation. JAMA. 1990; 263:1438-41. Google Scholar
- 8. Bailar JC 3d, Patterson K. The need for a research agenda. N Engl J Med. 1985; 312:654-7. Google Scholar
- 9. Rennie D. Guarding the guardians: a conference on editorial peer review (Editorial). JAMA. 1986; 256:2391-2. Google Scholar
- 10. Rennie D. Editorial peer review in biomedical publication. The first international congress. (Editorial). JAMA. 1990; 263:1317. Google Scholar
- 11. Gardner MJ, Machin D, Campbell MJ. Use of check lists in assessing the statistical content of medical studies. Br Med J. 1986; 292:810-2. Google Scholar
- 12. Chalmers TC, Smith HJ Jr, Blackburn B, Silverman B, Schroeder B, Reitman D, et al. A method for assessing the quality of a randomized control trial. Controlled Clin Trials. 1981; 2:31-49. Google Scholar
- 13. Meinert CL, Tonascia S, Higgins K. Content of reports on clinical trials: A critical review. Controlled Clin Trials. 1984; 5:328-47. Google Scholar
- 14. Meinert CL, Tonascia S. Clinical Trials: Design, Conduct and Analysis. New York: Oxford University Press; 1986:1-469. Google Scholar
- 15. Mosteller F, Gilbert J, McPeek B. Reporting standards and research strategies for controlled clinical trials; Agenda for the Editor. Controlled Clin Trials. 1980; 1:37-58. Google Scholar
- 16. Simon R, Wittes R. Methodologic guidelines for reports of clinical trials (Editorial). Cancer Trt Reps. 1985; 69:1-3. Google Scholar
- 17. Bailar JC 3d, Mosteller F. Guidelines for statistical reporting in articles for medical journals. Amplifications and explanations. Ann Intern Med. 1988; 108:266-73. Google Scholar
- 18. Pocock SJ, Hughes MD, Lee RJ. Statistical problems in the reporting of clinical trials. A survey of three medical journals. N Engl J Med. 1987; 317:426-32. Google Scholar
- 19. Gardner MJ, Altman DG. Confidence intervals rather than P values: estimation rather than hypothesis testing. Br Med J. 1986; 292:746-50. Google Scholar
- 20. Berry G. Statistical significance and confidence intervals (Editorial). Med J Aust. 1986; 144:618-9. Google Scholar
- 21. Braitman LE. Confidence intervals extract clinically useful information from data (Editorial). Ann Intern Med. 1988; 108:296-8. Google Scholar
- 22. Braitman LE. Confidence intervals assess both clinical significance and statistical significance (Editorial). Ann Intern Med. 1991; 114:515-7. Google Scholar
- 23. Concato J, Feinstein AR, Holford TR. The risk of determining risk with multivariable models. Ann Intern Med. 1993; 118:201-10. Google Scholar
- 24. Soffer A. Can you believe what you read in medical journals? Chest. 1992; 101:1417-9. Google Scholar
- 25. Cicchetti D. The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation. Behavioral and Brain Sciences. 1991; 14 [1]:119-35. Google Scholar
- 26. Bailar J, Angell M, Boots S, et al. Ethics and Policy in Scientific Publication. Bethesda, Maryland: Council of Biology Editors, Inc.; 1990. Google Scholar
- 27. Chubin DE, Hackett EJ. Peerless Science: Peer Review and US Science Policy. Albany: State University of New York; 1990. Google Scholar
- 28. Feinberg BA. Peer review and the NCI's clinical alert on node-negative breast cancer (Letter). JAMA. 1989; 261:695-6. Google Scholar
- 29. DeVita VT. Is a mechanism such as the NCI's Clinical Alert ever an appropriate alternative to peer review? In: DeVita VT, Hellman S, Rosenberg SA, eds. Important Advances in Oncology. Philadelphia: Lippincott; 1991:241-54. Google Scholar
- 30. Lock S. A difficult balance: editorial peer review in medicine. Philadelphia: ISI Press; 1986:1-172. Google Scholar
- 31. Gardner MJ, Bond J. An exploratory study of statistical assessment of papers published in the British Medical Journal. JAMA. 1990; 263:1355-8. Google Scholar
- 32. Wilson J. Peer review and publication: Presidential address before the 70th annual meeting of the American Society for Clinical Investigation. San Francisco, California, 30 April 1978. J Clin Invest. 1978; 61:1697-701. Google Scholar
- 33. Ernst E, Saradeth T, Resch KL. Drawbacks of peer review (Letter). Nature. 1993; 363:296. Google Scholar
- 34. Fleiss J. Statistical Methods for Rates and Proportions. Second edition. New York: Wiley; 1981. Google Scholar
- 35. Detsky AS, Naylor CD, O'Rourke K, McGeer AJ, L'Abbe KA. Incorporating variations in the quality of individual randomized trials into meta-analysis. J Clin Epidemiol. 1992; 45 [3]:255-65. Google Scholar
- 36. Greenland S. A critical look at some popular meta-analytic methods. Am J Epidemiol. 1994; (In press). Google Scholar
Author, Article, and Disclosure Information
From Johns Hopkins University School of Medicine, Baltimore, Maryland; the University of Pennsylvania School of Medicine, Philadelphia, Pennsylvania; Harvard Medical School, Boston, Massachusetts.
Corresponding Author: Steven Goodman, MD, PhD, Johns Hopkins University, Oncology Center, Division of Biostatistics, 550 North Broadway, Suite 1103, Baltimore, MD 21205.
Acknowledgments: The following “expert assessors” are thanked for their contributions to this project: Wendy Adams, MD; Eric Bass, MD; Greta Bunin, PhD; Jeffrey Carson, MD; Neal Dawson, MD; John Dickenson, MD; Kay Dickersin, PhD; Jose Escarce, MD; Harold Feldman, MD; Daniel Ford, MD, MPH; Martha Gerrity, MD; Matthew Gillman, MD, PhD; Deborah Grady, MD; Alan Hillman, MD; Thomas Imperiale, MD; Sharon Inouye, MD; Jeane Grisso, MD; Walter Kernan, MD; Thomas Keyserling, MD; Michael Klag, MD; Kurt Kroenke, MD; Seth Landefeld, MD; Suzanne Landis, MD; Valerie Lawrence, MD; Richard Lofgren, MD; Peter Margolis, MD, PhD; Wayne McCormick, MD; Alfredo Morabia, MD, PhD; Ann Naftinger, MD; Roberta Ness, MD; Warren Newton, MD; Sandra Norman, PhD; Patrick O'Connor, MD; Michelle Petri, MD, MPH; John Philbrick, MD; Jennifer Pinto-Martin, PhD; Peter Pompei, MD; Bruce Psaty, MD; Jeffrey Silber, MD, PhD; Rebecca Silliman, MD; John Steiner, MD; Ramon Velez, MD; Paul Whelton, MD, MSc; Larry Wissow, MD, MPH. The authors also thank Ms. Torii Ransome and Ms. Grace Lobb for their efforts in coordinating this project and Mr. Andrew Langman for assisting with the design of the questionnaire.
Submit a Comment
Contributors must reveal any conflict of interest. Comments are moderated. Please see our information for authorsregarding comments on an Annals publication.
*All comments submitted after October 1, 2021 and selected for publication will be published online only.