Academia and Clinic1 November 1997
    Author, Article, and Disclosure Information

    Abstract

    The final common pathway for most systematic reviews is a statistical summary of the data, or meta-analysis. The complex methods used in meta-analyses should always be complemented by clinical acumen and common sense in designing the protocol of a systematic review, deciding which data can be combined, and determining whether data should be combined. Both continuous and binary data can be pooled. Most meta-analyses summarize data from randomized trials, but other applications, such as the evaluation of diagnostic test performance and observational studies, have also been developed. The statistical methods of meta-analysis aim at evaluating the diversity (heterogeneity) among the results of different studies, exploring and explaining observed heterogeneity, and estimating a common pooled effect with increased precision. Fixed-effects models assume that an intervention has a single true effect, whereas random-effects models assume that an effect may vary across studies. Meta-regression analyses, by using each study rather than each patient as a unit of observation, can help to evaluate the effect of individual variables on the magnitude of an observed effect and thus may sometimes explain why study results differ. It is also important to assess the robustness of conclusions through sensitivity analyses and a formal evaluation of potential sources of bias, including publication bias and the effect of the quality of the studies on the observed effect.

    References

    • 1. Counsell C. Formulating questions and locating primary studies for inclusion in systematic reviews. Ann Intern Med. 1997; 127:380-7. Google Scholar
    • 2. Meade MO, Richardson WS. Selecting and appraising studies for a systematic review. Ann Intern Med. 1997; 127:531-7. Google Scholar
    • 3. McQuay HJ, Moore RA. Using numerical results from systematic reviews in clinical practice. Ann Intern Med. 1997; 126:712-20. Google Scholar
    • 4. Sinclair JC, Bracken MB. Clinically useful measures of effect in binary analyses of randomized trials. J Clin Epidemiol. 1994; 47:881-90. Google Scholar
    • 5. Cooper H, Hedges LV. The Handbook of Research Synthesis. New York: Russell Sage Foundation; 1994. Google Scholar
    • 6. Fleiss JL. Statistical Methods for Rates and Proportions. 2d ed. New York: J Wiley; 1981:161-5. Google Scholar
    • 7. Mantel N, Haenszel W. Statistical aspects of the analysis of data from retrospective studies of disease. J Natl Cancer Inst. 1959; 22:719-48. Google Scholar
    • 8. Laird NM, Mosteller F. Some statistical methods for combining experimental results. Int J Technol Assess Health Care. 1990; 6:5-30. Google Scholar
    • 9. Yusuf S, Peto R, Lewis J, Collins R, Sleight P. Beta blockade during and after myocardial infarction: an overview of the randomized trials. Prog Cardiovasc Dis. 1985; 27:335-71. Google Scholar
    • 10. Greenland S, Salvan A. Bias in the one-step method for pooling study results. Stat Med. 1990; 9:247-52. Google Scholar
    • 11. Fleiss JL. The statistical basis of meta-analysis. Stat Methods Med Res. 1993; 2:121-45. Google Scholar
    • 12. DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials. 1986; 7:177-88. Google Scholar
    • 13. Hedges LV, Olkin I. Statistical Methods for Meta-Analysis. Orlando: Academic Pr; 1985. Google Scholar
    • 14. Eddy DM, Hasselblad V, Schacter RD. Meta-Analysis by the Confidence Profile Method: The Statistical Synthesis of Evidence. New York: Academic Pr; 1991. Google Scholar
    • 15. Gelman A, Carlin JB, Stern HS, Rubin DB. Bayesian Data Analysis. London: Chapman & Hall; 1995:148-54. Google Scholar
    • 16. Lilford RJ, Thornton JG, Braunholtz D. Clinical trials and rare diseases: a way out of a conundrum. BMJ. 1995; 311:1621-5. Google Scholar
    • 17. Dumouchel W. Meta-analysis for dose-response models. Stat Med. 1995; 14:679-85. Google Scholar
    • 18. Berlin JA, Laird NM, Sacks HS, Chalmers TC. A comparison of statistical methods for combining event rates from clinical trials. Stat Med. 1989; 8:141-51. Google Scholar
    • 19. Borzak S, Ridker PM. Discordance between meta-analyses and large-scale randomized, controlled trials. Examples from the management of acute myocardial infarction. Ann Intern Med. 1995; 123:873-7. Google Scholar
    • 20. Lau J, Antman EM, Jimenez-Silva J, Kupelnick B, Mosteller F, Chalmers TC. Cumulative meta-analysis of therapeutic trials for myocardial infarction. N Engl J Med. 1992; 327:248-54. Google Scholar
    • 21. Lau J, Schmid CH, Chalmers TC. Cumulative meta-analysis of clinical trials builds evidence for exemplary medical care. J Clin Epidemiol. 1995; 48:45-57. Google Scholar
    • 22. Berlin JA, Antman EM. Advantages and limitations of metaanalytic regressions of clinical trials data. Online J Curr Clin Trials. 4 June 1994: Doc. No. 134. Google Scholar
    • 23. Morgenstern H. Uses of ecologic analysis in epidemiologic research. Am J Public Health. 1982; 72:1336-44. Google Scholar
    • 24. Langbein LI, Lichtman AJ. Ecological Inference. Beverly Hills, CA: Sage; 1978. (Sage University Paper Series on Quantitative Applications in the Social Sciences. Series no. 07-010.) Google Scholar
    • 25. Greenland S, Robins J. Invited commentary: ecologic studies-biases, misconceptions, and counterexamples. Am J Epidemiol. 1994; 139:747-60. Google Scholar
    • 26. McIntosh M. The population risk as an explanatory variable in research synthesis of clinical trials. Stat Med. 1996; 15:1713-28. Google Scholar
    • 27. Morris CN, Normand SL. Hierarchical models for combining information and for meta-analyses. In: Bernardo JM, Berger JO, Dawid AP, Smith AF. Bayesian Statistics 4. New York: Oxford Univ Pr; 1992. Google Scholar
    • 28. Smith TC, Spiegelhalter DJ, Thomas A. Bayesian approaches to random-effects meta-analysis: a comparative study. Stat Med. 1995; 14:2685-99. Google Scholar
    • 29. Oxman AD, Guyatt GH. A consumer's guide to subgroup analyses. Ann Intern Med. 1992; 116:78-84. Google Scholar
    • 30. Michels KB, Rosner BA. Data trawling: to fish or not to fish. Lancet. 1996; 348:1152-3. Google Scholar
    • 31. Dickersin K, Chan S, Chalmers TC, Sacks HS, Smith H Jr. Publication bias and clinical trials. Control Clin Trials. 1987; 8:343-53. Google Scholar
    • 32. Dickersin K. The existence of publication bias and risk factors for its occurrence. JAMA. 1990; 263:1385-9. Google Scholar
    • 33. Begg CB. Publication bias, In: Cooper H, Hedges L, eds. The Handbook of Research Synthesis. New York: Russell Sage Foundation; 1994. Google Scholar
    • 34. Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research. Lancet. 1991; 337:867-72. Google Scholar
    • 35. Light RJ, Pillemer DB. Summing up: the science of reviewing research. Cambridge, MA: Harvard Univ Pr; 1984. Google Scholar
    • 36. Begg CB, Mazumdar M. Operating characteristics of a rank correlation test for publication bias. Biometrics. 1994; 50:1088-101. Google Scholar
    • 37. Dear KB, Begg CB. An approach for assessing publication bias prior to performing a meta-analysis. Statistical Science. 1992; 7:237-45. Google Scholar
    • 38. Hedges LV. Modeling publication selection effects in random effects models in meta-analysis. Statistical Science. 1992; 7:246-55. Google Scholar
    • 39. Vevea JL, Hedges LV. A general linear model for estimating effect size in the presence of publication bias. Psychometrika. 1995; 60:419-35. Google Scholar
    • 40. Chalmers TC, Smith H Jr, Blackburn B, Silverman B, Schroeder B, Reitman D, et al. A method for assessing the quality of a randomized control trial. Control Clin Trials. 1981; 2:31-49. Google Scholar
    • 41. Mulrow CD, Linn WD, Gaul MK, Pugh JA. Assessing quality of a diagnostic test evaluation. J Gen Intern Med. 1989; 4:288-95. Google Scholar
    • 42. Detsky AS, Naylor CD, O'Rourke K, McGeer AJ, L'Abbe KA. Incorporating variations in the quality of individual randomized trials into meta-analysis. J Clin Epidemiol. 1992; 45:255-65. Google Scholar
    • 43. Moher D, Jadad AR, Nichol G, Penman M, Tugwell P, Walsh S. Assessing the quality of randomized, controlled trials: an annotated bibliography of scales and checklists. Control Clin Trials. 1995; 16:62-73. Google Scholar
    • 44. Emerson JD, Burdick E, Hoaglin DC, Mosteller F, Chalmers TC. An empirical study of the possible relation of treatment differences to quality scores in controlled randomized clinical trials. Control Clin Trials. 1990; 11:339-52. Google Scholar
    • 45. Greenland S. Invited commentary: a critical look at some popular meta-analytic methods. Am J Epidemiol. 1994; 140:290-6. Google Scholar
    • 46. Schultz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimension of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995; 273:408-12. Google Scholar
    • 47. Ioannidis JP, Cappelleri JC, Lau J, Skolnik PR, Melville B, Chalmers TC, et al. Early or deferred zidovudine therapy in HIV-infected patients without an AIDS-defining illness. Ann Intern Med. 1995; 122:856-66. Google Scholar
    • 48. Holme I. Relation of coronary heart disease incidence and total mortality to plasma cholesterol reduction in randomised trials: use of meta-analysis. Br Heart J. 1993; 69(1 Suppl):S42-7. Google Scholar
    • 49. Irwig L, Tosteson AN, Gatsonis C, Lau J, Colditz G, Chalmers TC, et al. Guidelines for meta-analyses evaluating diagnostic tests. Ann Intern Med. 1994; 120:667-76. Google Scholar
    • 50. Moses LE, Shapiro D, Littenberg B. Combining independent studies of a diagnostic test into a summary ROC curve: data-analytic approaches and some additional considerations. Stat Med. 1993; 12:1293-316. Google Scholar
    • 51. Tweedie RL, Mengersen KL. Meta-analytic approaches to dose-response relationships, with application in studies of lung cancer and exposure to environmental tobacco smoke. Stat Med. 1995; 14:545-69. Google Scholar
    • 52. Greenland S, Longnecker MP. Methods for trend estimation from summarized dose-response data, with applications to meta-analysis. Am J Epidemiol. 1992; 135:1301-9. Google Scholar
    • 53. Smith SJ, Caudill SP, Steinberg KK, Thacker SB. On combining dose-response data from epidemiological studies by meta-analysis. Stat Med. 1995; 14:531-44. Google Scholar
    • 54. Stewart LA, Clarke MJ. Practical methodology of meta-analyses (overviews) using updated individual patient data. Cochrane Working Group. Stat Med. 1995; 14:2057-79. Google Scholar
    • 55. Olkin I. Statistical and theoretical considerations in meta-analysis. J Clin Epidemiol. 1995; 48:133-46. Google Scholar
    • 56. Cook DJ, Witt LG, Cook RJ, Guyatt GH. Stress ulcer prophylaxis in the critically ill: a meta-analysis. Am J Med. 1991; 91:519-27. Google Scholar
    • 57. Tryba M. Prophylaxis of stress ulcer bleeding. A meta-analysis. J Clin Gastroenterol. 1991:13(Suppl 2):544-55. Google Scholar
    • 58. Villar J, Carroli G, Belizan JM. Predictive ability of meta-analyses of randomised controlled trials. Lancet. 1995; 345:772-6. Google Scholar
    • 59. Cappelleri JC, Ioannidis JP, Schmid CH, de Ferranti SD, Aubert M, Chalmers TC, et al. Large trials vs meta-analysis of smaller trials: how do their results compare? JAMA. 1996; 276:1332-8. Google Scholar
    • 60. Cook DJ, Reeve BK, Guyatt GH, Heyland DK, Griffith LE, Buckingham L, et al. Stress ulcer prophylaxis in critically ill patients. Resolving discordant meta-analyses. JAMA. 1996; 275:308-14. Google Scholar
    • 61. Ioannidis JP, Lau J. On meta-analyses of meta-analyses [Letter]. Lancet. 1996; 348:756. Google Scholar
    • 62. Altman DG. Better reporting of randomised controlled trials: the CONSORT statement. BMJ. 1996; 313:570-1. Google Scholar
    • 63. Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, et al. Improving the quality of reporting of randomized, controlled trials. The CONSORT statement. JAMA. 1996; 276:637-9. Google Scholar