Published evidence suggests that aspects of trial design lead to biased intervention effect estimates, but findings from different studies are inconsistent. This study combined data from 7 meta-epidemiologic studies and removed overlaps to derive a final data set of 234 unique meta-analyses containing 1973 trials. Outcome measures were classified as “mortality,” “other objective,” “or subjective,” and Bayesian hierarchical models were used to estimate associations of trial characteristics with average bias and between-trial heterogeneity. Intervention effect estimates seemed to be exaggerated in trials with inadequate or unclear (vs. adequate) random-sequence generation (ratio of odds ratios, 0.89 [95% credible interval {CrI}, 0.82 to 0.96]) and with inadequate or unclear (vs. adequate) allocation concealment (ratio of odds ratios, 0.93 [CrI, 0.87 to 0.99]). Lack of or unclear double-blinding (vs. double-blinding) was associated with an average of 13% exaggeration of intervention effects (ratio of odds ratios, 0.87 [CrI, 0.79 to 0.96]), and between-trial heterogeneity was increased for such studies (SD increase in heterogeneity, 0.14 [CrI, 0.02 to 0.30]). For each characteristic, average bias and increases in between-trial heterogeneity were driven primarily by trials with subjective outcomes, with little evidence of bias in trials with objective and mortality outcomes. This study is limited by incomplete trial reporting, and findings may be confounded by other study design characteristics. Bias associated with study design characteristics may lead to exaggeration of intervention effect estimates and increases in between-trial heterogeneity in trials reporting subjectively assessed outcomes.
References
- 1.
Gluud LL . Bias in clinical intervention research. Am J Epidemiol. 2006;163:493-501. [PMID:16443796 ] CrossrefMedlineGoogle Scholar - 2.
Egger M ,Juni P ,Bartlett C ,Holenstein F ,Sterne J . How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technol Assess. 2003;7:1-76. [PMID:12583822 ] CrossrefMedlineGoogle Scholar - 3.
Moher D ,Pham B ,Jones A ,Cook DJ ,Jadad AR ,Moher M ,et al . Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet. 1998;352:609-13. [PMID:9746022 ] CrossrefMedlineGoogle Scholar - 4.
Schulz KF ,Chalmers I ,Hayes RJ ,Altman DG . Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995;273:408-12. [PMID:7823387 ] CrossrefMedlineGoogle Scholar - 5. Als-Nielsen B, Chen W, Gluud LL, Siersma V, Hilden J, Gluud C. Are trial size and reported methodological quality associated with treatment effects? Observational study of 523 randomised trials. Presented at 12th Cochrane Colloquium: Bridging the Gaps, Ottawa, Ontario, Canada, 2–6 October 2004. Google Scholar
- 6.
Balk EM ,Bonis PA ,Moskowitz H ,Schmid CH ,Ioannidis JP ,Wang C ,et al . Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials. JAMA. 2002;287:2973-82. [PMID:12052127 ] CrossrefMedlineGoogle Scholar - 7.
Pildal J ,Hróbjartsson A ,Jørgensen KJ ,Hilden J ,Altman DG ,Gøtzsche PC . Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol. 2007;36:847-57. [PMID:17517809 ] CrossrefMedlineGoogle Scholar - 8.
Kjaergard LL ,Villumsen J ,Gluud C . Reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Ann Intern Med. 2001;135:982-9. [PMID:11730399 ] LinkGoogle Scholar - 9.
Wood L ,Egger M ,Gluud LL ,Schulz KF ,Jüni P ,Altman DG ,et al . Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ. 2008;336:601-5. [PMID:18316340 ] CrossrefMedlineGoogle Scholar - 10.
Savović J ,Harris RJ ,Wood L ,Beynon R ,Altman D ,Als-Nielsen B ,et al . Development of a combined database for meta-epidemiological research. Research Synthesis Methods. 2010;1:212-25. CrossrefMedlineGoogle Scholar - 11.
Siersma V ,Als-Nielsen B ,Chen W ,Hilden J ,Gluud LL ,Gluud C . Multivariable modelling for meta-epidemiological assessment of the association between trial quality and treatment effects estimated in randomized clinical trials. Stat Med. 2007;26:2745-58. [PMID:17117373 ] CrossrefMedlineGoogle Scholar - 12.
Contopoulos-Ioannidis DG ,Gilbody SM ,Trikalinos TA ,Churchill R ,Wahlbeck K ,Ioannidis JP . Comparison of large versus smaller randomized trials for mental health-related interventions. Am J Psychiatry. 2005;162:578-84. [PMID:15741476 ] CrossrefMedlineGoogle Scholar - 13. Savović J, Jones HE, Altman DG, Harris RJ, Jüni P, Pildal J, et al. Influence of study design characteristics on intervention effect estimates from randomised controlled trials: combined analysis of meta-epidemiological studies. Health Technol Assess. 2012. [Forthcoming]. Google Scholar
- 14.
Welton NJ ,Ades AE ,Carlin JB ,Altman DG ,Sterne JAC . Models for potentially biased evidence in meta-analysis using empirically based priors. J R Stat Soc Ser A Stat Soc. 2009;172:119-36. CrossrefGoogle Scholar - 15.
Moja LP ,Telaro E ,D'Amico R ,Moschetti I ,Coe L ,Liberati A . Assessment of methodological quality of primary studies by systematic reviews: results of the metaquality cross sectional study. BMJ. 2005;330:1053. [PMID:15817526 ] CrossrefMedlineGoogle Scholar - 16. International Statistical Classification of Diseases and Related Health Problems, 10th Revision. World Health Organization; 2007. Accessed at http://apps.who.int/classifications/apps/icd/icd10online on 19 August 2010. Google Scholar
- 17.
Dechartres A ,Boutron I ,Trinquart L ,Charles P ,Ravaud P . Single-center trials show larger treatment effects than multicenter trials: evidence from a meta-epidemiologic study. Ann Intern Med. 2011;155:39-51. [PMID:21727292 ] LinkGoogle Scholar - 18.
Hopewell S ,Dutton S ,Yu LM ,Chan AW ,Altman DG . The quality of reports of randomised trials in 2000 and 2006: comparative study of articles indexed in PubMed. BMJ. 2010;340:c723. [PMID:20332510 ] CrossrefMedlineGoogle Scholar - 19.
Huwiler-Müntener K ,Jüni P ,Junker C ,Egger M . Quality of reporting of randomized trials as a measure of methodologic quality. JAMA. 2002;287:2801-4. [PMID:12038917 ] CrossrefMedlineGoogle Scholar - 20.
Hill CL ,LaValley MP ,Felson DT . Discrepancy between published report and actual conduct of randomized clinical trials. J Clin Epidemiol. 2002;55:783-6. [PMID:12384192 ] CrossrefMedlineGoogle Scholar - 21.
Hróbjartsson A ,Pildal J ,Chan AW ,Haahr MT ,Altman DG ,Gøtzsche PC . Reporting on blinding in trial protocols and corresponding publications was often inadequate but rarely contradictory. J Clin Epidemiol. 2009;62:967-73. [PMID:19635403 ] CrossrefMedlineGoogle Scholar - 22.
Soares HP ,Daniels S ,Kumar A ,Clarke M ,Scott C ,Swann S ,et al ;Radiation Therapy Oncology Group . Bad reporting does not mean bad methods for randomised trials: observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group. BMJ. 2004;328:22-4. [PMID:14703540 ] CrossrefMedlineGoogle Scholar - 23.
Pildal J ,Chan AW ,Hróbjartsson A ,Forfang E ,Altman DG ,Gøtzsche PC . Comparison of descriptions of allocation concealment in trial protocols and the published reports: cohort study. BMJ. 2005;330:1049. [PMID:15817527 ] CrossrefMedlineGoogle Scholar - 24.
Higgins JPT ,Altman DG . Chapter 8: Assessing risk of bias in included studies. In: Higgins JPT, Green S, eds. Cochrane Handbook for Systematic Reviews of Interventions. Chichester, United Kingdom: J Wiley; 2008:187-241. Google Scholar - 25.
Higgins JP ,Altman DG ,Gøtzsche PC ,Jüni P ,Moher D ,Oxman AD ,et al ;Cochrane Bias Methods Group . The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928. [PMID:22008217 ] CrossrefMedlineGoogle Scholar - 26.
Nüesch E ,Trelle S ,Reichenbach S ,Rutjes AW ,Bürgi E ,Scherer M ,et al . The effects of excluding patients from the analysis in randomised controlled trials: meta-epidemiological study. BMJ. 2009;339:b3244. [PMID:19736281 ] CrossrefMedlineGoogle Scholar - 27.
Vestbo J ,Anderson JA ,Calverley PM ,Celli B ,Ferguson GT ,Jenkins C ,et al . Bias due to withdrawal in long-term randomised trials in COPD: evidence from the TORCH study. Clin Respir J. 2011;5:44-9. [PMID:21159140 ] CrossrefMedlineGoogle Scholar - 28.
Dwan K ,Altman DG ,Arnaiz JA ,Bloom J ,Chan AW ,Cronin E ,et al . Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One. 2008;3:3081. [PMID:18769481 ] CrossrefMedlineGoogle Scholar - 29.
Kirkham JJ ,Dwan KM ,Altman DG ,Gamble C ,Dodd S ,Smyth R ,et al . The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ. 2010;340:c365. [PMID:20156912 ] CrossrefMedlineGoogle Scholar - 30.
Nüesch E ,Reichenbach S ,Trelle S ,Rutjes AW ,Liewald K ,Sterchi R ,et al . The importance of allocation concealment and patient blinding in osteoarthritis trials: a meta-epidemiologic study. Arthritis Rheum. 2009;61:1633-41. [PMID:19950329 ] CrossrefMedlineGoogle Scholar - 31.
Hróbjartsson A ,Gøtzsche PC . Is the placebo powerless? An analysis of clinical trials comparing placebo with no treatment. N Engl J Med. 2001;344:1594-602. [PMID:11372012 ] CrossrefMedlineGoogle Scholar - 32.
Krogsbøll LT ,Hróbjartsson A ,Gøtzsche PC . Spontaneous improvement in randomised clinical trials: meta-analysis of three-armed trials comparing no treatment, placebo and active intervention. BMC Med Res Methodol. 2009;9:1. [PMID:19123933 ] CrossrefMedlineGoogle Scholar - 33.
Devereaux PJ ,Manns BJ ,Ghali WA ,Quan H ,Lacchetti C ,Montori VM ,et al . Physician interpretations and textbook definitions of blinding terminology in randomized controlled trials. JAMA. 2001;285:2000-3. [PMID:11308438 ] CrossrefMedlineGoogle Scholar - 34.
Montori VM ,Bhandari M ,Devereaux PJ ,Manns BJ ,Ghali WA ,Guyatt GH . In the dark: the reporting of blinding status in randomized controlled trials. J Clin Epidemiol. 2002;55:787-90. [PMID:12384193 ] CrossrefMedlineGoogle Scholar - 35.
Moher D ,Hopewell S ,Schulz KF ,Montori V ,Gøtzsche PC ,Devereaux PJ ,et al . CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;340:c869. [PMID:20332511 ] CrossrefMedlineGoogle Scholar - 36.
Schulz KF ,Altman DG ,Moher D ;CONSORT Group . CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;340:c332. [PMID:20332509 ] CrossrefMedlineGoogle Scholar - 37.
Forder PM ,Gebski VJ ,Keech AC . Allocation concealment and blinding: when ignorance is bliss. Med J Aust. 2005;182:87-9. [PMID:15651970 ] CrossrefMedlineGoogle Scholar - 38.
Siontis GC ,Tzoulaki I ,Ioannidis JP . Predicting death: an empirical evaluation of predictive tools for mortality. Arch Intern Med. 2011;171:1721-6. [PMID:21788535 ] CrossrefMedlineGoogle Scholar - 39.
Goodman S ,Dickersin K . Metabias: a challenge for comparative effectiveness research [Editorial]. Ann Intern Med. 2011;155:61-2. [PMID:21727295 ] LinkGoogle Scholar - 40.
Lunn DJ ,Thomas A ,Best N ,Spiegelhalter D . WinBUGS—A Bayesian modelling framework: concepts, structure, and extensibility. Statistics and Computing. 2000;10:325-37. CrossrefGoogle Scholar - 41.
Lambert PC ,Sutton AJ ,Burton PR ,Abrams KR ,Jones DR . How vague is vague? A simulation study of the impact of the use of vague prior distributions in MCMC using WinBUGS. Stat Med. 2005;24:2401-28. [PMID:16015676 ] CrossrefMedlineGoogle Scholar - 42.
Gelman A . Prior distributions for variance parameters in hierarchical models. Bayesian Analysis. 2006;1:515-33. CrossrefGoogle Scholar
Author, Article, and Disclosure Information
From the School of Social and Community Medicine, University of Bristol, Bristol, United Kingdom; Centre for Statistics in Medicine, University of Oxford, Oxford, United Kingdom; Health Protection Agency, London, United Kingdom; Institute of Social and Preventive Medicine, University of Bern, Switzerland; The Nordic Cochrane Centre and Copenhagen Trial Unit, Centre for Clinical Intervention Research, Copenhagen University Hospital, Copenhagen, Denmark; Hilleroed Hospital, Hilleroed, Denmark; Tufts Clinical and Translational Science Institute, Tufts Medical Center, Boston, Massachusetts; Copenhagen University Hospital Gentofte, Hellerup, Denmark; Stanford Prevention Research Center, Stanford University School of Medicine, and Stanford University School of Humanities and Sciences, Stanford, California; FHI 360, Durham, North Carolina; Ottawa Methods Centre, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada; The Children's Food Trust, Sheffield, United Kingdom; and Public Health, Epidemiology and Biostatistics, School of Health and Population Sciences, College of Medical and Dental Sciences, University of Birmingham, Birmingham, United Kingdom.
Note: Drs. Savović, Jones, and Sterne had full access to all study data and take responsibility for the integrity of the data and accuracy of the data analysis.
Acknowledgment: The authors thank Matthias Egger for providing data and contributing to the initial stages of this project.
Grant Support: This project was funded by the National Institute for Health Research Health Technology Assessment programme in the United Kingdom (project 06/91/10) and will be published in full (13). See the Health Technology Assessment programme Web site for further project information (www.hta.ac.uk). Dr. Savović was partially funded by the Medical Research Council UK (grant G0701659/1). Data contributed by Dr. Als-Nielsen and colleagues were from a study that was partially funded by the Danish Centre for Evaluation and Health Technology Assessment.
Disclosures: Disclosures can be viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum=M11-2327.
Corresponding Author: Jonathan A.C. Sterne, PhD, School of Social and Community Medicine, University of Bristol, Canynge Hall, 39 Whatley Road, Bristol BS8 2PS, United Kingdom.
Current Author Addresses: Drs. Savović, Jones, Welton, and Sterne and Ms. Beynon: University of Bristol, School of Social and Community Medicine, Canynge Hall, 39 Whatley Road, Bristol BS8 2PS, United Kingdom.
Professor Altman: Centre for Statistics in Medicine, University of Oxford, Wolfson College Annexe, Linton Road, Oxford OX2 6UD, United Kingdom.
Mr. Harris: Health Protection Services, Health Protection Agency, 61 Colindale Avenue, London NW9 5EQ, United Kingdom.
Dr. Jüni: Division of Clinical Epidemiology & Biostatistics, Institut für Sozial- und Präventivmedizin, Universität Bern, Finkenhubelweg 11, CH-3012 Bern, Switzerland.
Dr. Pildal: The Nordic Cochrane Centre, Rigshospitalet, 7811, Blegdamsvej 9, DK-2100 Copenhagen, Denmark.
Dr. Als-Nielsen: Department of Pediatrics, Hilleroed Hospital, Dyrehavevej 29, DK-3400 Hilleroed, Denmark.
Dr. Balk: Tufts Clinical and Translational Science Institute, Tufts Medical Center, 800 Washington Street, Box 63, Boston, MA 02111.
Dr. C. Gluud: Copenhagen Trial Unit, Centre for Clinical Intervention Research, Copenhagen University Hospital, Rigshospitalet, Blegdamsvej 9, DK-2100 Copenhagen, Denmark.
Dr. L.L. Gluud: Department of Internal Medicine, Copenhagen University Hospital Gentofte, Niels Andersens Vej 65, 2900 Hellerup, Denmark.
Dr. Ioannidis: Stanford University School of Medicine, Room X306, 1265 Welch Road, Stanford, CA 94305.
Dr. Schulz: Quantitative Sciences, FHI 360, Research Triangle Park, 2224 East NC Highway 54, Durham, NC 27713.
Dr. Wood: Children's Food Trust, Regus Temple Quay, 1 Friary, Temple Quay, Bristol BS1 1EA, United Kingdom.
Dr. Moher: Ottawa Methods Centre, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Box 208, 501 Smyth Road, Ottawa, Ontario K1H 8L6, Canada.
Dr. Deeks: College of Medical and Dental Sciences, University of Birmingham, Edgbaston, Birmingham B15 2TT, United Kingdom.
Author Contributions: Conception and design: J. Savović, D.G. Altman, P. Jüni, J. Pildal, J.P.A. Ioannidis, K. F. Schulz, D. Moher, J.J. Deeks, J.A.C. Sterne.
Analysis and interpretation of the data: J. Savović, H.E. Jones, D.G. Altman, R.J. Harris, B. Als-Nielsen, E.M. Balk, C. Gluud, J.P.A. Ioannidis, K.F. Schulz, N.J. Welton, D. Moher, J.J. Deeks, J.A.C. Sterne.
Drafting of the article: J. Savović, H.E. Jones, J.A.C. Sterne.
Critical revision of the article for important intellectual content: J. Savović, H.E. Jones, D.G. Altman, R.J. Harris, P. Jüni, J. Pildal, B. Als-Nielsen, C. Gluud, L.L. Gluud, J.P.A. Ioannidis, K.F. Schulz, N.J. Welton, L. Wood, D. Moher, J.J. Deeks, J.A.C. Sterne.
Final approval of the article: J. Savović, D.G. Altman, R.J. Harris, P. Jüni, J. Pildal, B. Als-Nielsen, C. Gluud, L.L. Gluud, J.P.A. Ioannidis, K.F. Schulz, N.J. Welton, L. Wood, D. Moher, J.J. Deeks, J.A.C. Sterne.
Provision of study materials or patients: J. Pildal, B. Als-Nielsen, E.M. Balk, L.L. Gluud, J.P.A. Ioannidis, K.F. Schulz.
Statistical expertise: H.E. Jones, D.G. Altman, R.J. Harris, P. Jüni, J.P.A. Ioannidis, K.F. Schulz, N.J. Welton, J.J. Deeks, J.A.C. Sterne.
Obtaining of funding: D.G. Altman, C. Gluud, J.J. Deeks, J.A.C. Sterne.
Collection and assembly of data: J. Savović, J. Pildal, B. Als-Nielsen, E.M. Balk, C. Gluud, L.L. Gluud, J.P.A. Ioannidis, K.F. Schulz, R. Beynon, L. Wood, J.A.C. Sterne.
This article was published at www.annals.org on 4 September 2012.
Submit a Comment
Contributors must reveal any conflict of interest. Comments are moderated. Please see our information for authorsregarding comments on an Annals publication.
*All comments submitted after October 1, 2021 and selected for publication will be published online only.