Research and Reporting Methods
11 July 2017

Sensitivity Analysis in Observational Research: Introducing the E-Value

Publication: Annals of Internal Medicine
Volume 167, Number 4

Abstract

Sensitivity analysis is useful in assessing how robust an association is to potential unmeasured or uncontrolled confounding. This article introduces a new measure called the “E-value,” which is related to the evidence for causality in observational studies that are potentially subject to confounding. The E-value is defined as the minimum strength of association, on the risk ratio scale, that an unmeasured confounder would need to have with both the treatment and the outcome to fully explain away a specific treatment–outcome association, conditional on the measured covariates. A large E-value implies that considerable unmeasured confounding would be needed to explain away an effect estimate. A small E-value implies little unmeasured confounding would be needed to explain away an effect estimate. The authors propose that in all observational studies intended to produce evidence for causality, the E-value be reported or some other sensitivity analysis be used. They suggest calculating the E-value for both the observed association estimate (after adjustments for measured confounders) and the limit of the confidence interval closest to the null. If this were to become standard practice, the ability of the scientific community to assess evidence from observational studies would improve considerably, and ultimately, science would be strengthened.

Get full access to this article

View all available purchase options and get full access to this article.

Supplemental Material

Supplement. Supplemental Appendix

References

1.
Wasserstein RLLazar NL. The ASA's statement on p-values: context, process, and purpose. Am Stat. 2016;70:129-33.
2.
Altman DGMachin DBryant TNGardner MJ. Statistics with Confidence. 2nd ed. London: BMJ Books; 2000.
3.
Rosner B. Fundamentals of Biostatistics. 8th ed. Boston: Cengage Learning; 2015.
4.
Pagano MGavreau K. Principles of Biostatistics. Belmont, CA: Brooks/Cole; 2000.
5.
Greenland SSenn SJRothman KJCarlin JBPoole CGoodman SNet al. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. Eur J Epidemiol. 2016;31:337-50. [PMID: 27209009]  doi: 10.1007/s10654-016-0149-3
6.
Goodman S. A dirty dozen: twelve p-value misconceptions. Semin Hematol. 2008;45:135-40. [PMID: 18582619]  doi: 10.1053/j.seminhematol.2008.04.003
7.
Greenland S. Null misinterpretation in statistical testing and its impact on health risk assessment. Prev Med. 2011;53:225-8. [PMID: 21871481]  doi: 10.1016/j.ypmed.2011.08.010
8.
Greenland SPoole C. Problems in common interpretations of statistics in scientific articles, expert reports, and testimony. Jurimetrics. 2011;51:11329.
9.
Sterne JADaveySmith G. Sifting the evidence—what's wrong with significance tests? BMJ. 2001;322:226-31. [PMID: 11159626]
10.
Stang APoole CKuss O. The ongoing tyranny of statistical significance testing in biomedical research. Eur J Epidemiol. 2010;25:225-30. [PMID: 20339903]  doi: 10.1007/s10654-010-9440-x
11.
Goodman SN. Toward evidence-based medical statistics. 1: The P value fallacy. Ann Intern Med. 1999;130:995-1004. [PMID: 10383371]
12.
Hill AB. The environment and disease: association or causation? Proc R Soc Med. 1965;58:295-300. [PMID: 14283879]
13.
Greenland S. Randomization, statistics, and causal inference. Epidemiology. 1990;1:421-9. [PMID: 2090279]
14.
Imbens GWRubin DB. Sensitivity analysis and bounds. In: Causal Inference for Statistics, Social, and Biomedical Sciences. New York: Cambridge Univ Pr; 2015:496-512.
15.
Hernan MA, Robins JR. Confounding. In: Causal Inference. 11 September 2016. Accessed at https://cdn1.sph.harvard.edu/wp-content/uploads/sites/1268/2016/09/hernanrobins_v1.10.31.pdf on 22 May 2017.
16.
Rosenbaum PR. Sensitivity to hidden bias. In: Observational Studies. 2nd ed. New York: Springer; 2002:105-70.
17.
Rosenbaum PR. Design sensitivity. In: Design of Observational Studies. New York: Springer; 2010:269-71.
18.
Rosenbaum PR. Design sensitivity and efficiency in observational studies. J Am Stat Assoc. 2010;105:692-702.
19.
Greenland S. Multiple-bias modeling for analysis of observational data. J R Stat Soc Ser A. 2005;168:267-308.
20.
Lash TLFox MPFink AK. Applying Quantitative Bias Analysis to Epidemiologic Data. New York: Springer; 2009.
21.
Greenland SLash TL. Bias analysis. In: Rothman KJ, Greenland S, Lash TL, eds. Modern Epidemiology. Philadelphia: Lippincott Williams & Wilkins; 2008:345-80.
22.
Cornfield JHaenszel WHammond ECLilienfeld AMShimkin MBWynder EL. Smoking and lung cancer: recent evidence and a discussion of some questions. J Natl Cancer Inst. 1959;22:173-203. [PMID: 13621204]
23.
Victora CGSmith PGVaughan JPNobre LCLombardi CTeixeira AMet al. Evidence for protection by breast-feeding against infant deaths from infectious diseases in Brazil. Lancet. 1987;2:319-22. [PMID: 2886775]
24.
Bross ID. Spurious effects from an extraneous variable. J Chronic Dis. 1966;19:637-47. [PMID: 5966011]
25.
Schlesselman JJ. Assessing effects of confounding variables. Am J Epidemiol. 1978;108:3-8. [PMID: 685974]
26.
Rosenbaum PRRubin DB. Assessing sensitivity to an unobserved binary covariate in an observational study with binary outcome. J R Stat Soc Series B Stat Methodol. 1983;45:212-8.
27.
Lin DYPsaty BMKronmal RA. Assessing the sensitivity of regression results to unmeasured confounders in observational studies. Biometrics. 1998;54:948-63. [PMID: 9750244]
28.
Imbens GW. Sensitivity to exogeneity assumptions in program evaluation. Am Econ Rev. 2003;93:126-32.
29.
Vanderweele TJArah OA. Bias formulas for sensitivity analysis of unmeasured confounding for general outcomes, treatments, and confounders. Epidemiology. 2011;22:42-52. [PMID: 21052008]  doi: 10.1097/EDE.0b013e3181f74493
30.
Bross ID. Pertinency of an extraneous variable. J Chronic Dis. 1967;20:487-95. [PMID: 6028268]
31.
Lee WC. Bounding the bias of unmeasured factors with confounding and effect modifying potentials. Stat Med. 2011;30:1007-17.
32.
Robins J MScharfstein DRotnitzky A. Sensitivity analysis for selection bias and unmeasured confounding in missing data and causal inference models. In: Halloran E, Berry D, eds. Statistical Models for Epidemiology, the Environment, and Clinical Trials. New York: Springer-Verlag; 2000:1-94.
33.
McCandless LCGustafson PLevy A. Bayesian sensitivity analysis for unmeasured confounding in observational studies. Stat Med. 2007;26:2331-47. [PMID: 16998821]
34.
Brumback BAHernán MAHaneuse SJRobins JM. Sensitivity analyses for unmeasured confounding assuming a marginal structural model for repeated measures. Stat Med. 2004;23:749-67. [PMID: 14981673]
35.
VanderWeele TJ. Sensitivity analysis for mediation. In: Explanation in Causal Inference: Methods for Mediation and Interaction. New York: Oxford Univ Pr; 2015:66-97.
36.
VanderWeele TJ. Sensitivity analysis for contagion effects in social networks. Sociol Methods Res. 2011;40:240-55. [PMID: 25580037]
37.
Ding PVanderWeele TJ. Sensitivity analysis without assumptions. Epidemiology. 2016;27:368-77. [PMID: 26841057]  doi: 10.1097/EDE.0000000000000457
38.
Gilbert PBBosch RJHudgens MG. Sensitivity analysis for the assessment of causal vaccine effects on viral load in HIV vaccine trials. Biometrics. 2003;59:531-41. [PMID: 14601754]
39.
Chiba YVanderWeele TJ. A simple method for principal strata effects when the outcome has been truncated due to death. Am J Epidemiol. 2011;173:745-51. [PMID: 21354986]  doi: 10.1093/aje/kwq418
40.
Huang THLee WC. Bounding formulas for selection bias. Am J Epidemiol. 2015;182:868-72. [PMID: 26519426]  doi: 10.1093/aje/kwv130
41.
Rosenbaum PR. Discussing hidden bias in observational studies. Ann Intern Med. 1991;115:901-5. [PMID: 1952480]
42.
Ip SChung MRaman GChew PMagula NDeVine Det al. Breastfeeding and Maternal and Infant Health Outcomes in Developed Countries. Evidence Report/Technology Assessment no. 153. (Prepared by Tufts-New England Medical Center Evidence-based Practice Center under contract no. 290-02-0022.) AHRQ publication no. 07-E007. Rockville: Agency for Healthcare Research and Quality; April 2007.
43.
Moorman PGCalingaert BPalmieri RTIversen ESBentley RCHalabi Set al. Hormonal risk factors for ovarian cancer in premenopausal and postmenopausal women. Am J Epidemiol. 2008;167:1059-69. [PMID: 18303003]  doi: 10.1093/aje/kwn006
44.
Taubes G. Epidemiology faces its limits. Science. 1995;269:164-9. [PMID: 7618077]
45.
VanderWeele TJ. On a square-root transformation of the odds ratio for a common outcome. Epidemiology. 2017. [Forthcoming].
46.
Borenstein MHedges LVHiggins JPTRothstein HR. Converting among effect sizes. In: Introduction to Meta-Analysis. Hoboken, NJ: Wiley; 2009:45-51.
47.
Hasselblad VHedges LV. Meta-analysis of screening and diagnostic tests. Psychol Bull. 1995;117:167-78. [PMID: 7870860]

Information & Authors

Information

Published In

cover image Annals of Internal Medicine
Annals of Internal Medicine
Volume 167Number 415 August 2017
Pages: 268 - 274

History

Published online: 11 July 2017
Published in issue: 15 August 2017

Keywords

Authors

Affiliations

Tyler J. VanderWeele, PhD
From Harvard T.H. Chan School of Public Health, Boston, Massachusetts, and University of California, Berkeley, Berkeley, California.
Peng Ding, PhD
From Harvard T.H. Chan School of Public Health, Boston, Massachusetts, and University of California, Berkeley, Berkeley, California.
Acknowledgment: The authors thank Sander Greenland, James Robins, 2 reviewers, and the editors for helpful comments on an earlier draft of this paper.
Grant Support: By National Institutes of Health grant ES017876.
Disclosures: Authors have disclosed no conflicts of interest. Forms can be viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum=M16-2607.
Corresponding Author: Tyler J. VanderWeele, PhD, Harvard T.H. Chan School of Public Health, 677 Huntington Avenue, Boston, MA 02115; e-mail, [email protected].
Current Author Addresses: Dr. VanderWeele: Harvard T.H. Chan School of Public Health, 677 Huntington Avenue, Boston, MA 02115.
Dr. Ding: University of California, Berkeley, 425 Evans Hall, Berkeley, CA 94720.
Author Contributions: Conception and design: T.J. VanderWeele, P. Ding.
Analysis and interpretation of the data: T.J. VanderWeele, P. Ding.
Drafting of the article: T.J. VanderWeele.
Critical revision for important intellectual content: T.J. VanderWeele.
Final approval of the article: T.J. VanderWeele, P. Ding.
Provision of study materials or patients: T.J. VanderWeele.
Statistical expertise: T.J. VanderWeele, P. Ding.
Obtaining of funding: T.J. VanderWeele.
Administrative, technical, or logistic support: T.J. VanderWeele.
Collection and assembly of data: T.J. VanderWeele, P. Ding.
This article was published at Annals.org on 11 July 2017.

Metrics & Citations

Metrics

Citations

If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. For an editable text file, please select Medlars format which will download as a .txt file. Simply select your manager software from the list below and click Download.

For more information or tips please see 'Downloading to a citation manager' in the Help menu.

Format





Download article citation data for:
Tyler J. VanderWeele, Peng Ding. Sensitivity Analysis in Observational Research: Introducing the E-Value. Ann Intern Med.2017;167:268-274. [Epub 11 July 2017]. doi:10.7326/M16-2607

View More

Get Access

Login Options:
Purchase

You will be redirected to acponline.org to sign-in to Annals to complete your purchase.

Access to EPUBs and PDFs for FREE Annals content requires users to be registered and logged in. A subscription is not required. You can create a free account below or from the following link. You will be redirected to acponline.org to create an account that will provide access to Annals. If you are accessing the Free Annals content via your institution's access, registration is not required.

Create your Free Account

You will be redirected to acponline.org to create an account that will provide access to Annals.

View options

PDF/ePub

View PDF/ePub

Related in ACP Journals

Full Text

View Full Text

Media

Figures

Other

Tables

Share

Share

Copy the content Link

Share on social media