Research and Reporting Methods
8 October 2024

Target Trial Emulation for Evaluating Health Policy

Publication: Annals of Internal Medicine
Volume 177, Number 11

Abstract

Target trial emulation is an approach to designing rigorous nonexperimental studies by “emulating” key features of a clinical trial. Most commonly used outside of policy contexts, this approach is also valuable for policy evaluation as policies typically are not randomly assigned. In this article, we discuss the application of the target trial emulation framework in a policy evaluation context. The policy trial emulation framework includes 7 components: the units and eligibility criteria, definitions of the exposure and comparison conditions, assignment mechanism, baseline (“time zero”) and follow-up, outcomes, causal estimand, and statistical analysis and assumptions. Policy evaluations that emulate a randomized trial across these dimensions can yield estimates of the causal effects of the policy on outcomes. Using the policy trial emulation framework to conduct and report on research design and methods supports transparent assessment of threats to causal inference in nonexperimental studies intended to assess the effect of a health policy on clinical or population health outcomes.

Get full access to this article

View all available purchase options and get full access to this article.

Supplemental Material

Supplementary Material

References

1.
Kingdon JW. Agendas, Alternatives, and Public Policies. Longman Classics in Political Science. 2nd ed. Longman; 2003:253.
2.
Rawat P, Morris JC. Kingdon’s “Streams” model at thirty: still relevant in the 21st century? Polit Policy. 2016;44:608-638. [10.1111/POLP.12168] doi: 10.1111/POLP.12168
3.
Dahabreh IJ, Matthews A, Steingrimsson JA, et al. Using trial and observational data to assess effectiveness: trial emulation, transportability, benchmarking, and joint analysis. Epidemiol Rev. 8 February 2023. [Epub ahead of print]. [PMID: 36752592] doi: 10.1093/epirev/mxac011
4.
Hansford HJ, Cashin AG, Jones MD, et al. Reporting of observational studies explicitly aiming to emulate randomized trials: a systematic review. JAMA Netw Open. 2023;6:e2336023. [PMID: 37755828] doi: 10.1001/jamanetworkopen.2023.36023
5.
Wan EYF, Yan VKC, Mok AHY, et al. Effectiveness of molnupiravir and nirmatrelvir-ritonavir in hospitalized patients with COVID-19: a target trial emulation study. Ann Intern Med. 2023;176:505-514. [PMID: 36913693] doi: 10.7326/M22-3057
6.
Hulme WJ, Williamson E, Horne EMF, et al. Challenges in estimating the effectiveness of COVID-19 vaccination using observational data. Ann Intern Med. 2023;176:685-693. [PMID: 37126810] doi: 10.7326/M21-4269
7.
Lazzati A, Epaud S, Ortala M, et al. Effect of bariatric surgery on cancer risk: results from an emulated target trial using population-based data. Br J Surg. 2022;109:433-438. [PMID: 35136932] doi: 10.1093/bjs/znac003
8.
Fu EL. Target trial emulation to improve causal inference from observational data: what, why, and how? J Am Soc Nephrol. 2023;34:1305-1314. [PMID: 37131279] doi: 10.1681/ASN.0000000000000152
9.
Hernán MA, Wang W, Leaf DE. Target trial emulation: a framework for causal inference from observational data. JAMA. 2022;328:2446-2447. [PMID: 36508210] doi: 10.1001/jama.2022.21383
10.
Matthews AA, Danaei G, Islam N, et al. Target trial emulation: applying principles of randomised trials to observational studies. BMJ. 2022;378:e071108. [PMID: 36041749] doi: 10.1136/bmj-2022-071108
11.
Ben-Michael E, Feller A, Stuart EA. A trial emulation approach for policy evaluations with group-level longitudinal data. Epidemiology. 2021;32:533-540. [PMID: 34001754] doi: 10.1097/EDE.0000000000001369
12.
McGinty EE, Tormohlen KN, Seewald NJ, et al. Effects of U.S. state medical cannabis laws on treatment of chronic noncancer pain. Ann Intern Med. 2023;176:904-912. [PMID: 37399549] doi: 10.7326/M23-0053
13.
Hansford HJ, Cashin AG, Jones MD, et al. Development of the TrAnsparent ReportinG of observational studies Emulating a Target trial (TARGET) guideline. BMJ Open. 2023;13:e074626. [PMID: 37699620] doi: 10.1136/bmjopen-2023-074626
14.
Hemming K, Taljaard M. Reflection on modern methods: when is a stepped-wedge cluster randomized trial a good study design choice? Int J Epidemiol. 2020;49:1043-1052. [PMID: 32386407] doi: 10.1093/ije/dyaa077
15.
Stuart EA, Huskamp HA, Duckworth K, et al. Using propensity scores in difference-in-differences models to estimate the effects of a policy change. Health Serv Outcomes Res Methodol. 2014;14:166-182. [PMID: 25530705] doi: 10.1007/s10742-014-0123-z
16.
Feller A, Connors MC, Weiland C, et al. Addressing missing data due to COVID-19: two early childhood case studies. J Res Educ Eff. 15 August 2024. [Epub ahead of print]. doi: 10.1080/19345747.2024.2321438.
17.
Miller D, Spybrook J, Caverly S. Missing Data in Group Design Studies: Revisions in WWC Standards Version 4.0. Institute of Education Sciences; 2019.
18.
Incze MA, Kelley AT, Singer PM. Heterogeneous state cannabis policies: potential implications for patients and health care professionals. JAMA. 2021;326:2363-2364. [PMID: 34797377] doi: 10.1001/jama.2021.21182
19.
Ramanathan T, Hulkower R, Holbrook J, et al. Legal epidemiology: the science of law. J Law Med Ethics. 2017;45:69-72. [PMID: 28661299] doi: 10.1177/1073110517703329
20.
Griffin BA, Schuler MS, Pane J, et al. Methodological considerations for estimating policy effects in the context of co-occurring policies. Health Serv Outcomes Res Methodol. 2023;23:149-165. [PMID: 37207017] doi: 10.1007/s10742-022-00284-w
21.
Matthay EC, Hagan E, Joshi S, et al. The revolution will be hard to evaluate: how co-occurring policy changes affect research on the health effects of social policies. Epidemiol Rev. 2021;43:19-32. [PMID: 34622277] doi: 10.1093/epirev/mxab009
22.
Hayes RJ, Moulton LH. Cluster Randomised Trials. CRC Biostatistics Series. 2nd ed. CRC Pr; 2017.
23.
Verbitsky-Savitz N, Raudenbush SW. Causal inference under interference in spatial settings: a case study evaluating community policing program in Chicago. Epidemiol Methods. 2012;1:107-130.
24.
Baker AC, Larcker DF, Wang CCY. How much should we trust staggered difference-in-differences estimates? J Financ Econ. 2022;144:370-395.
25.
Hernán MA, Robins JM. Using big data to emulate a target trial when a randomized trial is not available. Am J Epidemiol. 2016;183:758-764. [PMID: 26994063] doi: 10.1093/aje/kwv254
26.
Cengiz D, Dube A, Lindner A, et al. The effect of minimum wages on low-wage jobs. Q J Econ. 2019;134:1405-1454.
27.
Deshpande M, Li Y. Who is screened out? Application costs and the targeting of disability programs. Am Econ J Econ Policy. 2019;11:213-248.
28.
Wing C, Yozwiak M, Hollingsworth A, et al. Designing difference-in-difference studies with staggered treatment adoption: key concepts and practical guidelines. Annu Rev Public Health. 2024;45:485-505. [PMID: 38277791] doi: 10.1146/annurev-publhealth-061022-050825
29.
Hernán MA, Robins JM, García Rodríguez LA. Discussion on “Statistical Issues Arising in the Women’s Health Initiative”. Biometrics. 2005;61:922-930.
30.
Schaubel DE, Wolfe RA, Port FK. A sequential stratification method for estimating the effect of a time-dependent experimental treatment in observational studies. Biometrics. 2006;62:910-917. [PMID: 16984335] doi: 10.1111/j.1541-0420.2006.00527.x
31.
Rothbard S, Etheridge JC, Murray EJ. A tutorial on applying the difference-in-differences method to health data. Curr Epidemiol Rep. 2024;11:85-95.
32.
Rubin DB. Estimating causal effects of treatments in randomized and nonrandomized studies. J Educ Psychol. 1974;66:688-701.
33.
Musci RJ, Stuart E. Ensuring causal, not casual, inference. Prev Sci. 2019;20:452-456. [PMID: 30613853] doi: 10.1007/s11121-018-0971-9
34.
Hernán MA. The C-word: scientific euphemisms do not improve causal inference from observational data. Am J Public Health. 2018;108:616-619. [PMID: 29565659] doi: 10.2105/AJPH.2018.304337
35.
Hernán MA, Sauer BC, Hernández-Díaz S, et al. Specifying a target trial prevents immortal time bias and other self-inflicted injuries in observational analyses. J Clin Epidemiol. 2016;79:70-75. [PMID: 27237061] doi: 10.1016/j.jclinepi.2016.04.014
36.
Rassen JA, Brookhart MA, Glynn RJ, et al. Instrumental variables I: instrumental variables exploit natural variation in nonexperimental data to estimate causal relationships. J Clin Epidemiol. 2009;62:1226-1232. [PMID: 19356901] doi: 10.1016/j.jclinepi.2008.12.005
37.
Campbell MJ, Walters SJ. How to Design, Analyse and Report Cluster Randomised Trials in Medicine and Health Related Research. Statistics in Practice. J Wiley; 2014.
38.
Roth J, Sant’Anna PHC, Bilinski A, et al. What’s trending in difference-in-differences? A synthesis of the recent econometrics literature. J Econom. 2023;235:2218-2244.
39.
Zeldow B, Hatfield LA. Confounding and regression adjustment in difference-in-differences studies. Health Serv Res. 2021;56:932-941. [PMID: 33978956] doi: 10.1111/1475-6773.13666
40.
Roth J, Sant’Anna PHC. When is parallel trends sensitive to functional form? Econometrica. 2023;91:737-747.
41.
Daw JR, Hatfield LA. Matching and regression to the mean in difference-in-differences analysis. Health Serv Res. 2018;53:4138-4156. [PMID: 29957834] doi: 10.1111/1475-6773.12993
42.
Daw JR, Hatfield LA. Matching in difference-in-differences: between a rock and a hard place. Health Serv Res. 2018;53:4111-4117. [PMID: 30047131] doi: 10.1111/1475-6773.13017
43.
Abadie A, Diamond A, Hainmueller J. Synthetic control methods for comparative case studies: estimating the effect of California’s Tobacco Control Program. J Am Stat Assoc. 2010;105:493-505.
44.
Callaway B, Sant’Anna PHC. Difference-in-differences with multiple time periods. J Econom. 2021;225:200-230.
45.
Ben-Michael E, Feller A, Rothstein J. The augmented synthetic control method. J Am Stat Assoc. 2021;116:1789-1803.
46.
Bandara SN, Kennedy-Hendricks A, Stuart EA, et al. The effects of the Maryland Medicaid Health Home Waiver on Emergency Department and inpatient utilization among individuals with serious mental illness. Gen Hosp Psychiatry. 2020;64:99-104. [PMID: 31948691] doi: 10.1016/j.genhosppsych.2019.12.004
47.
Griffin BA, Schuler MS, Stuart EA, et al. Moving beyond the classic difference-in-differences model: a simulation study comparing statistical methods for estimating effectiveness of state-level policies. BMC Med Res Methodol. 2021;21:279. [PMID: 34895172] doi: 10.1186/s12874-021-01471-y
48.
Dahabreh IJ, Bibbins-Domingo K. Causal inference about the effects of interventions from observational studies in medical journals. JAMA. 2024;331:1845-1853. [PMID: 38722735] doi: 10.1001/jama.2024.7741
49.
Guallar E, Goodman SN, Localio AR, et al. Seeing the positive in negative studies. Ann Intern Med. 2023;176:561-562. [PMID: 36940441] doi: 10.7326/M23-0576

Comments

0 Comments
Sign In to Submit A Comment

Information & Authors

Information

Published In

cover image Annals of Internal Medicine
Annals of Internal Medicine
Volume 177Number 11November 2024
Pages: 1530 - 1538

History

Published online: 8 October 2024
Published in issue: November 2024

Keywords

Authors

Affiliations

Nicholas J. Seewald, PhD https://orcid.org/0000-0002-8367-0522
Department of Health Policy and Management, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland; and Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania (N.J.S.)
Division of Health Policy and Economics, Weill Cornell Medicine, New York, New York (E.E.M.)
Elizabeth A. Stuart, PhD https://orcid.org/0000-0002-9042-8611
Department of Biostatistics, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland (E.A.S.).
Grant Support: By National Institutes of Health National Institute on Drug Abuse (NIDA) grant R01DA049789.
Corresponding Author: Nicholas J. Seewald, PhD, University of Pennsylvania Perelman School of Medicine, 624 Blockley Hall, 423 Guardian Drive, Philadelphia, PA 19104; e-mail, [email protected].
Author Contributions: Conception and design: N.J. Seewald, E.E. McGinty, E.A. Stuart.
Analysis and interpretation of the data: N.J. Seewald, E.E. McGinty, E.A. Stuart.
Drafting of the article: N.J. Seewald.
Critical revision of the article for important intellectual content: N.J. Seewald, E.E. McGinty, E.A. Stuart.
Final approval of the article: N.J. Seewald, E.E. McGinty, E.A. Stuart.
Statistical expertise: N.J. Seewald, E.A. Stuart.
Obtaining of funding: E.E. McGinty, E.A. Stuart.
Administrative, technical, or logistic support: N.J. Seewald, E.E. McGinty.
This article was published at Annals.org on 8 October 2024.

Metrics & Citations

Metrics

Citations

If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. For an editable text file, please select Medlars format which will download as a .txt file. Simply select your manager software from the list below and click Download.

For more information or tips please see 'Downloading to a citation manager' in the Help menu.

Format





Download article citation data for:
Nicholas J. Seewald, Emma E. McGinty, Elizabeth A. Stuart. Target Trial Emulation for Evaluating Health Policy. Ann Intern Med.2024;177:1530-1538. [Epub 8 October 2024]. doi:10.7326/M23-2440

View More

Login Options:
Purchase

You will be redirected to acponline.org to sign-in to Annals to complete your purchase.

Access to EPUBs and PDFs for FREE Annals content requires users to be registered and logged in. A subscription is not required. You can create a free account below or from the following link. You will be redirected to acponline.org to create an account that will provide access to Annals. If you are accessing the Free Annals content via your institution's access, registration is not required.

Create your Free Account

You will be redirected to acponline.org to create an account that will provide access to Annals.

View options

PDF/EPUB

View PDF/EPUB

Related in ACP Journals

Full Text

View Full Text

Figures

Tables

Media

Share

Share

Copy the content Link

Share on social media