Original Research2 July 2019
An Instrumental Variable Analysis
    Author, Article, and Disclosure Information

    Accountable care organizations (ACOs) in the Medicare Shared Savings Program (MSSP) are associated with modest savings. However, prior research may overstate this effect if high-cost clinicians exit ACOs.


    To evaluate the effect of the MSSP on spending and quality while accounting for clinicians' nonrandom exit.


    Similar to prior MSSP analyses, this study compared MSSP ACO participants versus control beneficiaries using adjusted longitudinal models that accounted for secular trends, market factors, and beneficiary characteristics. To further account for selection effects, the share of nearby clinicians in the MSSP was used as an instrumental variable. Hip fracture served as a falsification outcome. The authors also tested for compositional changes among MSSP participants.


    Fee-for-service Medicare, 2008 through 2014.


    A 20% sample (97 204 192 beneficiary-quarters).


    Total spending, 4 quality indicators, and hospitalization for hip fracture.


    In adjusted longitudinal models, the MSSP was associated with spending reductions (change, −$118 [95% CI, −$151 to −$85] per beneficiary-quarter) and improvements in all 4 quality indicators. In instrumental variable models, the MSSP was not associated with spending (change, $5 [CI, −$51 to $62] per beneficiary-quarter) or quality. In falsification tests, the MSSP was associated with hip fracture in the adjusted model (−0.24 hospitalizations for hip fracture [CI, −0.32 to −0.16 hospitalizations] per 1000 beneficiary-quarters) but not in the instrumental variable model (0.05 hospitalizations [CI, −0.10 to 0.20 hospitalizations] per 1000 beneficiary-quarters). Compositional changes were driven by high-cost clinicians exiting ACOs: High-cost clinicians (99th percentile) had a 30.4% chance of exiting the MSSP, compared with a 13.8% chance among median-cost clinicians (50th percentile).


    The study used an observational design and administrative data.


    After adjustment for clinicians' nonrandom exit, the MSSP was not associated with improvements in spending or quality. Selection effects—including exit of high-cost clinicians—may drive estimates of savings in the MSSP.

    Primary Funding Source:

    Horowitz Foundation for Social Policy, Agency for Healthcare Research and Quality, and National Institute on Aging.


    • 1. Muhlestein D, Saunders RS, McClellan MB. Growth of ACOs and alternative payment models in 2017. Health Affairs Blog. 28 June 2017. Accessed at www.healthaffairs.org/do/10.1377/hblog20170628.060719/full on 1 April 2019. Google Scholar
    • 2. McWilliams JMHatfield LAChernew MELandon BESchwartz ALEarly performance of accountable care organizations in Medicare. N Engl J Med2016;374:2357-66. [PMID: 27075832] doi:10.1056/NEJMsa1600142 CrossrefMedlineGoogle Scholar
    • 3. McWilliams JMChanges in Medicare Shared Savings Program savings from 2013 to 2014 [Letter]. JAMA2016;316:1711-3. [PMID: 27612292] doi:10.1001/jama.2016.12049 CrossrefMedlineGoogle Scholar
    • 4. Colla CHLewis VAKao LSO'Malley AJChang CHFisher ESAssociation between Medicare accountable care organization implementation and spending among clinically vulnerable beneficiaries. JAMA Intern Med2016;176:1167-75. [PMID: 27322485] doi:10.1001/jamainternmed.2016.2827 CrossrefMedlineGoogle Scholar
    • 5. Schur CLSutton JPPhysicians in Medicare ACOs offer mixed views of model for health care cost and quality. Health Aff (Millwood)2017;36:649-54. [PMID: 28373330] doi:10.1377/hlthaff.2016.1427 CrossrefMedlineGoogle Scholar
    • 6. Markovitz AARamsay PPShortell SMRyan AMFinancial incentives and physician practice participation in Medicare's value-based reforms. Health Serv Res2018;53 Suppl 1:3052-69. [PMID: 28748535] doi:10.1111/1475-6773.12743 CrossrefMedlineGoogle Scholar
    • 7. Lewis VAColla CHCarluzzo KLKler SEFisher ESAccountable care organizations in the United States: market and demographic factors associated with formation. Health Serv Res2013;48:1840-58. [PMID: 24117222] doi:10.1111/1475-6773.12102 CrossrefMedlineGoogle Scholar
    • 8. Hsu JVogeli CPrice MBrand RChernew MEMohta Net alSubstantial physician turnover and beneficiary ‘churn' in a large Medicare Pioneer ACO. Health Aff (Millwood)2017;36:640-8. [PMID: 28373329] doi:10.1377/hlthaff.2016.1107 CrossrefMedlineGoogle Scholar
    • 9. Markovitz AAHollingsworth JMAyanian JZNorton ECMoloci NMYan PLet alRisk adjustment in Medicare ACO program deters coding increases but may lead ACOs to drop high-risk beneficiaries. Health Aff (Millwood)2019;38:253-61. [PMID: 30715995] doi:10.1377/hlthaff.2018.05407 CrossrefMedlineGoogle Scholar
    • 10. Centers for Medicare & Medicaid Services. Shared savings and losses and assignment methodology: specifications (version 3). 2014. Accessed at www.cms.gov/Medicare/Medicare-Fee-for-Service-Payment/sharedsavingsprogram/Downloads/Shared-Savings-Losses-Assignment-Spec.pdf on 6 January 2019. Google Scholar
    • 11. Nyweide DJLee WCuerdon TTPham HHCox MRajkumar Ret alAssociation of Pioneer Accountable Care Organizations vs traditional Medicare fee for service with spending, utilization, and patient experience. JAMA2015;313:2152-61. [PMID: 25938875] doi:10.1001/jama.2015.4930 CrossrefMedlineGoogle Scholar
    • 12. Gottlieb DJZhou WSong YAndrews KGSkinner JSSutherland JMPrices don't drive regional Medicare spending variations. Health Aff (Millwood)2010;29:537-43. [PMID: 20110290] doi:10.1377/hlthaff.2009.0609 CrossrefMedlineGoogle Scholar
    • 13. Agency for Healthcare Research and Quality. Prevention quality indicators technical specifications - version 6.0. 2016. Accessed at www.qualityindicators.ahrq.gov/modules/PQI_TechSpec.aspx on 6 January 2019. Google Scholar
    • 14. RTI International. Accountable care organization 2014 program analysis quality performance standards narrative measure specifications. 2014. Accessed at www.healthreform.ct.gov/ohri/lib/ohri/work_groups/quality/2014-09-03/aco_narrative_measures_specs_1.pdf on 6 January 2019 Google Scholar
    • 15. Prasad VJena ABPrespecified falsification end points: can they validate true observational associations? JAMA2013;309:241-2. [PMID: 23321761] doi:10.1001/jama.2012.96867 CrossrefMedlineGoogle Scholar
    • 16. Colla CHWennberg DEMeara ESkinner JSGottlieb DLewis VAet alSpending differences associated with the Medicare Physician Group Practice Demonstration. JAMA2012;308:1015-23. [PMID: 22968890] doi:10.1001/2012.jama.10812 CrossrefMedlineGoogle Scholar
    • 17. Fisher ESWennberg JEStukel TASharp SMHospital readmission rates for cohorts of Medicare beneficiaries in Boston and New Haven. N Engl J Med1994;331:989-95. [PMID: 8084356] CrossrefMedlineGoogle Scholar
    • 18. Wennberg JEFreeman JLShelton RMBubolz TAHospital use and mortality among Medicare beneficiaries in Boston and New Haven. N Engl J Med1989;321:1168-73. [PMID: 2677726] CrossrefMedlineGoogle Scholar
    • 19. McClellan MMcNeil BJNewhouse JPDoes more intensive treatment of acute myocardial infarction in the elderly reduce mortality? Analysis using instrumental variables. JAMA1994;272:859-66. [PMID: 8078163] CrossrefMedlineGoogle Scholar
    • 20. Valley TSSjoding MWRyan AMIwashyna TJCooke CRAssociation of intensive care unit admission with mortality among older patients with pneumonia. JAMA2015;314:1272-9. [PMID: 26393850] doi:10.1001/jama.2015.11068 CrossrefMedlineGoogle Scholar
    • 21. Bekelis KMarth NJWong KZhou WBirkmeyer JDSkinner JPrimary stroke center hospitalization for elderly patients with stroke: implications for case fatality and travel times. JAMA Intern Med2016;176:1361-8. [PMID: 27455403] doi:10.1001/jamainternmed.2016.3919 CrossrefMedlineGoogle Scholar
    • 22. Crooks VASchuurman NInterpreting the results of a modified gravity model: examining access to primary health care physicians in five Canadian provinces and territories. BMC Health Serv Res2012;12:230. [PMID: 22852816] doi:10.1186/1472-6963-12-230 CrossrefMedlineGoogle Scholar
    • 23. Davidson RMacKinnon JGEstimation and Inference in Econometrics. New York: Oxford Univ Pr; 1993. Google Scholar
    • 24. Luo WWang FMeasures of spatial accessibility to health care in a GIS environment: synthesis and a case study in the Chicago region. Environ Plan B Urban Anal City Sci2003;30:865-84. Google Scholar
    • 25. McGrail MRHumphreys JSMeasuring spatial accessibility to primary care in rural areas: improving the effectiveness of the two-step floating catchment area method. Appl Geogr2009;29:533-41. CrossrefGoogle Scholar
    • 26. WWAMI Rural Health Research CenterRural Urban Commuting Area Data: ZIP Code RUCA Approximation. Version 2.0. Seattle: WWAMI Rural Health Research Center; 2006. Google Scholar
    • 27. Winblad UMor VMcHugh JPRahman MACO-affiliated hospitals reduced rehospitalizations from skilled nursing facilities faster than other hospitals. Health Aff (Millwood)2017;36:67-73. [PMID: 28069848] doi:10.1377/hlthaff.2016.0759 CrossrefMedlineGoogle Scholar
    • 28. McWilliams JMChernew MELandon BESchwartz ALPerformance differences in year 1 of Pioneer Accountable Care Organizations. N Engl J Med2015;372:1927-36. [PMID: 25875195] doi:10.1056/NEJMsa1414929 CrossrefMedlineGoogle Scholar
    • 29. United States Census Bureau. American FactFinder. 2017. Accessed at https://factfinder.census.gov/faces/nav/jsf/pages/index.xhtml on 6 January 2019. Google Scholar
    • 30. McWilliams JMGilstrap LGStevenson DGChernew MEHuskamp HAGrabowski DCChanges in postacute care in the Medicare Shared Savings Program. JAMA Intern Med2017;177:518-26. [PMID: 28192556] doi:10.1001/jamainternmed.2016.9115 CrossrefMedlineGoogle Scholar
    • 31. Baiocchi MCheng JSmall DSInstrumental variable methods for causal inference. Stat Med2014;33:2297-340. [PMID: 24599889] doi:10.1002/sim.6128 CrossrefMedlineGoogle Scholar
    • 32. Staiger DStock JInstrumental variables regression with weak instruments. Econometrica1997;65:557-86. CrossrefGoogle Scholar
    • 33. Baum CFSchaffer MEStillman SInstrumental variables and GMM: estimation and testing. Stata J2003;3:1-31. CrossrefGoogle Scholar
    • 34. Clough JDMcClellan MImplementing MACRA: implications for physicians and for physician leadership. JAMA2016;315:2397-8. [PMID: 27213914] doi:10.1001/jama.2016.7041 CrossrefMedlineGoogle Scholar