Temporal Improvements in COVID-19 Outcomes for Hospitalized Adults: A Post Hoc Observational Study of Remdesivir Group Participants in the Adaptive COVID-19 Treatment Trial

The COVID-19 pandemic has caused more than 6 million deaths and half a billion cases globally (1). The standard of care (SOC) for patients hospitalized with COVID-19 has evolved rapidly during the pandemic and includes changes in oxygenation practices; airway management; use of prone positioning; anticoagulation practices; and use of antivirals, corticosteroids, and other immunomodulators (2–7). Some of these interventions were implemented after efficacy was determined by large clinical trials, whereas others were based on results of observational cohort studies or extrapolation from other disease states. These interventions have affected the morbidity and mortality of patients with COVID-19, but it is difficult to quantify their cumulative effect as the pandemic progresses.

In the United States, the in-hospital mortality rate for patients with COVID-19 peaked in March and April 2020, then decreased from approximately 20% to 10% by June 2020 (8). Recovery rates also improved during this period (9, 10). In-hospital mortality is associated with patient characteristics and hospital factors (10, 11), but adjustment for these factors explains only some of the decrease in the mortality rate over time (10–13). Trends in COVID-19 SOC, including medication use and oxygen supplementation, may further explain the reduction in mortality rates (9).

We analyzed clinical outcome data from 3 sequential cohorts of hospitalized patients in the first 3 stages of ACTT (Adaptive COVID-19 Treatment Trial), which was conducted in multiple countries, with 86% of the participants from the United States (Supplement Table 1) (14–17). The first 3 stages of ACTT each included a remdesivir group; it was the treatment group in ACTT-1 and the control group in ACTT-2 and ACTT-3 (Figure 1). Instead of comparing treatment groups within each stage, we analyzed the 3 remdesivir-only groups from these 3 stages. We compared recovery and mortality between remdesivir cohorts, using trial stage as a proxy for SOC given during that period. Propensity scores were used to balance cohorts on baseline characteristics.

• Download figure
• Download PowerPoint

ACTT-4 did not include a remdesivir monotherapy group. A secondary analysis applied the same methods to compare outcomes between participants in the remdesivir-plus-dexamethasone group in ACTT-4 versus remdesivir recipients in ACTT-3 who also received dexamethasone.

Methods

Data

The ACTT trials were sequential, double-blind, randomized, placebo-controlled trials that evaluated novel investigational therapeutics for the treatment of adults hospitalized with COVID-19. Figure 1 shows enrollment and follow-up periods for ACTT-1, ACTT-2, and ACTT-3, whose consecutive remdesivir groups make up our primary analysis population. These stages enrolled approximately 500 people per group (Supplement Figure 1) and completed follow-up before COVID-19 vaccination began. ACTT-4 compared remdesivir plus baricitinib to remdesivir plus dexamethasone and was included in a secondary analysis (Supplement Figure 2). The ACTT protocol is available at Annals.org.

Data collected on demographic characteristics, laboratory parameters, baseline disease severity, comorbidities, and concomitant medication use were similar for the 4 stages of ACTT, with additional data added as understanding of COVID-19 evolved and based on the interventions studied in each stage. For example, beginning with ACTT-2, additional data were collected on baseline risk factors, such as history of deep venous thrombosis, pulmonary embolism, and coagulopathy, and more specific data were collected on dexamethasone use for COVID-19 before enrollment. Clinical assessments were performed daily from day 1 through day 29 while patients were hospitalized and at follow-up visits on days 15, 22, and 29 for those who were discharged from the hospital. Disease severity was measured with an 8-category ordinal scale (Supplement Table 2) and the National Early Warning Score (NEWS) (Supplement Table 3) (18). Categories of ordinal scale for hospitalized patients are based on oxygen delivery method (Figure 2), and a person's value on the scale is their “ordinal score” (OS). Our outcomes are 28-day mortality and 28-day recovery, defined as the day of discharge or the first day of continued hospitalization without a requirement for supplemental oxygen or medical care.

• Download figure
• Download PowerPoint

Statistical Analysis

Our primary analysis included participants assigned to receive remdesivir plus SOC in ACTT-1, ACTT-2, and ACTT-3 (Figure 1). We used Cox regression to estimate recovery and mortality rates, with trial stage as a categorical predictor variable representing SOC given during that stage. Because we analyzed only remdesivir group participants, the data are effectively observational, as the randomization does not relate to our scientific question. Therefore, we analyzed the “as-treated” population and used propensity score weighting to balance baseline characteristics between cohorts. We followed STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) reporting guidelines for observational cohort studies (19). Although ACTT-4 did not include a remdesivir monotherapy group, a secondary analysis compared outcomes between those in the remdesivir-plus-dexamethasone group in ACTT-4 with the 77% of remdesivir recipients in ACTT-3 who also received dexamethasone as part of SOC.

The platform trial was designed to keep inclusion criteria relatively constant across stages, but exclusion criteria could be modified for study product safety considerations, including changes to laboratory thresholds and concomitant medication use. Modifications to stage-specific exclusion criteria were minor, and the proportions of patients who were excluded due to stage-specific exclusion criteria were 1.9% for ACTT-1, 1.6% for ACTT-2, 1.5% for ACTT-3, and 2.7% for ACTT-4 (Supplement Tables 4 and 5). Modifications to inclusion criteria were minor except for those relating to baseline OS (Supplement Table 5): ACTT-3 excluded patients with a baseline OS of 7 and also excluded those with a baseline OS of 6 after a mid-trial review by a data safety monitoring board (Figure 2). ACTT-4 included only participants with a baseline OS of 5 or 6. Causal inference with propensity scores requires all participants to have a nonzero probability of falling into any cohort (20). To ensure this, we omitted participants with a baseline OS of 7 when comparing ACTT-2 with ACTT-3 and included only those with an OS of 5 or 6 when comparing ACTT-3 with ACTT-4. We also excluded participants with chronic liver disease (an exclusion criterion for ACTT-3) from comparisons involving ACTT-3. Sensitivity analyses were planned to enforce identical laboratory exclusion criteria across stages (Supplement Table 5) and to include only sites participating in both stages for a given comparison.

Covariates were selected for inclusion in the propensity score model on the basis of clinical judgment and are summarized in Table 1. As measures of baseline disease severity, we included the NEWS and a modified 4C Mortality Score, a validated score that ranges from 0 to 21 and is based on age, sex, number of comorbidities, respiratory rate, peripheral oxygen saturation, level of consciousness, urea level, and C-reactive protein (CRP) level (Supplement Tables 3 and 6) (21–30). Because urea level was not recorded systematically in ACTT, our modified scale omits the urea point contributions (+1 for urea level of 7 to 14 mmol/L and +3 for urea level >14 mmol/L), but we included estimated glomerular filtration rate (eGFR) as a separate variable in the propensity score model to account for differences in kidney function. Although most covariates in the propensity score model were missing few data (0% to 2% of participants [Supplement Table 7]), CRP level was missing for 10% of participants (21% in ACTT-1, 3% in ACTT-2, 3% in ACTT-3, and 2% in ACTT-4). Multiple imputation was performed for missing CRP values. The Supplement provides further details on the statistical analysis.

Evolving clinical understanding and external epidemiologic data suggest that intubation practices became more conservative between ACTT-1 and ACTT-2 (31, 32), but precise intubation protocols were not documented for ACTT sites. To estimate a change in intubation practice between ACTT-1 and ACTT-2, we fit a logistic regression model with baseline intubation as the outcome and trial stage as the predictor. The analysis included only patients with a baseline OS of 6 or 7 because we believe that few participants with an OS of 4 or 5 in ACTT-2 would have been intubated at baseline if they had instead contracted COVID-19 during ACTT-1 (Supplement Figure 3). Propensity scores were used to adjust for baseline characteristics. We modeled baseline intubations rather than time to intubation to prevent confounding from temporal improvements in SOC causing fewer patients to progress to severe illness in ACTT-2 than in ACTT-1. A small amount of confounding could have been introduced by care received between hospitalization and enrollment, so a sensitivity analysis repeated the model for only those enrolled within 2 days of hospitalization.

Institutional Review Board Approval

The trial protocol was approved by the institutional review board at each site or by a centralized institutional review board as appropriate.

Role of the Funding Source

The ACTT protocols were designed and written by the ACTT investigators and the study sponsor (National Institute of Allergy and Infectious Diseases [NIAID]), with input from the manufacturers of remdesivir (Gilead), baricitinib (Eli Lilly), and interferon-β1a (EMD Serono). Principal investigators and site staff gathered the data, which were analyzed by statisticians at the statistical and data coordinating center (The Emmes Company) and NIAID. The funder (NIAID) participated in the writing of the manuscript and the decision to submit the manuscript for publication.

Results

Figure 1 shows time trends in use of concomitant medications by remdesivir recipients in ACTT. Enrollment and follow-up periods are distinguished visually because periods with no new enrollments include more participants later in their disease trajectory who tended to have less severe disease. Hydroxychloroquine was used by about 40% of participants each month during ACTT-1, but very little was used in ACTT-2 and ACTT-3. Use of antibiotics in general and azithromycin in particular decreased across the 3 stages. Steroids were not recommended initially because of concerns about worsening viral replication from prior studies of severe acute respiratory syndrome (33). Use of corticosteroids, which were allowed for indications other than COVID-19, was about 20% in ACTT-1 and ACTT-2 and surged to about 70% in ACTT-3. Anticoagulant use (both prophylactic and therapeutic) was high during all 3 stages. Use of antivirals other than remdesivir was generally under 10% and decreased over time.

Table 1 shows unweighted and propensity score–weighted statistical summaries for remdesivir recipients analyzed in ACTT-1 and ACTT-2. Several variables, including age, race, OS distribution, modified 4C Mortality Score, and interval between symptom onset and enrollment, suggest that the ACTT-2 cohort was at lower risk for poor outcomes than the ACTT-1 cohort. ACTT-2 had substantially more Hispanic patients, whose risk profile may differ from that in non-Hispanic patients (34), than ACTT-1 (52% vs. 26%). A “Love plot” (Supplement Figure 4) (35) shows excellent balance after weighting. Figure 3 shows propensity score–weighted and unweighted survival curves with hazard ratio (HR) estimates and CIs. The unadjusted recovery rate for ACTT-2 participants was 1.23 times higher than for ACTT-1 participants (95% CI, 1.06 to 1.40). However, the adjusted HR was close to 1 (1.04 [CI, 0.92 to 1.17]), indicating that remdesivir recipients with comparable characteristics had similar recovery rates between ACTT-1 and ACTT-2. Similarly, although the unweighted 28-day mortality rate was estimated to be 0.69 times lower in ACTT-2 than ACTT-1 (CI, 0.45 to 1.04), the weighted HR was closer to the null value of 1 (0.90 [CI, 0.56 to 1.40]).

• Download figure
• Download PowerPoint

Figure 2 summarizes oxygen delivery systems by day and trial stage for remdesivir participants. For those with a baseline OS of 6 or 7, the unadjusted odds of intubation in ACTT-2 were 0.31 times lower than in ACTT-1 (CI, 0.20 to 0.48). The adjusted odds ratio was 0.75 (CI, 0.53 to 0.97), suggesting a change in practice between stages. Sensitivity analysis results were similar (Supplement Figure 5 and Supplement Table 9).

Table 2 shows unweighted and weighted statistical summaries for remdesivir recipients in ACTT-2 and ACTT-3. Although several variables (including race, ethnicity, OS, NEWS, and CRP level) suggest that the ACTT-3 population was at lower risk for poor outcomes, the ACTT-3 population was older (mean age, 59 vs. 55 years). The 2 cohorts were well balanced after weighting (Supplement Figure 6). Figure 4 shows survival curves with HR estimates. The adjusted HR for recovery indicates that the recovery rate was 1.43 times higher during the ACTT-3 period than the ACTT-2 period for people with comparable characteristics (CI, 1.24 to 1.64). The 28-day mortality rate in ACTT-3 was 0.45 times that in ACTT-2 for people with comparable characteristics (CI, 0.21 to 0.97).

• Download figure
• Download PowerPoint

Supplement Figure 7 and Supplement Table 10 show results from 3 preplanned sensitivity analyses that 1) applied the same laboratory-based eligibility criteria across study stages, 2) included only sites that enrolled participants in both stages being compared, and 3) analyzed only patients with complete baseline data. Results of these analyses are similar to the main results.

Outcomes for recipients of remdesivir plus dexamethasone between ACTT-3 and ACTT-4 were similar (Supplement Figure 8). Recovery rates were nearly identical between stages for this group: The unweighted HR was 0.97 (CI, 0.84 to 1.11), and the weighted HR was 1.02 (CI, 0.89 to 1.19). The mortality rate was higher in ACTT-4 than in ACTT-3 (weighted HR, 1.75 [CI, 0.84 to 3.78]), but the CI was wide and these findings did not provide strong evidence of a difference in mortality rates between stages.

Discussion

This study compared 28-day recovery and mortality between comparable cohorts of hospitalized adults with COVID-19 who participated in 4 sequential stages of ACTT spanning February 2020 to May 2021. Although our unadjusted HR estimates describe differences in outcomes between trial stages, the propensity score–weighted HRs account for changes in patient composition over time and represent a “stage effect” attributed to the SOC received during different stages.

Remdesivir recipients in ACTT-2 (spanning May to July 2020) recovered faster and had numerically better mortality outcomes than those in ACTT-1 (February to May 2020). Observed SOC changes included a dramatic decrease in hydroxychloroquine use between ACTT-1 and ACTT-2 and a gradual decrease in empirical antibiotic use. We also found that the odds of baseline intubation in ACTT-2 were 25% lower than for comparable ACTT-1 participants. However, we did not find evidence that these changes affected 28-day recovery or mortality: Our adjusted HR estimates were close to 1, indicating that the better outcomes in ACTT-2 were due to differences in patient composition rather than improved SOC.

When comparing ACTT-3 (August to December 2020) with ACTT-2, the adjusted analyses showed improved 28-day recovery and mortality, suggesting improved SOC. Given that 77% of remdesivir recipients in ACTT-3 but only 11% in ACTT-2 received dexamethasone as SOC and the RECOVERY trial found a mortality benefit from dexamethasone, dexamethasone use is likely a key contributor to these improvements, although this observational analysis cannot confirm causality (36, 37). Another change in SOC between these stages was a gradual decrease in the use of antibiotics (including azithromycin). The large mortality reduction of 0.45 must be interpreted in the context of its wide CI (0.21 to 0.97), which is consistent with more moderate reductions. A detailed examination suggests low risk of bias from differences in stage-specific laboratory eligibility criteria (Supplement Table 11).

We did not find evidence for improved outcomes between ACTT-3 and ACTT-4 (December 2020 to May 2021) among people receiving remdesivir plus dexamethasone. This may be because concomitant medication use stayed fairly constant between these periods. The use of combination immunomodulatory therapy (dexamethasone plus a Janus kinase inhibitor or an interleukin-6 inhibitor) in patients with the most severe disease did not become part of SOC until March 2021 with the COV-BARRIER and RECOVERY baricitinib and tocilizumab trials (6, 7). Hospitalization rates were higher during ACTT-4 (Supplement Figure 2), and COVID-19 surges can stress the hospital system, increasing mortality (38), which could explain the numerically worse mortality outcomes in this stage. The appearance of the B.1.1.7 (Alpha) variant during ACTT-4 could also explain part of this mortality difference (39) (Supplement Figure 9). ACTT-4 spanned the initial availability of vaccines, but only 25 (0.5%) analyzed ACTT-4 participants were vaccinated, so bias from vaccination is probably low.

This study illustrates several issues related to the inclusion of nonconcurrent controls in analysis of data from platform trials (40–42). Even with similar eligibility criteria across stages, participant composition changed enough over time to substantially affect clinical outcomes. Early in a pandemic of a novel pathogen, patients may be more hesitant to present to a hospital, so those who do may tend to have more severe disease. Similarly, patients may be more hesitant to enroll in a trial of a novel therapeutic unless they are experiencing severe disease. We also found that SOC changed enough over time to substantially affect clinical outcomes for comparable patients.

These analyses also highlight issues related to outcome definitions. Although time to intubation and time to intubation or death have been used as outcome measures in clinical trials (43, 44), we found that intubation practices became more conservative over time. This could make outcomes appear worse for nonconcurrent controls, thus exaggerating the treatment effect. When contemporaneous controls are used, the temporal shift in the relationship between the outcome and the underlying disease severity affects treated and control participants simultaneously and equally.

Time to recovery is also not a purely objective outcome. ACTT participants were defined as having “recovered” when they were discharged from the hospital or if they remained hospitalized without requiring supplemental oxygen or ongoing medical care. This definition is a proxy for a certain point in the underlying, unobserved, actual disease trajectory. The relationship between this proxy and the actual disease trajectory can vary. As the pandemic progressed, clinicians may have discharged patients earlier in their recovery trajectory, which could artificially inflate a treatment effect in a comparison with nonconcurrent controls. It could also explain part of the improvement in recovery outcomes between ACTT-2 and ACTT-3. Mortality is potentially more objective, although this can depend on varying protocols for withdrawal of care. Furthermore, low death rates mean that very large trials are needed for adequate power to detect a mortality reduction. In addition, treatments that reduce objectively measured symptoms are beneficial even if they do not affect mortality, and objective outcomes are needed to test such treatments.

This study has limitations. First, valid propensity score inference requires inclusion of all confounders in the analysis. Although we included many important confounders, some potential confounders were not measured. In particular, urea level was not recorded in ACTT, and this variable contributes either 1 or 3 points to the 21-point 4C Mortality Score. Our modified scale omitting urea level is therefore less predictive than the complete scale, although we included eGFR as a separate variable to account for differences in kidney function. D-dimer, interleukin-6, and interleukin-10 were not collected across ACTT stages. Another limitation is the restriction of mortality comparisons to a 28-day interval. It is possible that SOC changes between ACTT-2 and ACTT-3 delayed deaths past the 28-day time point without reducing the overall in-hospital mortality rate. However, more people in ACTT-2 were still intubated on day 28 than in ACTT-3 (4% vs. 1% weighted [Supplement Table 12]), making a reversal of the mortality HR after day 28 unlikely. Improved outcomes may have been influenced by unmeasured time-dependent factors, such as greater clinician bedside experience, although this may be more applicable to the comparison of ACTT-1 versus ACTT-2. COVID-19 surges can increase mortality, but hospitalization rates were higher in ACTT-3 than ACTT-2, which would tend to attenuate the estimated mortality improvement from SOC (average weekly rates were 8.5 and 6.3 per 100 000 persons in ACTT-3 and ACTT-2, respectively [Figure 1]). Circulating variants were generally unidentified during 2020 (Supplement Figure 9), and their effect on mortality was probably small. This trial mostly enrolled patients at academic research sites; results are generalizable to similar hospitals whose populations resemble the ACTT patient population.

This study analyzed time trends of mortality and recovery for comparable cohorts of hospitalized COVID-19 clinical trial participants and described evolving SOC practices. We found that intubation practice became more conservative between the period from February to May 2020 and the period from May to July 2020 and that improvements in recovery and mortality between these intervals were explained by differences in cohort composition. This contrasts with other studies examining U.S. in-hospital mortality, which found that mortality improvements persisted after patient characteristics were accounted for (10–13). Our mortality rates were lower: The unadjusted rate decreased from 11% to 7%, whereas other studies found a decrease from approximately 20% to 10% (10–13). The difference could be because the hospitals participating in ACTT may have more resources and experience and/or a different learning curve than other hospitals, followed protocol-specified care requirements, or drew a different patient population. We found improved recovery and mortality outcomes between the period from May to July 2020 and the period from August to December 2020, likely due to increased dexamethasone use. We did not find evidence for improvements in recovery or mortality between the period from August to December 2020 and the period from December 2020 to May 2021, possibly because SOC did not change substantially (the use of combination immunomodulatory treatment was implemented later). These findings support the exclusion of nonconcurrent controls when analyzing data from platform trials, particularly for COVID-19 treatments and vaccines.