Effect of Clinical Decision-Support Systems
FREEAbstract
Background:
Despite increasing emphasis on the role of clinical decision-support systems (CDSSs) for improving care and reducing costs, evidence to support widespread use is lacking.
Purpose:
To evaluate the effect of CDSSs on clinical outcomes, health care processes, workload and efficiency, patient satisfaction, cost, and provider use and implementation.
Data Sources:
MEDLINE, CINAHL, PsycINFO, and Web of Science through January 2011.
Study Selection:
Investigators independently screened reports to identify randomized trials published in English of electronic CDSSs that were implemented in clinical settings; used by providers to aid decision making at the point of care; and reported clinical, health care process, workload, relationship-centered, economic, or provider use outcomes.
Data Extraction:
Investigators extracted data about study design, participant characteristics, interventions, outcomes, and quality.
Data Synthesis:
148 randomized, controlled trials were included. A total of 128 (86%) assessed health care process measures, 29 (20%) assessed clinical outcomes, and 22 (15%) measured costs. Both commercially and locally developed CDSSs improved health care process measures related to performing preventive services (n = 25; odds ratio [OR], 1.42 [95% CI, 1.27 to 1.58]), ordering clinical studies (n = 20; OR, 1.72 [CI, 1.47 to 2.00]), and prescribing therapies (n = 46; OR, 1.57 [CI, 1.35 to 1.82]). Few studies measured potential unintended consequences or adverse effects.
Limitations:
Studies were heterogeneous in interventions, populations, settings, and outcomes. Publication bias and selective reporting cannot be excluded.
Conclusion:
Both commercially and locally developed CDSSs are effective at improving health care process measures across diverse settings, but evidence for clinical, economic, workload, and efficiency outcomes remains sparse. This review expands knowledge in the field by demonstrating the benefits of CDSSs outside of experienced academic centers.
Primary Funding Source:
Agency for Healthcare Research and Quality.
Despite increasing emphasis on clinical decision-support systems (CDSSs) in improving care and reducing costs, evidence supporting widespread use is limited. A CDSS is “any electronic system designed to aid directly in clinical decision making, in which characteristics of individual patients are used to generate patient-specific assessments or recommendations that are then presented to clinicians for consideration” (1). This review examines 3 types of decision-support interventions.
Classic CDSSs include alerts, reminders, order sets, drug-dose calculations that automatically remind the clinician of a specific action, or care summary dashboards that provide performance feedback on quality indicators. Information retrieval tools, such as an “infobutton” embedded in a clinical information system, are designed to aid clinicians in the search and retrieval of context-specific knowledge from information sources based on patient-specific information from a clinical information system. Knowledge resources, such as UpToDate, Epocrates, and MD Consult, consist of distilled primary literature that allows selection of content germane to a specific patient to facilitate decision making at the point of care or for a specific care situation.
Until recently, most studies of CDSSs came from 4 benchmark settings (Brigham and Women's Hospital/Partners HealthCare, the Department of Veterans Affairs, LDS Hospital/Intermountain Healthcare, and the Regenstrief Institute) (2). Although several reviews have examined the effects of CDSSs (1, 3–9), many questions remain about their impact. This systematic review adds to the literature by summarizing trials of CDSSs implemented in a clinical setting to aid decision making at the point of care or for a specific care situation.
Methods
We developed and followed a standard protocol for our review. Full details of our methods, search strategies, results, and conclusions are provided in a report commissioned by the Agency for Healthcare Research and Quality, available at www.effectivehealthcare.ahrq.gov.
Data Sources and Searches
We searched for studies done between January 1976 and January 2011 in MEDLINE accessed through PubMed, CINAHL, PsycINFO, and Web of Science.
Study Selection
We identified randomized trials of CDSSs implemented in a real clinical setting and used by health care providers to aid decision making at the point of care or for a specific care situation. Studies had to report at least one of the following types of outcomes: clinical (length of stay, morbidity, mortality, health-related quality of life, and adverse events), health care process (recommended preventive care, clinical study, or treatment ordered or completed), user workload and efficiency (user knowledge, number of patients seen, clinician workload, and efficiency), relationship-centered (patient satisfaction), economic (cost and cost-effectiveness), or use and implementation by a health care provider (acceptance, satisfaction, use, and implementation). We excluded studies that described nonelectronic CDSSs, included fewer than 50 participants, were not published in English, described closed-loop systems that did not involve a provider, evaluated systems that required mandatory compliance with the CDSS, or evaluated only the performance of the system as opposed to its effect on clinical practice.
Data Extraction and Quality Assessment
Data related to study setting and design, sample characteristics, intervention characteristics, comparators, and outcomes were extracted by 1 reviewer and confirmed by another. Two reviewers used a standardized approach to independently categorize the quality of individual studies as good, fair, or poor (10) and evaluated the overall strength of evidence for each outcome as high, moderate, low, or insufficient (11). Reviewers also identified issues related to study setting, interventions, and outcomes that limited applicability of evidence (10, 12).
Data Synthesis and Analysis
A priori–defined outcomes believed to be important in measuring the effect of CDSSs in improving clinical practice guided our synthesis process. Studies with a common outcome were grouped together to facilitate qualitative analysis. Quantitative analysis was done where 4 or more studies assessed the same outcome in the same manner, regardless of the specific CDSS intervention. Summary estimates were calculated by using the DerSimonian and Laird (13) random-effects model as implemented in Comprehensive Meta-analysis, version 2.2.055 (Biostat, Englewood, New Jersey).
Role of the Funding Source
Primary funding was provided by the Agency for Healthcare Research and Quality. The funding source formulated the initial study questions but otherwise had no role in the design, analysis, or interpretation of the data or in the decision to submit the manuscript for publication.
Results
We screened 15 176 abstracts, evaluated 1407 full-text articles, and included 160 articles, representing 148 unique studies (Appendix Figure). Appendix Table 1 summarizes important characteristics of the studies and their quality rating. A total of 128 studies (86%) assessed health care process measures, 29 (20%) assessed clinical outcomes, and 22 (15%) measured costs. Many studies (n = 51) were performed in environments with established health information technology (IT); many were multisite studies involving multiple institutions (n = 46).
CDSS = clinical decision-support system; KQ = key question; RCT = randomized, controlled trial.
 |
The Table summarizes the most important findings and the strength of supporting evidence for those findings, whereas Appendix Table 2 provides examples of interventions that assessed the outcomes of interest. Both commercially and locally developed CDSSs improved health care process measures related to performing preventive services, ordering clinical studies, and prescribing therapies. Few studies (n = 15) indicated conceptualization of potential unintended consequences or measured potential adverse effects of implementing decision-support tools.
 |
 |
Clinical Outcomes
Morbidity
Several studies assessed morbidity outcomes (14–38), such as hospitalizations, Apgar scores, surgical site infections, cardiovascular events, colorectal cancer, deep venous thrombosis, and hypoglycemia events. Topics addressed included diagnosis (16, 21, 29, 30, 34), pharmacotherapy (15, 18–22, 25, 28, 33), chronic disease management (14, 19, 20, 22, 26–28, 31, 32, 36, 38), laboratory test ordering (19, 37), immunizations (19, 35), preventive care (17, 19, 28–30, 37), and discharge planning (23, 24). Approximately 50% of the studies were performed in an academic setting.
Many studies evaluated locally developed interventions implemented in the ambulatory environment. Typical interventions were automatically delivered, system-initiated recommendations provided synchronously at the point of care to enable decision making during the health care provider–patient encounter. Three such interventions required a mandatory response (that is, required that the user respond to the given recommendation, whether that response was to accept or dismiss the recommendation or to modify the user's action) (17, 18, 37).
Comparators included usual care or no CDSS and the same CDSS with additional features. Limitations included short follow-up, low statistical power to detect important differences, and the potential for contamination of providers in control groups that improved because of knowledge of interventions.
Meta-analysis of these heterogeneous studies (n = 16) suggested that CDSSs improved morbidity outcomes (relative risk, 0.88 [95% CI, 0.80 to 0.96]). We rated this level of evidence as moderate. Most studies were good quality, and many of the interventions were evaluated in multiple institutions. However, the interventions were often paper-based or standalone systems implemented in academic or Veterans Affairs settings and were targeted toward a single condition.
Mortality
Seven studies (17, 20–22, 33, 39, 40) reported mortality outcomes. Issues addressed included diagnosis (21), pharmacotherapy (20–22, 33, 40), chronic disease management (20, 22), preventing deep venous thrombosis (17), and detecting and notifying clinicians of critical laboratory values (39). Most CDSSs were locally developed and integrated into a computerized physician order entry (CPOE) or electronic health record (EHR) system and had system-initiated recommendations delivered synchronously at the point of care that did not require a clinician response.
Interventions were evaluated against usual care or no CDSS, except for 2 studies (20, 22) that compared the same intervention with additional features. Limitations included small sample size, duration shorter than 1 year, and possible contamination of control providers.
Meta-analysis of these heterogeneous studies (n = 6) suggested no significant effect of CDSSs on mortality, although CIs were wide (odds ratio [OR], 0.79 [CI, 0.54 to 1.15]). Of note, 2 studies reported a significant reduction in mortality with use of CDSSs (20, 22). We rated the overall strength of evidence related to mortality outcomes as low. Most studies were conducted in a single academic or Veterans Affairs setting with a comprehensive, well-established health IT infrastructure.
Adverse Events
Five studies assessed the effectiveness of CDSSs in reducing or preventing adverse events (23, 24, 39–42). Studies were mostly implemented in an academic, inpatient setting. Studies evaluated the effect of these interventions to improve the timing of warfarin therapy (41), improve discharge planning (23, 24), prevent adverse drug events (42), detect critical laboratory values (39), and detect potentially inappropriate or inadequate antimicrobial therapy (40).
Typical interventions were locally developed, were integrated into a CPOE or an EHR system, and automatically delivered system-imitated recommendations in real time to enable decision making during the provider–patient encounter. Only 1 study clearly required a mandatory response (39).
All of the CDSSs were evaluated against usual care or no CDSS. Limitations included evaluation at a single institution, evaluation periods less than 1 year, and potential improvement in physician performance because of their knowledge of the intervention.
Meta-analysis of these heterogeneous studies estimated a relative risk of 1.01 (CI, 0.90 to 1.14), and neither this summary nor any individual studies demonstrated a significant effect. We rated this level of evidence as low. Most studies were good quality, and 2 were conducted at multiple sites; however, these interventions primarily contained locally developed knowledge, and results may not be generalizable to nonteaching settings.
Health Care Process Measures
Recommended Preventive Care Service Ordered or Completed
Forty-three studies examined the effect of CDSSs on the rates of ordering or completing recommended preventive care services (17, 19, 27, 28, 35, 37, 43–82). Most studies were conducted in the academic or ambulatory environment. Topics addressed included diagnosis (43, 47, 62, 71, 82), pharmacotherapy (19, 28, 49, 53, 54, 70, 77, 80), chronic disease management (19, 26, 28, 43, 44, 48, 51, 70, 72, 74, 76), laboratory test ordering (19, 37, 55, 60, 70, 71, 74, 75, 80, 81), preventive care (17, 19, 28, 37, 43, 45, 48, 52–60, 62–65, 68–71, 73–75, 79–82), immunizations (19, 35, 48–50, 52, 56, 66, 67, 78, 79, 81), and initiating discussions with patients (60, 61, 72).
Most interventions were locally developed, paper-based, or standalone systems and automatically delivered recommendations in real time to enable decision making during the health care provider–patient encounter. Only 7 of the interventions required a mandatory response (17, 37, 47, 49) or justification (64, 65, 68, 69, 75) for not adhering to the recommendation.
Comparators included usual care or no CDSS, direct comparison against the same CDSS with additional features, or comparison of the same CDSS for different conditions. Limitations included sparse data measuring patient or economic outcomes; few assessments of long-term outcomes of interventions; and the Hawthorne effect, which probably stimulated more comprehensive preventive care across groups.
In meta-analysis of these 25 heterogeneous studies (17, 27, 28, 35, 37, 43–63), the effect of CDSSs on preventive care services was significant (OR, 1.42 [CI, 1.27 to 1.58]) (Figure 1). We rated this level of evidence as high. Approximately one half of the studies were good quality, one third were evaluated in multicenter trials, and one fourth addressed multiple clinical conditions. However, most CDSSs were locally developed, not integrated into a CPOE or an EHR, and evaluated in academic medical centers, all of which can affect the generalizability of these findings.
Studies reporting the odds ratio of adhering to recommendations for ordering or completing preventive care services of CDSS vs. control groups. In the 25 studies comparing CDSS with control groups, the random-effects–combined odds ratio of adherence to preventive care recommendations was 1.42 (95% CI, 1.27 to 1.58). CDSS = clinical decision-support system.
Recommended Clinical Study Ordered or Completed
Twenty-nine studies evaluated the effect of CDSSs on ordering and completing recommended clinical studies (26, 31, 32, 83–109). Many studies were conducted in the academic setting, and most were evaluated in the ambulatory environment. Topics addressed included diagnosis (85, 89–91, 95, 98, 101–103), pharmacotherapy (94, 98), chronic disease management (26, 31, 32, 84, 85, 103), laboratory test ordering (83, 87, 89, 92–94, 96, 97, 99–101, 104–108), initiating discussions with patients (108, 109), and additional clinical tasks (86, 88, 90, 109).
Typical interventions were locally developed, were integrated CDSS recommendations in a CPOE or an EHR system, and automatically delivered system-initiated recommendations to enable decision making during the provider–patient encounter. Eight interventions required a mandatory response (90, 99, 100, 104, 106, 108) or justification (83, 105) for not adhering to the recommendation.
Comparators included usual care or no CDSS, direct comparison against the same CDSS with additional features, or comparison of the same CDSS for different conditions. Limitations of the evidence base included diverse metrics to assess adherence to ordering and completing a recommended action, studies not designed to evaluate the clinical or economic outcomes associated with the CDSS interventions, and limited evidence of the effect of CDSSs on a broad set of conditions.
Meta-analysis of these 20 heterogeneous studies (26, 31, 32, 83, 84, 88, 89, 91, 92, 94–96, 98–103, 105, 108, 109) found a significant effect of CDSSs on ordering or completing of clinical studies (OR, 1.72 [CI, 1.47 to 2.00]) (Figure 2). We rated this level of evidence as moderate. Most studies were good quality, approximately one third were implemented in multiple sites, and almost one fourth included a direct comparison of the effectiveness of the CDSS against the same intervention with additional features. However, we noted a strong suggestion of publication bias in these studies, and these results may not be generalizable to all settings because most studies were either evaluated in environments with a well-established health IT infrastructure or conducted outside of the United States.
Studies reporting the odds ratio of adhering to recommendations for ordering or completing recommended clinical studies of CDSS vs. control groups. In the 20 studies comparing CDSS with control groups, the random-effects–combined odds ratio of adherence to clinical study recommendations was 1.72 (95% CI, 1.47 to 2.00). CDSS = clinical decision-support system.
Recommended Treatment Ordered or Prescribed
Sixty-seven studies evaluated the effect of CDSSs on ordering and prescribing therapy (14, 18, 20–22, 25–28, 33, 36, 38–41, 43, 44, 53, 54, 70, 80, 82, 84, 88, 94, 98, 110–154). Many studies were conducted in the academic setting, and most were evaluated in the ambulatory environment. Topics addressed included diagnosis (21, 43, 82, 98, 112, 123, 129, 138, 151, 152), pharmacotherapy (18, 20–22, 25, 28, 33, 40, 53, 54, 70, 80, 94, 98, 110, 111, 114, 116, 117, 119, 122, 123, 125–134, 136, 138–150, 153, 154), laboratory test ordering (70, 80, 94, 110, 130), chronic disease management (14, 20, 22, 26–28, 36, 38, 43, 44, 70, 84, 110, 112, 113, 115, 118, 120, 121, 123–125, 132, 133, 135, 141), preventive care (28, 43, 53, 54, 70, 80, 82, 120, 148), and additional clinical tasks (39, 41, 82, 88, 110, 137, 151, 152).
Typical interventions were locally developed, were integrated into a CPOE or an EHR system, and automatically delivered system-initiated recommendations in real time to enable decision making during the provider–patient encounter. Eighteen CDSSs required a mandatory response (18, 39, 119, 122, 134, 139–143, 147, 148, 151, 152) or justification (112, 113, 136, 137, 145) for not adhering to the recommendation. Limitations included inadequate follow-up periods to observe sustained results, sparse data demonstrating how changes in clinician ordering and prescribing led to improvements in clinical or economic outcomes, and study designs that did not capture the extent to which nonadherence with recommended therapy resulted in adverse events.
Meta-analysis of 46 heterogeneous studies (14, 18, 20–22, 25–28, 33, 36, 38, 40, 43, 44, 53, 54, 70, 82, 84, 94, 98, 110, 112–117, 121–124, 129, 130, 134–136, 141–144, 146, 148, 151–153) showed that intervention providers with decision support were more likely to order the appropriate treatment or therapy (OR, 1.57 [CI, 1.35 to 1.82]) (Figure 3). We rated this level of evidence as high. Most studies were good quality, and most were evaluated in multisite trials. However, generalizability may be limited because most studies were implemented in the ambulatory environment, were evaluated in settings where clinicians were experienced EHR users or provided care in an established health IT infrastructure, and incorporated knowledge that was targeted toward specific conditions.
Studies reporting the odds ratio of adhering to recommendations for ordering or prescribing treatment of CDSS vs. control groups. In the 46 studies comparing CDSS with control groups, the random-effects–combined odds ratio of adherence to treatment recommendations was 1.57 (95% CI, 1.35 to 1.82). CDSS = clinical decision-support system.
User Workload and Efficiency Outcomes
Evidence on the effect of CDSSs on clinician knowledge or improved confidence in managing patient care was insufficient (71, 72, 86, 155, 156). Seven studies examined the effect of CDSSs on efficiency (23, 24, 40, 106, 141, 155–157). Limitations included contamination of clinicians in the control group that improved because of knowledge of the intervention, evaluation periods that were too brief to demonstrate an effect on efficiency, and small clinician sample sizes.
We rated the level of evidence as low. Most interventions contained locally developed knowledge, such as protocols or algorithms derived on the basis of local performance, quality, and outcome data not representative of other sites, and were evaluated in academic settings.
Economic Outcomes
Cost
Twenty-two studies reported costs (21, 26, 27, 31, 32, 36, 40, 43, 53, 54, 71, 83, 90, 106, 109, 113, 118, 130, 141, 158–163). Objectives of the CDSSs included diagnosis (21, 43, 71, 90, 160), pharmacotherapy (21, 40, 53, 54, 130, 141), chronic disease management (26, 27, 31, 32, 36, 43, 113, 118, 141, 161, 162), laboratory test ordering (71, 83, 106, 130, 160, 163), preventive care (43, 53, 54, 71, 158, 159), initiating discussions with patients (109, 159), and additional clinical tasks (90, 109). One study reported reduced hospitalization expenses with CDSS use (31, 32), and 12 studies reported that use had a positive effect on costs compared with control groups and other non-CDSS groups (21, 27, 40, 53, 54, 83, 90, 106, 113, 130, 141, 159, 160).
Modest evidence from academic and community inpatient and ambulatory settings showed that locally and commercially developed CDSSs had lower treatment costs, total costs, and reduced costs compared with control groups and other non-CDSS intervention groups. Most studies were conducted in the academic ambulatory setting and evaluated locally developed, integrated CDSSs in CPOE or EHR systems that automatically delivered system-initiated recommendations synchronously at the point of care and did not require a mandatory clinician response.
Cost-Effectiveness
Six studies (53, 54, 56, 57, 78, 95, 96, 161, 162) examined the cost-effectiveness of CDSSs or their effect on cost-effectiveness of care. These demonstrated conflicting findings, with 3 studies suggesting that CDSSs were cost-effective (53, 54, 78, 95) and 3 reporting that CDSSs were not cost-effective (56, 57, 161, 162). Objectives included diagnosis (95), pharmacotherapy (53, 54), chronic disease management (161, 162), preventive care (53, 54, 56, 57), and immunizations (56, 78).
Use and Implementation Outcomes
Twenty-four studies assessed the effect of provider acceptance of CDSSs (19, 41, 55, 62, 75, 90, 105, 113, 119, 120, 136, 137, 145–147, 149, 158, 159, 164–170). Topics addressed included diagnosis (62, 90, 166), pharmacotherapy (19, 119, 136, 145–147, 149, 164, 165), chronic disease management (19, 113, 120, 168–170), laboratory test ordering (19, 55, 75, 105), preventive care (19, 55, 62, 75, 120, 147, 158, 159, 167), immunizations (19), initiating discussions with patients (159), and additional clinical tasks (41, 90, 137).
Comparators included usual care or no CDSS and direct comparison with the same CDSS with additional features. One half of the studies required a mandatory response (55, 90, 119, 147, 166) or justification (75, 105, 113, 120, 136, 137, 145) for not adhering to the recommendation; however, there was no significant effect on provider acceptance. Limitations included an inconsistent definition of provider acceptance; small sample sizes; and scarce data on clinical outcomes, such as morbidity, length of stay, or adverse events. We rated this level of evidence as low. Most of these studies were fair quality and were evaluated in academic medical settings with established health IT infrastructures and experienced EHR users, which may limit the generalizability of the findings.
Provider satisfaction with CDSSs was examined in 19 studies (14, 23–25, 37, 43, 80, 86, 105, 109, 112, 119, 127, 128, 141, 151–153, 155, 156, 158, 165). Topics addressed included diagnosis (43, 112, 151, 152), pharmacotherapy (25, 80, 119, 127, 128, 141, 153, 165), chronic disease management (14, 43, 112, 141), laboratory test ordering (37, 80, 105), preventive care (37, 43, 80), and initiating discussions with patients (109).
Comparators included usual care or no CDSS and direct comparison with the same CDSS with additional features. Seven CDSSs required a mandatory response (37, 119, 141, 151, 152, 155) or justification (105, 112) for not adhering to the recommendation. Limitations included the narrow assessment of the role of provider satisfaction with CDSSs on patient-specific outcomes and small sample sizes of clinicians.
Twelve studies demonstrated provider satisfaction with CDSSs (25, 37, 80, 109, 112, 119, 127, 128, 141, 155, 156, 158, 165); 4 showed a significant effect of satisfaction among intervention providers compared with control providers (109, 112, 155, 165). Provider dissatisfaction with CDSSs was also reported in 6 studies (14, 43, 86, 105, 151–153). We rated this level of evidence as moderate. Most studies were good quality and evaluated CDSSs integrated into CPOE or EHR systems in multiple interventions outside of environments with an established and robust health IT. However, most CDSSs were locally developed and implemented in the ambulatory setting.
Seventeen studies examined provider use of CDSSs by using such metrics as the number of times the CDSS was accessed by the clinician or provided a recommendation to the clinician (51, 71, 80, 86, 107, 110, 117, 119, 123, 138, 142, 145, 156, 165, 168–172). Objectives included diagnosis (71, 123, 138), pharmacotherapy (80, 110, 117, 119, 123, 138, 142, 145, 165), chronic disease management (51, 110, 123, 168–172), laboratory test ordering (71, 80, 107, 110), preventive care (71, 80), and additional clinical tasks (86, 110, 156).
Comparators included usual care or no CDSS, direct comparison with the same CDSS with additional features, or comparison of the same CDSS for different conditions. Limitations included sparse data demonstrating how provider use translated into more appropriate patient care and small sample sizes of clinicians. We rated this level of evidence as low.
Among the 12 studies (80, 107, 117, 119, 123, 138, 145, 168–172) that provided statistical data about provider use, 8 (80, 110, 119, 123, 145, 165, 168–170) documented low use (<50% of the clinician's time or of patient visits) or that less than 50% of clinicians used the CDSS or received alerts to guide therapeutic action. Most of these studies were fair quality and evaluated locally developed interventions in multiple community and ambulatory settings. Additional results are available from the technical report at www.effectivehealthcare.ahrq.gov.
Discussion
Our systematic review investigated the continuum of information support for clinical care, including traditional CDSSs, as well as information retrieval systems and knowledge resources developed for access at the point of care. Studies were primarily conducted outside of institutions with an established health IT infrastructure. Most interventions targeted specific medical conditions and were evaluated in single settings.
Clinical decision support had a favorable effect on prescribing treatments, facilitating preventive care services, and ordering clinical studies across diverse venues and systems. This finding contrasts with that of another review (2), which showed that most reports of successful CDSS implementation were based on locally developed systems at 4 sites.
Evidence demonstrating positive effects of CDSSs on clinical and economic outcomes remains surprisingly sparse, although this could be because of the relative difficulty of implementing randomized, controlled trials in real clinical settings, as well as the logistics of measuring the direct clinical effect of CDSSs. Evidence was also limited in showing an effect of CDSSs on clinical workload and efficiency. Furthermore, available evidence is insufficient to draw conclusions about the potential negative effect of implementing decision-support tools, which is necessary to truly fulfill the goal of evaluating these interventions and to better address implementation challenges (173).
Our findings are important in light of the increasing political interest and financial investment of the U.S. government in resources for health IT. Meaningful use of CDSSs needs to be objectively informed about the role that they can and should play in reshaping health care delivery. Further understanding is increasingly important to optimally define their role in the context of meaningful use for EHRs.
Our systematic review has several limitations. The heterogeneity of the studies limited general observations about CDSSs. We minimized this limitation in our meta-analyses by including studies that assessed the same outcome in the same manner. Although this investigation was a comprehensive review of randomized, controlled trials that provided the best evidence on CDSS effectiveness, these studies may provide less information about issues related to CDSS implementation, effect on workflow, and factors affecting usability.
Most studies (76%) evaluated the effectiveness of the intervention by using usual care rather than a direct comparator, which may contribute to more positive results. Finally, we acknowledge the possibility of selective reporting or publication bias. However, a recent review by Buntin and colleagues (173) found that 62% of included studies on health IT reported positive effects where the technology was associated with improvement in 1 or more aspects of care (173). Formal assessment by using funnel plots found no consistent bias for most outcomes, except for ordering or completing of clinical studies, where there was a strong suggestion of publication bias.
Significant research is still required to promote widespread use of CDSSs and to augment their clinical effectiveness. Future studies should investigate how to expand CDSS content to accommodate multiple comorbid conditions simultaneously and to determine which members of the care team should receive clinical decision support, what effect CDSSs have on clinical and economic outcomes, and how CDSSs can be most effectively integrated into workflow and deployed across diverse settings. Further work is also needed to understand how CDSSs can aid in the transformation of care delivery models, such as accountable care organizations and patient-centered medical homes; how to incorporate CDSSs into workflow tools, such as medical registries and provider–provider messaging capabilities; and how to integrate CDSSs with workflow-oriented quality improvement programs.
Promoting extensive use of CDSSs will require a better definition of the clinical decision-support infrastructure. Such infrastructure could include consistent underlying frameworks for describing CDSSs, such as the “Clinical Decision Support Five Rights” (174), to aid in the aggregation and synthesis of results; development and evaluation of models for porting CDSSs across settings; and improved identification of characteristics of the environment and workflow into which a CDSS is deployed, as well as characteristics of the intended users.
In summary, evidence demonstrated the efficacy of CDSSs on health care process outcomes across diverse settings by using both commercially and locally developed systems, but data showing an effect on clinical and economic outcomes were sparse. Broad penetration of clinical decision-support tools will require aggressively seeking a better understanding of what the right information is and when and how it should be delivered to the right person, and a critical examination of the unintended consequences of CDSS implementation.
References
- 1.
Kawamoto K ,Houlihan CA ,Balas EA ,Lobach DF . Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ. 2005;330:765. [PMID: 15767266] CrossrefMedlineGoogle Scholar - 2.
Chaudhry B ,Wang J ,Wu S ,Maglione M ,Mojica W ,Roth E ,et al . Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med. 2006;144:742-52. [PMID: 16702590] LinkGoogle Scholar - 3.
Bryan C ,Boren SA . The use and effectiveness of electronic clinical decision support tools in the ambulatory/primary care setting: a systematic review of the literature. Inform Prim Care. 2008;16:79-91. [PMID: 18713524] MedlineGoogle Scholar - 4.
Garg AX ,Adhikari NK ,McDonald H ,Rosas-Arellano MP ,Devereaux PJ ,Beyene J ,et al . Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA. 2005;293:1223-38. [PMID: 15755945] CrossrefMedlineGoogle Scholar - 5.
Grimshaw J ,Freemantle N ,Wallace S ,Russell I ,Hurwitz B ,Watt I ,et al . Developing and implementing clinical practice guidelines. Qual Health Care. 1995;4:55-64. [PMID: 10142039] CrossrefMedlineGoogle Scholar - 6.
Hunt DL ,Haynes RB ,Hanna SE ,Smith K . Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA. 1998;280:1339-46. [PMID: 9794315] CrossrefMedlineGoogle Scholar - 7.
Sintchenko V ,Magrabi F ,Tipper S . Are we measuring the right end-points? Variables that affect the impact of computerised decision support on patient outcomes: a systematic review. Med Inform Internet Med. 2007;32:225-40. [PMID: 17701828] CrossrefMedlineGoogle Scholar - 8.
Shojania KG ,Jennings A ,Mayhew A ,Ramsay C ,Eccles M ,Grimshaw J . Effect of point-of-care computer reminders on physician behaviour: a systematic review. Can Med Assoc J. 2010;182:216-25. CrossrefGoogle Scholar - 9.
Kaushal R ,Shojania KG ,Bates DW . Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med. 2003;163:1409-16. [PMID: 12824090] CrossrefMedlineGoogle Scholar - 10. Agency for Healthcare Research and Quality. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. Rockville, MD: Agency for Healthcare Research and Quality; 2011. Accessed at www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productid=318 on 24 January 2012. Google Scholar
- 11.
Owens DK ,Lohr KN ,Atkins D ,Treadwell JR ,Reston JT ,Bass EB ,et al . AHRQ series paper 5: grading the strength of a body of evidence when comparing medical interventions—agency for healthcare research and quality and the effective health-care program. J Clin Epidemiol. 2010;63:513-23. [PMID: 19595577] CrossrefMedlineGoogle Scholar - 12.
Atkins D ,Chang SM ,Gartlehner G ,Buckley DI ,Whitlock EP ,Berliner E ,et al . Assessing applicability when comparing medical interventions: AHRQ and the Effective Health Care Program. J Clin Epidemiol. 2011;64:1198-207. [PMID: 21463926] CrossrefMedlineGoogle Scholar - 13.
DerSimonian R ,Laird N . Meta-analysis in clinical trials. Control Clin Trials. 1986;7:177-88. [PMID: 3802833] CrossrefMedlineGoogle Scholar - 14.
McCowan C ,Neville RG ,Ricketts IW ,Warner FC ,Hoskins G ,Thomas GE . Lessons from a randomized controlled trial designed to evaluate computer decision support software to improve the management of asthma. Med Inform Internet Med. 2001;26:191-201. [PMID: 11706929] CrossrefMedlineGoogle Scholar - 15.
Cavalcanti AB ,Silva E ,Pereira AJ ,Caldeira-Filho M ,Almeida FP ,Westphal GA ,et al . A randomized controlled trial comparing a computer-assisted insulin infusion protocol with a strict and a conventional protocol for glucose control in critically ill patients. J Crit Care. 2009;24:371-8. [PMID: 19592202] CrossrefMedlineGoogle Scholar - 16.
Kline JA ,Zeitouni RA ,Hernandez-Nino J ,Jones AE . Randomized trial of computerized quantitative pretest probability in low-risk chest pain patients: effect on safety and resource use. Ann Emerg Med. 2009;53:727-35.e1. [PMID: 19135281] CrossrefMedlineGoogle Scholar - 17.
Kucher N ,Koo S ,Quiroz R ,Cooper JM ,Paterno MD ,Soukonnikov B ,et al . Electronic alerts to prevent venous thromboembolism among hospitalized patients. N Engl J Med. 2005;352:969-77. [PMID: 15758007] CrossrefMedlineGoogle Scholar - 18.
Zanetti G ,Flanagan HL ,Cohn LH ,Giardina R ,Platt R . Improvement of intraoperative antibiotic prophylaxis in prolonged cardiac surgery by automated alerts in the operating room. Infect Control Hosp Epidemiol. 2003;24:13-6. [PMID: 12558230] CrossrefMedlineGoogle Scholar - 19.
McDonald CJ ,Hui SL ,Smith DM ,Tierney WM ,Cohen SJ ,Weinberger M ,et al . Reminders to physicians from an introspective computer medical record. A two-year randomized trial. Ann Intern Med. 1984;100:130-8. [PMID: 6691639] LinkGoogle Scholar - 20.
Roumie CL ,Elasy TA ,Greevy R ,Griffin MR ,Liu X ,Stone WJ ,et al . Improving blood pressure control through provider education, provider alerts, and patient education: a cluster randomized trial. Ann Intern Med. 2006;145:165-75. [PMID: 16880458] LinkGoogle Scholar - 21.
Paul M ,Andreassen S ,Tacconelli E ,Nielsen AD ,Almanasreh N ,Frank U ,et al ;TREAT Study Group . Improving empirical antibiotic treatment using TREAT, a computerized decision support system: cluster randomized trial. J Antimicrob Chemother. 2006;58:1238-45. [PMID: 16998208] CrossrefMedlineGoogle Scholar - 22.
Ansari M ,Shlipak MG ,Heidenreich PA ,Van Ostaeyen D ,Pohl EC ,Browner WS ,et al . Improving guideline adherence: a randomized trial evaluating strategies to increase beta-blocker use in heart failure. Circulation. 2003;107:2799-804. [PMID: 12756157] CrossrefMedlineGoogle Scholar - 23.
Graumlich JF ,Novotny NL ,Nace GS ,Aldag JC . Patient and physician perceptions after software-assisted hospital discharge: cluster randomized trial. J Hosp Med. 2009;4:356-63. [PMID: 19621342] CrossrefMedlineGoogle Scholar - 24.
Graumlich JF ,Novotny NL ,Stephen Nace G ,Kaushal H ,Ibrahim-Ali W ,Theivanayagam S ,et al . Patient readmissions, emergency visits, and adverse events after software-assisted discharge from hospital: cluster randomized trial. J Hosp Med. 2009;4:11-9. [PMID: 19479782] CrossrefMedlineGoogle Scholar - 25.
Heidenreich PA ,Gholami P ,Sahay A ,Massie B ,Goldstein MK . Clinical reminders attached to echocardiography reports of patients with reduced left ventricular ejection fraction increase use of beta-blockers: a randomized trial. Circulation. 2007;115:2829-34. [PMID: 17515459] CrossrefMedlineGoogle Scholar - 26.
Tierney WM ,Overhage JM ,Murray MD ,Harris LE ,Zhou XH ,Eckert GJ ,et al . Can computer-generated evidence-based care suggestions enhance evidence-based management of asthma and chronic obstructive pulmonary disease? A randomized, controlled trial. Health Serv Res. 2005;40:477-97. [PMID: 15762903] CrossrefMedlineGoogle Scholar - 27.
Tierney WM ,Overhage JM ,Murray MD ,Harris LE ,Zhou XH ,Eckert GJ ,et al . Effects of computerized guidelines for managing heart disease in primary care. J Gen Intern Med. 2003;18:967-76. [PMID: 14687254] CrossrefMedlineGoogle Scholar - 28.
Gilutz H ,Novack L ,Shvartzman P ,Zelingher J ,Bonneh DY ,Henkin Y ,et al . Computerized community cholesterol control (4C): meeting the challenge of secondary prevention. Isr Med Assoc J. 2009;11:23-9. [PMID: 19344008] MedlineGoogle Scholar - 29.
Holt TA ,Thorogood M ,Griffiths F ,Munday S . Protocol for the ‘e-Nudge trial’: a randomised controlled trial of electronic feedback to reduce the cardiovascular risk of individuals in general practice [ISRCTN64828380]. Trials. 2006;7:11. [PMID: 16646967] CrossrefMedlineGoogle Scholar - 30.
Holt TA ,Thorogood M ,Griffiths F ,Munday S ,Friede T ,Stables D . Automated electronic reminders to facilitate primary cardiovascular disease prevention: randomised controlled trial. Br J Gen Pract. 2010;60:137-43. [PMID: 20353659] CrossrefMedlineGoogle Scholar - 31.
Khan S ,Maclean CD ,Littenberg B . The effect of the Vermont Diabetes Information System on inpatient and emergency room use: results from a randomized trial. Health Outcomes Res Med. 2010;1:61-e66. [PMID: 20975923] CrossrefMedlineGoogle Scholar - 32.
Maclean CD ,Gagnon M ,Callas P ,Littenberg B . The Vermont diabetes information system: a cluster randomized trial of a population based decision support system. J Gen Intern Med. 2009;24:1303-10. [PMID: 19862578] CrossrefMedlineGoogle Scholar - 33.
Brier ME ,Gaweda AE ,Dailey A ,Aronoff GR ,Jacobs AA . Randomized trial of model predictive control for improved anemia management. Clin J Am Soc Nephrol. 2010;5:814-20. [PMID: 20185598] CrossrefMedlineGoogle Scholar - 34.
Hamilton E ,Platt R ,Gauthier R ,McNamara H ,Miner L ,Rothenberg S ,et al . The effect of computer-assisted evaluation of labor on cesarean rates. J Healthc Qual. 2004;26:37-44. [PMID: 14763319] CrossrefMedlineGoogle Scholar - 35.
McDonald CJ ,Hui SL ,Tierney WM . Effects of computer reminders for influenza vaccination on morbidity during influenza epidemics. MD Comput. 1992;9:304-12. [PMID: 1522792] MedlineGoogle Scholar - 36.
Murray MD ,Harris LE ,Overhage JM ,Zhou XH ,Eckert GJ ,Smith FE ,et al . Failure of computerized treatment suggestions to improve health outcomes of outpatients with uncomplicated hypertension: results of a randomized controlled trial. Pharmacotherapy. 2004;24:324-37. [PMID: 15040645] CrossrefMedlineGoogle Scholar - 37.
Sequist TD ,Zaslavsky AM ,Marshall R ,Fletcher RH ,Ayanian JZ . Patient and physician reminders to promote colorectal cancer screening: a randomized controlled trial. Arch Intern Med. 2009;169:364-71. [PMID: 19237720] CrossrefMedlineGoogle Scholar - 38.
Subramanian U ,Fihn SD ,Weinberger M ,Plue L ,Smith FE ,Udris EM ,et al . A controlled trial of including symptom data in computer-based care suggestions for managing patients with chronic heart failure. Am J Med. 2004;116:375-84. [PMID: 15006586] CrossrefMedlineGoogle Scholar - 39.
Kuperman GJ ,Teich JM ,Tanasijevic MJ ,Ma'Luf N ,Rittenberg E ,Jha A ,et al . Improving response to critical laboratory results with automation: results of a randomized controlled trial. J Am Med Inform Assoc. 1999;6:512-22. [PMID: 10579608] CrossrefMedlineGoogle Scholar - 40.
McGregor JC ,Weekes E ,Forrest GN ,Standiford HC ,Perencevich EN ,Furuno JP ,et al . Impact of a computerized clinical decision support system on reducing inappropriate antimicrobial use: a randomized controlled trial. J Am Med Inform Assoc. 2006;13:378-84. [PMID: 16622162] CrossrefMedlineGoogle Scholar - 41.
Fihn SD ,McDonell MB ,Vermes D ,Henikoff JG ,Martin DC ,Callahan CM ,et al . A computerized intervention to improve timing of outpatient follow-up: a multicenter randomized trial in patients treated with warfarin. National Consortium of Anticoagulation Clinics. J Gen Intern Med. 1994;9:131-9. [PMID: 8195911] CrossrefMedlineGoogle Scholar - 42.
Gurwitz JH ,Field TS ,Rochon P ,Judge J ,Harrold LR ,Bell CM ,et al . Effect of computerized provider order entry with clinical decision support on adverse drug events in the long-term care setting. J Am Geriatr Soc. 2008;56:2225-33. [PMID: 19093922] CrossrefMedlineGoogle Scholar - 43.
Apkon M ,Mattera JA ,Lin Z ,Herrin J ,Bradley EH ,Carbone M ,et al . A randomized outpatient trial of a decision-support information technology tool. Arch Intern Med. 2005;165:2388-94. [PMID: 16287768] CrossrefMedlineGoogle Scholar - 44.
Bertoni AG ,Bonds DE ,Chen H ,Hogan P ,Crago L ,Rosenberger E ,et al . Impact of a multifaceted intervention on cholesterol management in primary care practices: guideline adherence for heart health randomized trial. Arch Intern Med. 2009;169:678-86. [PMID: 19364997] CrossrefMedlineGoogle Scholar - 45.
Burack RC ,Gimotty PA ,George J ,McBride S ,Moncrease A ,Simon MS ,et al . How reminders given to patients and physicians affected pap smear use in a health maintenance organization: results of a randomized controlled trial. Cancer. 1998;82:2391-400. [PMID: 9635532] CrossrefMedlineGoogle Scholar - 46.
Burack RC ,Gimotty PA ,Simon M ,Moncrease A ,Dews P . The effect of adding Pap smear information to a mammography reminder system in an HMO: results of randomized controlled trial. Prev Med. 2003;36:547-54. [PMID: 12689799] CrossrefMedlineGoogle Scholar - 47.
Cannon DS ,Allen SN . A comparison of the effects of computer and manual reminders on compliance with a mental health clinical practice guideline. J Am Med Inform Assoc. 2000;7:196-203. [PMID: 10730603] CrossrefMedlineGoogle Scholar - 48.
Demakis JG ,Beauchamp C ,Cull WL ,Denwood R ,Eisen SA ,Lofgren R ,et al . Improving residents' compliance with standards of ambulatory care: results from the VA Cooperative Study on Computerized Reminders. JAMA. 2000;284:1411-6. [PMID: 10989404] CrossrefMedlineGoogle Scholar - 49.
Dexter PR ,Perkins S ,Overhage JM ,Maharry K ,Kohler RB ,McDonald CJ . A computerized reminder system to increase the use of preventive care for hospitalized patients. N Engl J Med. 2001;345:965-70. [PMID: 11575289] CrossrefMedlineGoogle Scholar - 50.
Dexter PR ,Perkins SM ,Maharry KS ,Jones K ,McDonald CJ . Inpatient computer-based standing orders vs physician reminders to increase influenza and pneumococcal vaccination rates: a randomized trial. JAMA. 2004;292:2366-71. [PMID: 15547164] CrossrefMedlineGoogle Scholar - 51.
Eccles M ,McColl E ,Steen N ,Rousseau N ,Grimshaw J ,Parkin D ,et al . Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial. BMJ. 2002;325:941. [PMID: 12399345] CrossrefMedlineGoogle Scholar - 52.
Frank O ,Litt J ,Beilby J . Opportunistic electronic reminders. Improving performance of preventive care in general practice. Aust Fam Physician. 2004;33:87-90. [PMID: 14988972] MedlineGoogle Scholar - 53.
Fretheim A ,Aaserud M ,Oxman AD . Rational prescribing in primary care (RaPP): economic evaluation of an intervention to improve professional practice. PLoS Med. 2006;3:216. [PMID: 16737349] CrossrefMedlineGoogle Scholar - 54.
Fretheim A ,Oxman AD ,Håvelsrud K ,Treweek S ,Kristoffersen DT ,Bjørndal A . Rational prescribing in primary care (RaPP): a cluster randomized trial of a tailored intervention. PLoS Med. 2006;3:134. [PMID: 16737346] CrossrefMedlineGoogle Scholar - 55.
Litzelman DK ,Dittus RS ,Miller ME ,Tierney WM . Requiring physicians to respond to computerized reminders improves their compliance with preventive care protocols. J Gen Intern Med. 1993;8:311-7. [PMID: 8320575] CrossrefMedlineGoogle Scholar - 56.
McDowell I ,Newell C ,Rosser W . Comparison of three methods of recalling patients for influenza vaccination. CMAJ. 1986;135:991-7. [PMID: 3093045] MedlineGoogle Scholar - 57.
McDowell I ,Newell C ,Rosser W . Computerized reminders to encourage cervical screening in family practice. J Fam Pract. 1989;28:420-4. [PMID: 2495337] MedlineGoogle Scholar - 58.
Overhage JM ,Tierney WM ,McDonald CJ . Computer reminders to implement preventive care guidelines for hospitalized patients. Arch Intern Med. 1996;156:1551-6. [PMID: 8687263] CrossrefMedlineGoogle Scholar - 59.
Price M . Can hand-held computers improve adherence to guidelines? A (Palm) Pilot study of family doctors in British Columbia. Can Fam Physician. 2005;51:1506-7. [PMID: 16926943] MedlineGoogle Scholar - 60.
Taylor V ,Thompson B ,Lessler D ,Yasui Y ,Montano D ,Johnson KM ,et al . A clinic-based mammography intervention targeting inner-city women. J Gen Intern Med. 1999;14:104-11. [PMID: 10051781] CrossrefMedlineGoogle Scholar - 61.
Unrod M ,Smith M ,Spring B ,DePue J ,Redd W ,Winkel G . Randomized controlled trial of a computer-based, tailored intervention to increase smoking cessation counseling by primary care physicians. J Gen Intern Med. 2007;22:478-84. [PMID: 17372796] CrossrefMedlineGoogle Scholar - 62.
Dykes PC ,Carroll DL ,Hurley A ,Lipsitz S ,Benoit A ,Chang F ,et al . Fall prevention in acute care hospitals: a randomized trial. JAMA. 2010;304:1912-8. [PMID: 21045097] CrossrefMedlineGoogle Scholar - 63.
Chambers CV ,Balaban DJ ,Carlson BL ,Ungemack JA ,Grasberger DM . Microcomputer-generated reminders. Improving the compliance of primary care physicians with mammography screening guidelines. J Fam Pract. 1989;29:273-80. [PMID: 2769192] MedlineGoogle Scholar - 64.
Burack RC ,Gimotty PA . Promoting screening mammography in inner-city settings. The sustained effectiveness of computerized reminders in a randomized controlled trial. Med Care. 1997;35:921-31. [PMID: 9298081] CrossrefMedlineGoogle Scholar - 65.
Burack RC ,Gimotty PA ,George J ,Stengle W ,Warbasse L ,Moncrease A . Promoting screening mammography in inner-city settings: a randomized controlled trial of computerized reminders as a component of a program to facilitate mammography. Med Care. 1994;32:609-24. [PMID: 8189778] CrossrefMedlineGoogle Scholar - 66.
Fiks AG ,Hunter KF ,Localio AR ,Grundmeier RW ,Bryant-Stephens T ,Luberti AA ,et al . Impact of electronic health record-based alerts on influenza vaccination for children with asthma. Pediatrics. 2009;124:159-69. [PMID: 19564296] CrossrefMedlineGoogle Scholar - 67.
Flanagan JR ,Doebbeling BN ,Dawson J ,Beekmann S . Randomized study of online vaccine reminders in adult primary care. Proc AMIA Symp. 1999;:755-9. [PMID: 10566461] MedlineGoogle Scholar - 68.
Fordham D ,McPhee SJ ,Bird JA ,Rodnick JE ,Detmer WM . The Cancer Prevention Reminder System. MD Comput. 1990;7:289-95. [PMID: 2243544] MedlineGoogle Scholar - 69.
McPhee SJ ,Bird JA ,Jenkins CN ,Fordham D . Promoting cancer screening. A randomized, controlled trial of three interventions. Arch Intern Med. 1989;149:1866-72. [PMID: 2764657] CrossrefMedlineGoogle Scholar - 70.
Gill JM ,Chen YX ,Glutting JJ ,Diamond JJ ,Lieberman MI . Impact of decision support in electronic medical records on lipid management in primary care. Popul Health Manag. 2009;12:221-6. [PMID: 19848563] CrossrefMedlineGoogle Scholar - 71.
Hobbs FD ,Delaney BC ,Carson A ,Kenkre JE . A prospective controlled trial of computerized decision support for lipid management in primary care. Fam Pract. 1996;13:133-7. [PMID: 8732323] CrossrefMedlineGoogle Scholar - 72.
Holbrook A ,Thabane L ,Keshavjee K ,Dolovich L ,Bernstein B ,Chan D ,et al ;COMPETE II Investigators . Individualized electronic decision support and reminders to improve diabetes care in the community: COMPETE II randomized trial. CMAJ. 2009;181:37-44. [PMID: 19581618] CrossrefMedlineGoogle Scholar - 73.
Kenealy T ,Arroll B ,Petrie KJ . Patients and computers as reminders to screen for diabetes in family practice. Randomized-controlled trial. J Gen Intern Med. 2005;20:916-21. [PMID: 16191138] CrossrefMedlineGoogle Scholar - 74.
Lobach DF ,Hammond WE . Development and evaluation of a Computer-Assisted Management Protocol (CAMP): improved compliance with care guidelines for diabetes mellitus. Proc Annu Symp Comput Appl Med Care. 1994;:787-91. [PMID: 7950032] MedlineGoogle Scholar - 75.
Ornstein SM ,Garr DR ,Jenkins RG ,Rust PF ,Arnon A . Computer-generated physician and patient reminders. Tools to improve population adherence to selected preventive services. J Fam Pract. 1991;32:82-90. [PMID: 1985140] MedlineGoogle Scholar - 76.
Peterson KA ,Radosevich DM ,O'Connor PJ ,Nyman JA ,Prineas RJ ,Smith SA ,et al . Improving diabetes care in practice: findings from the TRANSLATE trial. Diabetes Care. 2008;31:2238-43. [PMID: 18809622] CrossrefMedlineGoogle Scholar - 77.
Reeve JF ,Tenni PC ,Peterson GM . An electronic prompt in dispensing software to promote clinical interventions by community pharmacists: a randomized controlled trial. Br J Clin Pharmacol. 2008;65:377-85. [PMID: 17764471] CrossrefMedlineGoogle Scholar - 78.
Rosser WW ,Hutchison BG ,McDowell I ,Newell C . Use of reminders to increase compliance with tetanus booster vaccination. CMAJ. 1992;146:911-7. [PMID: 1544078] MedlineGoogle Scholar - 79.
Rosser WW ,McDowell I ,Newell C . Use of reminders for preventive procedures in family medicine. CMAJ. 1991;145:807-14. [PMID: 1913409] MedlineGoogle Scholar - 80.
Sequist TD ,Gandhi TK ,Karson AS ,Fiskio JM ,Bugbee D ,Sperling M ,et al . A randomized trial of electronic clinical reminders to improve quality of care for diabetes and coronary artery disease. J Am Med Inform Assoc. 2005;12:431-7. [PMID: 15802479] CrossrefMedlineGoogle Scholar - 81.
Tierney WM ,Hui SL ,McDonald CJ . Delayed feedback of physician performance versus immediate reminders to perform preventive care. Effects on physician compliance. Med Care. 1986;24:659-66. [PMID: 3736141] CrossrefMedlineGoogle Scholar - 82.
van Wyk JT ,van Wijk MA ,Sturkenboom MC ,Mosseveld M ,Moorman PW ,van der Lei J . Electronic alerts versus on-demand decision support to improve dyslipidemia treatment: a cluster randomized controlled trial. Circulation. 2008;117:371-8. [PMID: 18172036] CrossrefMedlineGoogle Scholar - 83.
Bates DW ,Kuperman GJ ,Rittenberg E ,Teich JM ,Fiskio J ,Ma'luf N ,et al . A randomized trial of a computer-based intervention to reduce utilization of redundant laboratory tests. Am J Med. 1999;106:144-50. [PMID: 10230742] CrossrefMedlineGoogle Scholar - 84.
Bell LM ,Grundmeier R ,Localio R ,Zorc J ,Fiks AG ,Zhang X ,et al . Electronic health record-based decision support to improve asthma care: a cluster-randomized trial. Pediatrics. 2010;125:770-7. [PMID: 20231191] CrossrefMedlineGoogle Scholar - 85.
Downs M ,Turner S ,Bryans M ,Wilcock J ,Keady J ,Levin E ,et al . Effectiveness of educational interventions in improving detection and management of dementia in primary care: cluster randomised controlled study. BMJ. 2006;332:692-6. [PMID: 16565124] CrossrefMedlineGoogle Scholar - 86.
Emery J ,Morris H ,Goodchild R ,Fanshawe T ,Prevost AT ,Bobrow M ,et al . The GRAIDS Trial: a cluster randomised controlled trial of computer decision support for the management of familial cancer risk in primary care. Br J Cancer. 2007;97:486-93. [PMID: 17700548] CrossrefMedlineGoogle Scholar - 87.
Feldstein AC ,Smith DH ,Perrin N ,Yang X ,Rix M ,Raebel MA ,et al . Improved therapeutic monitoring with several interventions: a randomized trial. Arch Intern Med. 2006;166:1848-54. [PMID: 17000941] CrossrefMedlineGoogle Scholar - 88.
Flottorp S ,Oxman AD ,Håvelsrud K ,Treweek S ,Herrin J . Cluster randomised controlled trial of tailored interventions to improve the management of urinary tract infections in women and sore throat. BMJ. 2002;325:367. [PMID: 12183309] CrossrefMedlineGoogle Scholar - 89.
Greiver M ,Drummond N ,White D ,Weshler J ,Moineddin R ;North Toronto Primary Care Research Network (Nortren) . Angina on the Palm: randomized controlled pilot trial of Palm PDA software for referrals for cardiac testing. Can Fam Physician. 2005;51:382-3. [PMID: 16926933] MedlineGoogle Scholar - 90.
Harpole LH ,Khorasani R ,Fiskio J ,Kuperman GJ ,Bates DW . Automated evidence-based critiquing of orders for abdominal radiographs: impact on utilization and appropriateness. J Am Med Inform Assoc. 1997;4:511-21. [PMID: 9391938] CrossrefMedlineGoogle Scholar - 91.
Lee NJ ,Chen ES ,Currie LM ,Donovan M ,Hall EK ,Jia H ,et al . The effect of a mobile clinical decision support system on the diagnosis of obesity and overweight in acute and primary care encounters. ANS Adv Nurs Sci. 2009;32:211-21. [PMID: 19707090] CrossrefMedlineGoogle Scholar - 92.
Lo HG ,Matheny ME ,Seger DL ,Bates DW ,Gandhi TK . Impact of non-interruptive medication laboratory monitoring alerts in ambulatory care. J Am Med Inform Assoc. 2009;16:66-71. [PMID: 18952945] CrossrefMedlineGoogle Scholar - 93.
Matheny ME ,Sequist TD ,Seger AC ,Fiskio JM ,Sperling M ,Bugbee D ,et al . A randomized trial of electronic clinical reminders to improve medication laboratory monitoring. J Am Med Inform Assoc. 2008;15:424-9. [PMID: 18436905] CrossrefMedlineGoogle Scholar - 94.
McDonald CJ . Use of a computer to detect and respond to clinical events: its effect on clinician behavior. Ann Intern Med. 1976;84:162-7. [PMID: 1252043] LinkGoogle Scholar - 95.
McDowell I ,Newell C ,Rosser W . A randomized trial of computerized reminders for blood pressure screening in primary care. Med Care. 1989;27:297-305. [PMID: 2494397] CrossrefMedlineGoogle Scholar - 96.
Palen TE ,Raebel M ,Lyons E ,Magid DM . Evaluation of laboratory monitoring alerts within a computerized physician order entry system for medication orders. Am J Manag Care. 2006;12:389-95. [PMID: 16834525] MedlineGoogle Scholar - 97.
Palen TE ,Price DW ,Snyder AJ ,Shetterly SM . Computerized alert reduced D-dimer testing in the elderly. Am J Manag Care. 2010;16:267-75. [PMID: 21087072] MedlineGoogle Scholar - 98.
Player MS ,Gill JM ,Mainous AG ,Everett CJ ,Koopman RJ ,Diamond JJ ,et al . An electronic medical record-based intervention to improve quality of care for gastro-esophageal reflux disease (GERD) and atypical presentations of GERD. Qual Prim Care. 2010;18:223-9. [PMID: 20836938] MedlineGoogle Scholar - 99.
Raebel MA ,Lyons EE ,Chester EA ,Bodily MA ,Kelleher JA ,Long CL ,et al . Improving laboratory monitoring at initiation of drug therapy in ambulatory care: a randomized trial. Arch Intern Med. 2005;165:2395-401. [PMID: 16287769] CrossrefMedlineGoogle Scholar - 100.
Raebel MA ,Chester EA ,Newsom EE ,Lyons EE ,Kelleher JA ,Long C ,et al . Randomized trial to improve laboratory safety monitoring of ongoing drug therapy in ambulatory patients. Pharmacotherapy. 2006;26:619-26. [PMID: 16637791] CrossrefMedlineGoogle Scholar - 101.
Roukema J ,Steyerberg EW ,van der Lei J ,Moll HA . Randomized trial of a clinical decision support system: impact on the management of children with fever without apparent source. J Am Med Inform Assoc. 2008;15:107-13. [PMID: 17947627] CrossrefMedlineGoogle Scholar - 102.
Roy PM ,Durieux P ,Gillaizeau F ,Legall C ,Armand-Perroux A ,Martino L ,et al . A computerized handheld decision-support system to improve pulmonary embolism diagnosis: a randomized trial. Ann Intern Med. 2009;151:677-86. [PMID: 19920268] LinkGoogle Scholar - 103.
Schriefer SP ,Landis SE ,Turbow DJ ,Patch SC . Effect of a computerized body mass index prompt on diagnosis and treatment of adult obesity. Fam Med. 2009;41:502-7. [PMID: 19582636] MedlineGoogle Scholar - 104.
Stiell IG ,Clement CM ,Grimshaw J ,Brison RJ ,Rowe BH ,Schull MJ ,et al . Implementation of the Canadian C-Spine Rule: prospective 12 centre cluster randomised trial. BMJ. 2009;339:b4146. [PMID: 19875425] CrossrefMedlineGoogle Scholar - 105.
Sundaram V ,Lazzeroni LC ,Douglass LR ,Sanders GD ,Tempio P ,Owens DK . A randomized trial of computer-based reminders and audit and feedback to improve HIV screening in a primary care setting. Int J STD AIDS. 2009;20:527-33. [PMID: 19625582] CrossrefMedlineGoogle Scholar - 106.
Tierney WM ,McDonald CJ ,Martin DK ,Rogers MP . Computerized display of past test results. Effect on outpatient testing. Ann Intern Med. 1987;107:569-74. [PMID: 3631792] LinkGoogle Scholar - 107.
van Wijk MA ,van der Lei J ,Mosseveld M ,Bohnen AM ,van Bemmel JH . Assessment of decision support for blood test ordering in primary care. a randomized trial. Ann Intern Med. 2001;134:274-81. [PMID: 11182837] LinkGoogle Scholar - 108.
Walker J ,Fairley CK ,Walker SM ,Gurrin LC ,Gunn JM ,Pirotta MV ,et al . Computer reminders for Chlamydia screening in general practice: a randomized controlled trial. Sex Transm Dis. 2010;37:445-50. [PMID: 20375930] CrossrefMedlineGoogle Scholar - 109.
Wilson BJ ,Torrance N ,Mollison J ,Watson MS ,Douglas A ,Miedzybrodzka Z ,et al . Cluster randomized trial of a multifaceted primary care decision-support intervention for inherited breast cancer risk. Fam Pract. 2006;23:537-44. [PMID: 16787957] CrossrefMedlineGoogle Scholar - 110.
Bourgeois FC ,Linder J ,Johnson SA ,Co JP ,Fiskio J ,Ferris TG . Impact of a computerized template on antibiotic prescribing for acute respiratory infections in children and adolescents. Clin Pediatr (Phila). 2010;49:976-83. [PMID: 20724348] CrossrefMedlineGoogle Scholar - 111.
Christakis DA ,Zimmerman FJ ,Wright JA ,Garrison MM ,Rivara FP ,Davis RL . A randomized controlled trial of point-of-care evidence to improve the antibiotic prescribing practices for otitis media in children. Pediatrics. 2001;107:15. [PMID: 11158489] CrossrefMedlineGoogle Scholar - 112.
Co JP ,Johnson SA ,Poon EG ,Fiskio J ,Rao SR ,Van Cleave J ,et al . Electronic health record decision support and quality of care for children with ADHD. Pediatrics. 2010;126:239-46. [PMID: 20643719] CrossrefMedlineGoogle Scholar - 113.
Cobos A ,Vilaseca J ,Asenjo C ,Pedro-Botet J ,Sánchez E ,Val A ,et al . Cost effectiveness of a clinical decision support system based on the recommendations of the European Society of Cardiology and other societies for the management of hypercholesterolemia: report of a cluster-randomized trial. Disease Management & Health Outcomes. 2005;13:421-32. CrossrefGoogle Scholar - 114.
Davis RL ,Wright J ,Chalmers F ,Levenson L ,Brown JC ,Lozano P ,et al . A cluster randomized clinical trial to improve prescribing patterns in ambulatory pediatrics. PLoS Clin Trials. 2007;2:25. [PMID: 17525793] CrossrefMedlineGoogle Scholar - 115.
Feldstein A ,Elmer PJ ,Smith DH ,Herson M ,Orwoll E ,Chen C ,et al . Electronic medical record reminder improves osteoporosis management after a fracture: a randomized, controlled trial. J Am Geriatr Soc. 2006;54:450-7. [PMID: 16551312] CrossrefMedlineGoogle Scholar - 116.
Field TS ,Rochon P ,Lee M ,Gavendo L ,Baril JL ,Gurwitz JH . Computerized clinical decision support during medication ordering for long-term care residents with renal insufficiency. J Am Med Inform Assoc. 2009;16:480-5. [PMID: 19390107] CrossrefMedlineGoogle Scholar - 117.
Filippi A ,Sabatini A ,Badioli L ,Samani F ,Mazzaglia G ,Catapano A ,et al . Effects of an automated electronic reminder in changing the antiplatelet drug-prescribing behavior among Italian general practitioners in diabetic patients: an intervention trial. Diabetes Care. 2003;26:1497-500. [PMID: 12716811] CrossrefMedlineGoogle Scholar - 118.
Fitzmaurice DA ,Hobbs FD ,Murray ET ,Holder RL ,Allan TF ,Rose PE . Oral anticoagulation management in primary care with the use of computerized decision support and near-patient testing: a randomized, controlled trial. Arch Intern Med. 2000;160:2343-8. [PMID: 10927732] CrossrefMedlineGoogle Scholar - 119.
Fortuna RJ ,Zhang F ,Ross-Degnan D ,Campion FX ,Finkelstein JA ,Kotch JB ,et al . Reducing the prescribing of heavily marketed medications: a randomized controlled trial. J Gen Intern Med. 2009;24:897-903. [PMID: 19475459] CrossrefMedlineGoogle Scholar - 120.
Goud R ,de Keizer NF ,ter Riet G ,Wyatt JC ,Hasman A ,Hellemans IM ,et al . Effect of guideline based computerised decision support on decision making of multidisciplinary teams: cluster randomised trial in cardiac rehabilitation. BMJ. 2009;338:b1440. [PMID: 19398471] CrossrefMedlineGoogle Scholar - 121.
Hicks LS ,Sequist TD ,Ayanian JZ ,Shaykevich S ,Fairchild DG ,Orav EJ ,et al . Impact of computerized decision support on blood pressure management and control: a randomized controlled trial. J Gen Intern Med. 2008;23:429-41. [PMID: 18373141] CrossrefMedlineGoogle Scholar - 122.
Krall MA ,Traunweiser K ,Towery W . Effectiveness of an electronic medical record clinical quality alert prepared by off-line data analysis. Stud Health Technol Inform. 2004;107:135-9. [PMID: 15360790] MedlineGoogle Scholar - 123.
Linder JA ,Rigotti NA ,Schneider LI ,Kelley JH ,Brawarsky P ,Haas JS . An electronic health record-based intervention to improve tobacco treatment in primary care: a cluster-randomized controlled trial. Arch Intern Med. 2009;169:781-7. [PMID: 19398690] CrossrefMedlineGoogle Scholar - 124.
Locatelli F ,Covic A ,Macdougall IC ,Scherhag A ,Wiecek A ;ORAMA Study Group . Effect of computer-assisted European Best Practice Guideline implementation on adherence and target attainment: ORAMA results. J Nephrol. 2009;22:662-74. [PMID: 19810000] MedlineGoogle Scholar - 125.
Manotti C ,Moia M ,Palareti G ,Pengo V ,Ria L ,Dettori AG . Effect of computer-aided management on the quality of treatment in anticoagulated patients: a prospective, randomized, multicenter trial of APROAT (Automated PRogram for Oral Anticoagulant Treatment). Haematologica. 2001;86:1060-70. [PMID: 11602412] MedlineGoogle Scholar - 126.
Marco F ,Sedano C ,Bermúdez A ,López-Duarte M ,Fernández-Fontecha E ,Zubizarreta A . A prospective controlled study of a computer-assisted acenocoumarol dosage program. Pathophysiol Haemost Thromb. 2003;33:59-63. [PMID: 14624045] CrossrefMedlineGoogle Scholar - 127.
Martens JD ,van der Aa A ,Panis B ,van der Weijden T ,Winkens RA ,Severens JL . Design and evaluation of a computer reminder system to improve prescribing behaviour of GPs. Stud Health Technol Inform. 2006;124:617-23. [PMID: 17108585] MedlineGoogle Scholar - 128.
Martens JD ,van der Weijden T ,Severens JL ,de Clercq PA ,de Bruijn DP ,Kester AD ,et al . The effect of computer reminders on GPs' prescribing behaviour: a cluster-randomised trial. Int J Med Inform. 2007;76Suppl 3 S403-16. [PMID: 17569575] CrossrefMedlineGoogle Scholar - 129.
Montgomery AA ,Fahey T ,Peters TJ ,MacIntosh C ,Sharp DJ . Evaluation of computer based clinical decision support system and risk chart for management of hypertension in primary care: randomised controlled trial. BMJ. 2000;320:686-90. [PMID: 10710578] CrossrefMedlineGoogle Scholar - 130.
Overhage JM ,Tierney WM ,Zhou XH ,McDonald CJ . A randomized trial of “corollary orders” to prevent errors of omission. J Am Med Inform Assoc. 1997;4:364-75. [PMID: 9292842] CrossrefMedlineGoogle Scholar - 131.
Peterson JF ,Rosenbaum BP ,Waitman LR ,Habermann R ,Powers J ,Harrell D ,et al . Physicians' response to guided geriatric dosing: initial results from a randomized trial. Stud Health Technol Inform. 2007;129:1037-40. [PMID: 17911873] MedlineGoogle Scholar - 132.
Phillips LS ,Ziemer DC ,Doyle JP ,Barnes CS ,Kolm P ,Branch WT ,et al . An endocrinologist-supported intervention aimed at providers improves diabetes management in a primary care site: improving primary care of African Americans with diabetes (IPCAAD) 7. Diabetes Care. 2005;28:2352-60. [PMID: 16186262] CrossrefMedlineGoogle Scholar - 133.
Ziemer DC ,Doyle JP ,Barnes CS ,Branch WT ,Cook CB ,El-Kebbi IM ,et al . An intervention to overcome clinical inertia and improve diabetes mellitus control in a primary care setting: Improving Primary Care of African Americans with Diabetes (IPCAAD) 8. Arch Intern Med. 2006;166:507-13. [PMID: 16534036] CrossrefMedlineGoogle Scholar - 134.
Raebel MA ,Charles J ,Dugan J ,Carroll NM ,Korner EJ ,Brand DW ,et al . Randomized trial to improve prescribing safety in ambulatory elderly patients. J Am Geriatr Soc. 2007;55:977-85. [PMID: 17608868] CrossrefMedlineGoogle Scholar - 135.
Rood E ,Bosman RJ ,van der Spoel JI ,Taylor P ,Zandstra DF . Use of a computerized guideline for glucose regulation in the intensive care unit improved both guideline adherence and glucose regulation. J Am Med Inform Assoc. 2005;12:172-80. [PMID: 15561795] CrossrefMedlineGoogle Scholar - 136.
Rossi RA ,Every NR . A computerized intervention to decrease the use of calcium channel blockers in hypertension. J Gen Intern Med. 1997;12:672-8. [PMID: 9383135] CrossrefMedlineGoogle Scholar - 137.
Rothschild JM ,McGurk S ,Honour M ,Lu L ,McClendon AA ,Srivastava P ,et al . Assessment of education and computerized decision support interventions for improving transfusion practice. Transfusion. 2007;47:228-39. [PMID: 17302768] CrossrefMedlineGoogle Scholar - 138.
Samore MH ,Bateman K ,Alder SC ,Hannah E ,Donnelly S ,Stoddard GJ ,et al . Clinical decision support and appropriateness of antimicrobial prescribing: a randomized trial. JAMA. 2005;294:2305-14. [PMID: 16278358] CrossrefMedlineGoogle Scholar - 139.
Shojania KG ,Yokoe D ,Platt R ,Fiskio J ,Ma'luf N ,Bates DW . Reducing vancomycin use utilizing a computer guideline: results of a randomized controlled trial. J Am Med Inform Assoc. 1998;5:554-62. [PMID: 9824802] CrossrefMedlineGoogle Scholar - 140.
Simon SR ,Smith DH ,Feldstein AC ,Perrin N ,Yang X ,Zhou Y ,et al . Computerized prescribing alerts and group academic detailing to reduce the use of potentially inappropriate medications in older people. J Am Geriatr Soc. 2006;54:963-8. [PMID: 16776793] CrossrefMedlineGoogle Scholar - 141.
Smith SA ,Shah ND ,Bryant SC ,Christianson TJ ,Bjornsen SS ,Giesler PD ,et al ;Evidens Research Group . Chronic care model and shared care in diabetes: randomized trial of an electronic decision support system. Mayo Clin Proc. 2008;83:747-57. [PMID: 18613991] CrossrefMedlineGoogle Scholar - 142.
Strom BL ,Schinnar R ,Aberra F ,Bilker W ,Hennessy S ,Leonard CE ,et al . Unintended effects of a computerized physician order entry nearly hard-stop alert to prevent a drug interaction: a randomized controlled trial. Arch Intern Med. 2010;170:1578-83. [PMID: 20876410] CrossrefMedlineGoogle Scholar - 143.
Strom BL ,Schinnar R ,Bilker W ,Hennessy S ,Leonard CE ,Pifer E . Randomized clinical trial of a customized electronic alert requiring an affirmative response compared to a control group receiving a commercial passive CPOE alert: NSAID—warfarin co-prescribing as a test case. J Am Med Inform Assoc. 2010;17:411-5. [PMID: 20595308] CrossrefMedlineGoogle Scholar - 144.
Tamblyn R ,Huang A ,Perreault R ,Jacques A ,Roy D ,Hanley J ,et al . The medical office of the 21st century (MOXXI): effectiveness of computerized decision-making support in reducing inappropriate prescribing in primary care. CMAJ. 2003;169:549-56. [PMID: 12975221] MedlineGoogle Scholar - 145.
Tamblyn R ,Huang A ,Taylor L ,Kawasumi Y ,Bartlett G ,Grad R ,et al . A randomized trial of the effectiveness of on-demand versus computer-triggered drug decision support in primary care. J Am Med Inform Assoc. 2008;15:430-8. [PMID: 18436904] CrossrefMedlineGoogle Scholar - 146.
Tamblyn R ,Reidel K ,Huang A ,Taylor L ,Winslade N ,Bartlett G ,et al . Increasing the detection and response to adherence problems with cardiovascular medication in primary care through computerized drug management systems: a randomized controlled trial. Med Decis Making. 2010;30:176-88. [PMID: 19675319] CrossrefMedlineGoogle Scholar - 147.
Terrell KM ,Perkins AJ ,Dexter PR ,Hui SL ,Callahan CM ,Miller DK . Computerized decision support to reduce potentially inappropriate prescribing to older emergency department patients: a randomized, controlled trial. J Am Geriatr Soc. 2009;57:1388-94. [PMID: 19549022] CrossrefMedlineGoogle Scholar - 148.
Terrell KM ,Perkins AJ ,Hui SL ,Callahan CM ,Dexter PR ,Miller DK . Computerized decision support for medication dosing in renal insufficiency: a randomized, controlled trial. Ann Emerg Med. 2010;56:623-9. [PMID: 20452703] CrossrefMedlineGoogle Scholar - 149.
Vadher BD ,Patterson DL ,Leaning M . Comparison of oral anticoagulant control by a nurse-practitioner using a computer decision-support system with that by clinicians. Clin Lab Haematol. 1997;19:203-7. [PMID: 9352146] CrossrefMedlineGoogle Scholar - 150.
Vadher B ,Patterson DL ,Leaning M . Evaluation of a decision support system for initiation and control of oral anticoagulation in a randomised trial. BMJ. 1997;314:1252-6. [PMID: 9154031] CrossrefMedlineGoogle Scholar - 151.
Vissers MC ,Biert J ,van der Linden CJ ,Hasman A . Effects of a supportive protocol processing system (ProtoVIEW) on clinical behaviour of residents in the accident and emergency department. Comput Methods Programs Biomed. 1996;49:177-84. [PMID: 8735024] CrossrefMedlineGoogle Scholar - 152.
Vissers MC ,Hasman A ,van der Linden CJ . Protocol processing system (ProtoVIEW) to support residents at the emergency ward. Comput Methods Programs Biomed. 1995;48:53-8. [PMID: 8846712] CrossrefMedlineGoogle Scholar - 153.
Weir CJ ,Lees KR ,MacWalter RS ,Muir KW ,Wallesch CW ,McLelland EV ,et al ;PRISM Study Group . Cluster-randomized, controlled trial of computer-based decision support for selecting long-term anti-thrombotic therapy after acute ischaemic stroke. QJM. 2003;96:143-53. [PMID: 12589012] CrossrefMedlineGoogle Scholar - 154.
White KS ,Lindsay A ,Pryor TA ,Brown WF ,Walsh K . Application of a computerized medical decision-making process to the problem of digoxin intoxication. J Am Coll Cardiol. 1984;4:571-6. [PMID: 6381570] CrossrefMedlineGoogle Scholar - 155.
Alper BS ,White DS ,Ge B . Physicians answer more clinical questions and change clinical decisions more often with synthesized evidence: a randomized trial in primary care. Ann Fam Med. 2005;3:507-13. [PMID: 16338914] CrossrefMedlineGoogle Scholar - 156.
Del Fiol G ,Haug PJ ,Cimino JJ ,Narus SP ,Norlin C ,Mitchell JA . Effectiveness of topic-specific infobuttons: a randomized controlled trial. J Am Med Inform Assoc. 2008;15:752-9. [PMID: 18755999] CrossrefMedlineGoogle Scholar - 157.
Etchells E ,Adhikari NK ,Cheung C ,Fowler R ,Kiss A ,Quan S ,et al . Real-time clinical alerting: effect of an automated paging system on response time to critical laboratory values—a randomised controlled trial. Qual Saf Health Care. 2010;19:99-102. [PMID: 20351157] CrossrefMedlineGoogle Scholar - 158.
Bird JA ,McPhee SJ ,Jenkins C ,Fordham D . Three strategies to promote cancer screening. How feasible is wide-scale implementation? Med Care. 1990;28:1005-12. [PMID: 2250488] CrossrefMedlineGoogle Scholar - 159.
Frame PS ,Zimmer JG ,Werth PL ,Hall WJ ,Eberly SW . Computer-based vs manual health maintenance tracking. A controlled trial. Arch Fam Med. 1994;3:581-8. [PMID: 7921293] CrossrefMedlineGoogle Scholar - 160.
Tierney WM ,McDonald CJ ,Hui SL ,Martin DK . Computer predictions of abnormal test results. Effects on outpatient testing. JAMA. 1988;259:1194-8. [PMID: 3339821] CrossrefMedlineGoogle Scholar - 161.
Cleveringa FG ,Gorter KJ ,van den Donk M ,Rutten GE . Combined task delegation, computerized decision support, and feedback improve cardiovascular risk for type 2 diabetic patients: a cluster randomized trial in primary care. Diabetes Care. 2008;31:2273-5. [PMID: 18796619] CrossrefMedlineGoogle Scholar - 162.
Cleveringa FG ,Welsing PM ,van den Donk M ,Gorter KJ ,Niessen LW ,Rutten GE ,et al . Cost-effectiveness of the diabetes care protocol, a multifaceted computerized decision support diabetes management intervention that reduces cardiovascular risk. Diabetes Care. 2010;33:258-63. [PMID: 19933991] CrossrefMedlineGoogle Scholar - 163.
Smith DH ,Feldstein AC ,Perrin NA ,Yang X ,Rix MM ,Raebel MA ,et al . Improving laboratory monitoring of medications: an economic analysis alongside a clinical trial. Am J Manag Care. 2009;15:281-9. [PMID: 19435396] MedlineGoogle Scholar - 164.
Judge J ,Field TS ,DeFlorio M ,Laprino J ,Auger J ,Rochon P ,et al . Prescribers' responses to alerts during medication ordering in the long term care setting. J Am Med Inform Assoc. 2006;13:385-90. [PMID: 16622171] CrossrefMedlineGoogle Scholar - 165.
Maviglia SM ,Yoon CS ,Bates DW ,Kuperman G . KnowledgeLink: impact of context-sensitive information retrieval on clinicians' information needs. J Am Med Inform Assoc. 2006;13:67-73. [PMID: 16221942] CrossrefMedlineGoogle Scholar - 166.
Rollman BL ,Hanusa BH ,Gilbert T ,Lowe HJ ,Kapoor WN ,Schulberg HC . The electronic medical record. A randomized trial of its impact on primary care physicians' initial management of major depression [corrected]. Arch Intern Med. 2001;161:189-97. [PMID: 11176732] CrossrefMedlineGoogle Scholar - 167.
McLaughlin D ,Hayes JR ,Kelleher K . Office-based interventions for recognizing abnormal pediatric blood pressures. Clin Pediatr (Phila). 2010;49:355-62. CrossrefMedlineGoogle Scholar - 168.
Hetlevik I ,Holmen J ,Krüger O . Implementing clinical guidelines in the treatment of hypertension in general practice. Evaluation of patient outcome related to implementation of a computer-based clinical decision support system. Scand J Prim Health Care. 1999;17:35-40. [PMID: 10229991] CrossrefMedlineGoogle Scholar - 169.
Hetlevik I ,Holmen J ,Krüger O ,Kristensen P ,Iversen H . Implementing clinical guidelines in the treatment of hypertension in general practice. Blood Press. 1998;7:270-6. [PMID: 10321438] CrossrefMedlineGoogle Scholar - 170.
Hetlevik I ,Holmen J ,Krüger O ,Kristensen P ,Iversen H ,Furuseth K . Implementing clinical guidelines in the treatment of diabetes mellitus in general practice. Evaluation of effort, process, and patient outcome related to implementation of a computer-based decision support system. Int J Technol Assess Health Care. 2000;16:210-27. [PMID: 10815366] CrossrefMedlineGoogle Scholar - 171.
Bosworth HB ,Olsen MK ,Dudley T ,Orr M ,Goldstein MK ,Datta SK ,et al . Patient education and provider decision support to control blood pressure in primary care: a cluster randomized trial. Am Heart J. 2009;157:450-6. [PMID: 19249414] CrossrefMedlineGoogle Scholar - 172.
Bosworth HB ,Olsen MK ,Goldstein MK ,Orr M ,Dudley T ,McCant F ,et al . The veterans' study to improve the control of hypertension (V-STITCH): design and methodology. Contemp Clin Trials. 2005;26:155-68. [PMID: 15837438] CrossrefMedlineGoogle Scholar - 173.
Buntin MB ,Burke MF ,Hoaglin MC ,Blumenthal D . The benefits of health information technology: a review of the recent literature shows predominantly positive results. Health Aff (Millwood). 2011;30:464-71. [PMID: 21383365] CrossrefMedlineGoogle Scholar - 174.
Osheroff JA ;Healthcare Information and Management Systems Society . Improving Outcomes with Clinical Decision Support: An Implementer's Guide. 2nd ed. Chicago: Healthcare Information and Management Systems Society; 2005. Google Scholar
Author, Article, and Disclosure Information
Tiffani J. Bright,
From Duke Evidence-based Practice Center, Duke Clinical Research Institute, Duke University School of Medicine, Durham, North Carolina, and School of Medicine, University of Utah, Salt Lake City, Utah.
Disclaimer: The authors of this report are responsible for its content. Statements in the report should not be construed as endorsements by the Agency for Healthcare Research and Quality or the U.S. Department of Health and Human Services.
Acknowledgment: The authors thank Connie Schardt, MSLS, for help with the literature search and retrieval.
Grant Support: This project was funded under contract 290-2007-10066-I from the Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services.
Disclosures: Disclosures can be viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum=M11-1215.
Corresponding Author: Gillian D. Sanders, PhD, Evidence-based Practice Center, Director, Duke Clinical Research Institute, 2400 Pratt Street, Durham, NC 27705; e-mail, gillian.
Current Author Addresses: Dr. Bright: 16 Kenilworth Drive, Hampton, VA 23666.
Mr. Wong: 2618 Briar Trail, Apartment 202, Schaumburg, IL 60173.
Dr. Dhurjati: D330-1 Mayo (MMC 729), 420 Delaware Street SE, University of Minnesota, Minneapolis, MN 55455.
Ms. Bristow: 728 Irolo Street, Apartment D, Los Angeles, CA 90005.
Dr. Bastian: Health Services Research & Development, 152 Veterans Affairs Medical Center, 508 Fulton Street, Durham, NC 27705.
Drs. Coeytaux, Hasselblad, Williams, and Sanders; Mr. Musty; Ms. Wing; and Ms. Kendrick: Duke Evidence-based Practice Center, Duke Clinical Research Institute, 2400 Pratt Street, Durham, NC 27705.
Dr. Samsa: Department of Biostatistics and Bioinformatics, Duke University School of Medicine, Durham, NC 27705.
Dr. Lobach: Klesis LLC, 6 Harvey Place, Durham, NC 27705.
Author Contributions: Conception and design: T.J. Bright, A. Wong, J.W. Williams, G.D. Sanders, D. Lobach.
Analysis and interpretation of the data: T.J. Bright, A. Wong, R. Dhurjati, G. Samsa, V. Hasselblad, G.D. Sanders, D. Lobach.
Drafting of the article: T.J. Bright, A. Wong, E. Bristow, G.D. Sanders, D. Lobach.
Critical revision of the article for important intellectual content: T.J. Bright, G. Samsa, G.D. Sanders, D. Lobach.
Final approval of the article: T.J. Bright, G. Samsa, G.D. Sanders, D. Lobach.
Provision of study materials or patients: T.J. Bright, M.D. Musty.
Statistical expertise: G. Samsa, V. Hasselblad.
Obtaining of funding: G.D. Sanders, D. Lobach.
Administrative, technical, or logistic support: T.J. Bright, M.D. Musty, L. Wing, A.S. Kendrick, G.D. Sanders.
Collection and assembly of data: T.J. Bright, A. Wong, R. Dhurjati, E. Bristow, L. Bastian, R.R. Coeytaux, M.D. Musty.
This article was published at www.annals.org on 24 April 2012.
To the Editor
The article by Bright et al on Clinical Decision-Support Systems (CDSS) (1) was timely and interesting, but had an important omission. No mention was made of CDSS that focused on helping doctors make the correct diagnosis. Such systems have been broadly available for some 25 years, including QMR/Internist-I, ILIAD and DXplain. Currently, at least two general-purpose Diagnostic Decision Support systems, DXplain and ISABEL, are widely available. Reviews (2,3) have shown that these systems suggest important diseases the clinician had not previously considered, and a study (4) from the Mayo Clinic found that use of DXplain lowered length-of-stay and hospital costs.
1. Bright TJ et al “Effect of Clinical Decision-Support Systems” Ann Intern Med 2012;157:29-432.
2. Berner ES et al “Performance of Four Computer-Based Diagnostic Systems” New Engl J Med 1994; 330:1792-63.
3. Bond WF et al “Differential Diagnosis Generators: an Evaluation of Currently Available Computer Programs” J Gen Intern Med 2011;27:213-94.
4. Elkin PL et al “The Introduction of a Diagnostic Decision Support System (DXplain) into the Workflow of a Teaching Hospital Service can Decrease the Cost of Service for Diagnostically Challenging DRGs” Internat Jl Med Informatics 2010; 79:772-