Effect of Clinical Decision-Support Systems: A Systematic ReviewFREE
Publication: Annals of Internal Medicine
Volume 157, Number 1
Submit a Comment
Contributors must reveal any conflict of interest. Comments are moderated. Please see our information for authorsregarding comments on an Annals publication.
Abstract
Background:
Despite increasing emphasis on the role of clinical decision-support systems (CDSSs) for improving care and reducing costs, evidence to support widespread use is lacking.
Purpose:
To evaluate the effect of CDSSs on clinical outcomes, health care processes, workload and efficiency, patient satisfaction, cost, and provider use and implementation.
Data Sources:
MEDLINE, CINAHL, PsycINFO, and Web of Science through January 2011.
Study Selection:
Investigators independently screened reports to identify randomized trials published in English of electronic CDSSs that were implemented in clinical settings; used by providers to aid decision making at the point of care; and reported clinical, health care process, workload, relationship-centered, economic, or provider use outcomes.
Data Extraction:
Investigators extracted data about study design, participant characteristics, interventions, outcomes, and quality.
Data Synthesis:
148 randomized, controlled trials were included. A total of 128 (86%) assessed health care process measures, 29 (20%) assessed clinical outcomes, and 22 (15%) measured costs. Both commercially and locally developed CDSSs improved health care process measures related to performing preventive services (n = 25; odds ratio [OR], 1.42 [95% CI, 1.27 to 1.58]), ordering clinical studies (n = 20; OR, 1.72 [CI, 1.47 to 2.00]), and prescribing therapies (n = 46; OR, 1.57 [CI, 1.35 to 1.82]). Few studies measured potential unintended consequences or adverse effects.
Limitations:
Studies were heterogeneous in interventions, populations, settings, and outcomes. Publication bias and selective reporting cannot be excluded.
Conclusion:
Both commercially and locally developed CDSSs are effective at improving health care process measures across diverse settings, but evidence for clinical, economic, workload, and efficiency outcomes remains sparse. This review expands knowledge in the field by demonstrating the benefits of CDSSs outside of experienced academic centers.
Primary Funding Source:
Agency for Healthcare Research and Quality.
Despite increasing emphasis on clinical decision-support systems (CDSSs) in improving care and reducing costs, evidence supporting widespread use is limited. A CDSS is “any electronic system designed to aid directly in clinical decision making, in which characteristics of individual patients are used to generate patient-specific assessments or recommendations that are then presented to clinicians for consideration” (1). This review examines 3 types of decision-support interventions.
Classic CDSSs include alerts, reminders, order sets, drug-dose calculations that automatically remind the clinician of a specific action, or care summary dashboards that provide performance feedback on quality indicators. Information retrieval tools, such as an “infobutton” embedded in a clinical information system, are designed to aid clinicians in the search and retrieval of context-specific knowledge from information sources based on patient-specific information from a clinical information system. Knowledge resources, such as UpToDate, Epocrates, and MD Consult, consist of distilled primary literature that allows selection of content germane to a specific patient to facilitate decision making at the point of care or for a specific care situation.
Until recently, most studies of CDSSs came from 4 benchmark settings (Brigham and Women's Hospital/Partners HealthCare, the Department of Veterans Affairs, LDS Hospital/Intermountain Healthcare, and the Regenstrief Institute) (2). Although several reviews have examined the effects of CDSSs (1, 3–9), many questions remain about their impact. This systematic review adds to the literature by summarizing trials of CDSSs implemented in a clinical setting to aid decision making at the point of care or for a specific care situation.
Methods
We developed and followed a standard protocol for our review. Full details of our methods, search strategies, results, and conclusions are provided in a report commissioned by the Agency for Healthcare Research and Quality, available at www.effectivehealthcare.ahrq.gov.
Data Sources and Searches
We searched for studies done between January 1976 and January 2011 in MEDLINE accessed through PubMed, CINAHL, PsycINFO, and Web of Science.
Study Selection
We identified randomized trials of CDSSs implemented in a real clinical setting and used by health care providers to aid decision making at the point of care or for a specific care situation. Studies had to report at least one of the following types of outcomes: clinical (length of stay, morbidity, mortality, health-related quality of life, and adverse events), health care process (recommended preventive care, clinical study, or treatment ordered or completed), user workload and efficiency (user knowledge, number of patients seen, clinician workload, and efficiency), relationship-centered (patient satisfaction), economic (cost and cost-effectiveness), or use and implementation by a health care provider (acceptance, satisfaction, use, and implementation). We excluded studies that described nonelectronic CDSSs, included fewer than 50 participants, were not published in English, described closed-loop systems that did not involve a provider, evaluated systems that required mandatory compliance with the CDSS, or evaluated only the performance of the system as opposed to its effect on clinical practice.
Data Extraction and Quality Assessment
Data related to study setting and design, sample characteristics, intervention characteristics, comparators, and outcomes were extracted by 1 reviewer and confirmed by another. Two reviewers used a standardized approach to independently categorize the quality of individual studies as good, fair, or poor (10) and evaluated the overall strength of evidence for each outcome as high, moderate, low, or insufficient (11). Reviewers also identified issues related to study setting, interventions, and outcomes that limited applicability of evidence (10, 12).
Data Synthesis and Analysis
A priori–defined outcomes believed to be important in measuring the effect of CDSSs in improving clinical practice guided our synthesis process. Studies with a common outcome were grouped together to facilitate qualitative analysis. Quantitative analysis was done where 4 or more studies assessed the same outcome in the same manner, regardless of the specific CDSS intervention. Summary estimates were calculated by using the DerSimonian and Laird (13) random-effects model as implemented in Comprehensive Meta-analysis, version 2.2.055 (Biostat, Englewood, New Jersey).
Role of the Funding Source
Primary funding was provided by the Agency for Healthcare Research and Quality. The funding source formulated the initial study questions but otherwise had no role in the design, analysis, or interpretation of the data or in the decision to submit the manuscript for publication.
Results
We screened 15 176 abstracts, evaluated 1407 full-text articles, and included 160 articles, representing 148 unique studies (Appendix Figure). Appendix Table 1 summarizes important characteristics of the studies and their quality rating. A total of 128 studies (86%) assessed health care process measures, 29 (20%) assessed clinical outcomes, and 22 (15%) measured costs. Many studies (n = 51) were performed in environments with established health information technology (IT); many were multisite studies involving multiple institutions (n = 46).
Appendix Figure. Summary of evidence search and selection.
CDSS = clinical decision-support system; KQ = key question; RCT = randomized, controlled trial.
The Table summarizes the most important findings and the strength of supporting evidence for those findings, whereas Appendix Table 2 provides examples of interventions that assessed the outcomes of interest. Both commercially and locally developed CDSSs improved health care process measures related to performing preventive services, ordering clinical studies, and prescribing therapies. Few studies (n = 15) indicated conceptualization of potential unintended consequences or measured potential adverse effects of implementing decision-support tools.
Clinical Outcomes
Morbidity
Several studies assessed morbidity outcomes (14–38), such as hospitalizations, Apgar scores, surgical site infections, cardiovascular events, colorectal cancer, deep venous thrombosis, and hypoglycemia events. Topics addressed included diagnosis (16, 21, 29, 30, 34), pharmacotherapy (15, 18–22, 25, 28, 33), chronic disease management (14, 19, 20, 22, 26–28, 31, 32, 36, 38), laboratory test ordering (19, 37), immunizations (19, 35), preventive care (17, 19, 28–30, 37), and discharge planning (23, 24). Approximately 50% of the studies were performed in an academic setting.
Many studies evaluated locally developed interventions implemented in the ambulatory environment. Typical interventions were automatically delivered, system-initiated recommendations provided synchronously at the point of care to enable decision making during the health care provider–patient encounter. Three such interventions required a mandatory response (that is, required that the user respond to the given recommendation, whether that response was to accept or dismiss the recommendation or to modify the user's action) (17, 18, 37).
Comparators included usual care or no CDSS and the same CDSS with additional features. Limitations included short follow-up, low statistical power to detect important differences, and the potential for contamination of providers in control groups that improved because of knowledge of interventions.
Meta-analysis of these heterogeneous studies (n = 16) suggested that CDSSs improved morbidity outcomes (relative risk, 0.88 [95% CI, 0.80 to 0.96]). We rated this level of evidence as moderate. Most studies were good quality, and many of the interventions were evaluated in multiple institutions. However, the interventions were often paper-based or standalone systems implemented in academic or Veterans Affairs settings and were targeted toward a single condition.
Mortality
Seven studies (17, 20–22, 33, 39, 40) reported mortality outcomes. Issues addressed included diagnosis (21), pharmacotherapy (20–22, 33, 40), chronic disease management (20, 22), preventing deep venous thrombosis (17), and detecting and notifying clinicians of critical laboratory values (39). Most CDSSs were locally developed and integrated into a computerized physician order entry (CPOE) or electronic health record (EHR) system and had system-initiated recommendations delivered synchronously at the point of care that did not require a clinician response.
Interventions were evaluated against usual care or no CDSS, except for 2 studies (20, 22) that compared the same intervention with additional features. Limitations included small sample size, duration shorter than 1 year, and possible contamination of control providers.
Meta-analysis of these heterogeneous studies (n = 6) suggested no significant effect of CDSSs on mortality, although CIs were wide (odds ratio [OR], 0.79 [CI, 0.54 to 1.15]). Of note, 2 studies reported a significant reduction in mortality with use of CDSSs (20, 22). We rated the overall strength of evidence related to mortality outcomes as low. Most studies were conducted in a single academic or Veterans Affairs setting with a comprehensive, well-established health IT infrastructure.
Adverse Events
Five studies assessed the effectiveness of CDSSs in reducing or preventing adverse events (23, 24, 39–42). Studies were mostly implemented in an academic, inpatient setting. Studies evaluated the effect of these interventions to improve the timing of warfarin therapy (41), improve discharge planning (23, 24), prevent adverse drug events (42), detect critical laboratory values (39), and detect potentially inappropriate or inadequate antimicrobial therapy (40).
Typical interventions were locally developed, were integrated into a CPOE or an EHR system, and automatically delivered system-imitated recommendations in real time to enable decision making during the provider–patient encounter. Only 1 study clearly required a mandatory response (39).
All of the CDSSs were evaluated against usual care or no CDSS. Limitations included evaluation at a single institution, evaluation periods less than 1 year, and potential improvement in physician performance because of their knowledge of the intervention.
Meta-analysis of these heterogeneous studies estimated a relative risk of 1.01 (CI, 0.90 to 1.14), and neither this summary nor any individual studies demonstrated a significant effect. We rated this level of evidence as low. Most studies were good quality, and 2 were conducted at multiple sites; however, these interventions primarily contained locally developed knowledge, and results may not be generalizable to nonteaching settings.
Health Care Process Measures
Recommended Preventive Care Service Ordered or Completed
Forty-three studies examined the effect of CDSSs on the rates of ordering or completing recommended preventive care services (17, 19, 27, 28, 35, 37, 43–82). Most studies were conducted in the academic or ambulatory environment. Topics addressed included diagnosis (43, 47, 62, 71, 82), pharmacotherapy (19, 28, 49, 53, 54, 70, 77, 80), chronic disease management (19, 26, 28, 43, 44, 48, 51, 70, 72, 74, 76), laboratory test ordering (19, 37, 55, 60, 70, 71, 74, 75, 80, 81), preventive care (17, 19, 28, 37, 43, 45, 48, 52–60, 62–65, 68–71, 73–75, 79–82), immunizations (19, 35, 48–50, 52, 56, 66, 67, 78, 79, 81), and initiating discussions with patients (60, 61, 72).
Most interventions were locally developed, paper-based, or standalone systems and automatically delivered recommendations in real time to enable decision making during the health care provider–patient encounter. Only 7 of the interventions required a mandatory response (17, 37, 47, 49) or justification (64, 65, 68, 69, 75) for not adhering to the recommendation.
Comparators included usual care or no CDSS, direct comparison against the same CDSS with additional features, or comparison of the same CDSS for different conditions. Limitations included sparse data measuring patient or economic outcomes; few assessments of long-term outcomes of interventions; and the Hawthorne effect, which probably stimulated more comprehensive preventive care across groups.
In meta-analysis of these 25 heterogeneous studies (17, 27, 28, 35, 37, 43–63), the effect of CDSSs on preventive care services was significant (OR, 1.42 [CI, 1.27 to 1.58]) (Figure 1). We rated this level of evidence as high. Approximately one half of the studies were good quality, one third were evaluated in multicenter trials, and one fourth addressed multiple clinical conditions. However, most CDSSs were locally developed, not integrated into a CPOE or an EHR, and evaluated in academic medical centers, all of which can affect the generalizability of these findings.
Figure 1. Results of studies that examined whether recommended preventive care services were ordered.
Studies reporting the odds ratio of adhering to recommendations for ordering or completing preventive care services of CDSS vs. control groups. In the 25 studies comparing CDSS with control groups, the random-effects–combined odds ratio of adherence to preventive care recommendations was 1.42 (95% CI, 1.27 to 1.58). CDSS = clinical decision-support system.
Recommended Clinical Study Ordered or Completed
Twenty-nine studies evaluated the effect of CDSSs on ordering and completing recommended clinical studies (26, 31, 32, 83–109). Many studies were conducted in the academic setting, and most were evaluated in the ambulatory environment. Topics addressed included diagnosis (85, 89–91, 95, 98, 101–103), pharmacotherapy (94, 98), chronic disease management (26, 31, 32, 84, 85, 103), laboratory test ordering (83, 87, 89, 92–94, 96, 97, 99–101, 104–108), initiating discussions with patients (108, 109), and additional clinical tasks (86, 88, 90, 109).
Typical interventions were locally developed, were integrated CDSS recommendations in a CPOE or an EHR system, and automatically delivered system-initiated recommendations to enable decision making during the provider–patient encounter. Eight interventions required a mandatory response (90, 99, 100, 104, 106, 108) or justification (83, 105) for not adhering to the recommendation.
Comparators included usual care or no CDSS, direct comparison against the same CDSS with additional features, or comparison of the same CDSS for different conditions. Limitations of the evidence base included diverse metrics to assess adherence to ordering and completing a recommended action, studies not designed to evaluate the clinical or economic outcomes associated with the CDSS interventions, and limited evidence of the effect of CDSSs on a broad set of conditions.
Meta-analysis of these 20 heterogeneous studies (26, 31, 32, 83, 84, 88, 89, 91, 92, 94–96, 98–103, 105, 108, 109) found a significant effect of CDSSs on ordering or completing of clinical studies (OR, 1.72 [CI, 1.47 to 2.00]) (Figure 2). We rated this level of evidence as moderate. Most studies were good quality, approximately one third were implemented in multiple sites, and almost one fourth included a direct comparison of the effectiveness of the CDSS against the same intervention with additional features. However, we noted a strong suggestion of publication bias in these studies, and these results may not be generalizable to all settings because most studies were either evaluated in environments with a well-established health IT infrastructure or conducted outside of the United States.
Figure 2. Results of studies that examined whether recommended clinical studies were ordered.
Studies reporting the odds ratio of adhering to recommendations for ordering or completing recommended clinical studies of CDSS vs. control groups. In the 20 studies comparing CDSS with control groups, the random-effects–combined odds ratio of adherence to clinical study recommendations was 1.72 (95% CI, 1.47 to 2.00). CDSS = clinical decision-support system.
Recommended Treatment Ordered or Prescribed
Sixty-seven studies evaluated the effect of CDSSs on ordering and prescribing therapy (14, 18, 20–22, 25–28, 33, 36, 38–41, 43, 44, 53, 54, 70, 80, 82, 84, 88, 94, 98, 110–154). Many studies were conducted in the academic setting, and most were evaluated in the ambulatory environment. Topics addressed included diagnosis (21, 43, 82, 98, 112, 123, 129, 138, 151, 152), pharmacotherapy (18, 20–22, 25, 28, 33, 40, 53, 54, 70, 80, 94, 98, 110, 111, 114, 116, 117, 119, 122, 123, 125–134, 136, 138–150, 153, 154), laboratory test ordering (70, 80, 94, 110, 130), chronic disease management (14, 20, 22, 26–28, 36, 38, 43, 44, 70, 84, 110, 112, 113, 115, 118, 120, 121, 123–125, 132, 133, 135, 141), preventive care (28, 43, 53, 54, 70, 80, 82, 120, 148), and additional clinical tasks (39, 41, 82, 88, 110, 137, 151, 152).
Typical interventions were locally developed, were integrated into a CPOE or an EHR system, and automatically delivered system-initiated recommendations in real time to enable decision making during the provider–patient encounter. Eighteen CDSSs required a mandatory response (18, 39, 119, 122, 134, 139–143, 147, 148, 151, 152) or justification (112, 113, 136, 137, 145) for not adhering to the recommendation. Limitations included inadequate follow-up periods to observe sustained results, sparse data demonstrating how changes in clinician ordering and prescribing led to improvements in clinical or economic outcomes, and study designs that did not capture the extent to which nonadherence with recommended therapy resulted in adverse events.
Meta-analysis of 46 heterogeneous studies (14, 18, 20–22, 25–28, 33, 36, 38, 40, 43, 44, 53, 54, 70, 82, 84, 94, 98, 110, 112–117, 121–124, 129, 130, 134–136, 141–144, 146, 148, 151–153) showed that intervention providers with decision support were more likely to order the appropriate treatment or therapy (OR, 1.57 [CI, 1.35 to 1.82]) (Figure 3). We rated this level of evidence as high. Most studies were good quality, and most were evaluated in multisite trials. However, generalizability may be limited because most studies were implemented in the ambulatory environment, were evaluated in settings where clinicians were experienced EHR users or provided care in an established health IT infrastructure, and incorporated knowledge that was targeted toward specific conditions.
Figure 3. Results of studies that examined whether recommended treatments were ordered.
Studies reporting the odds ratio of adhering to recommendations for ordering or prescribing treatment of CDSS vs. control groups. In the 46 studies comparing CDSS with control groups, the random-effects–combined odds ratio of adherence to treatment recommendations was 1.57 (95% CI, 1.35 to 1.82). CDSS = clinical decision-support system.
User Workload and Efficiency Outcomes
Evidence on the effect of CDSSs on clinician knowledge or improved confidence in managing patient care was insufficient (71, 72, 86, 155, 156). Seven studies examined the effect of CDSSs on efficiency (23, 24, 40, 106, 141, 155–157). Limitations included contamination of clinicians in the control group that improved because of knowledge of the intervention, evaluation periods that were too brief to demonstrate an effect on efficiency, and small clinician sample sizes.
We rated the level of evidence as low. Most interventions contained locally developed knowledge, such as protocols or algorithms derived on the basis of local performance, quality, and outcome data not representative of other sites, and were evaluated in academic settings.
Economic Outcomes
Cost
Twenty-two studies reported costs (21, 26, 27, 31, 32, 36, 40, 43, 53, 54, 71, 83, 90, 106, 109, 113, 118, 130, 141, 158–163). Objectives of the CDSSs included diagnosis (21, 43, 71, 90, 160), pharmacotherapy (21, 40, 53, 54, 130, 141), chronic disease management (26, 27, 31, 32, 36, 43, 113, 118, 141, 161, 162), laboratory test ordering (71, 83, 106, 130, 160, 163), preventive care (43, 53, 54, 71, 158, 159), initiating discussions with patients (109, 159), and additional clinical tasks (90, 109). One study reported reduced hospitalization expenses with CDSS use (31, 32), and 12 studies reported that use had a positive effect on costs compared with control groups and other non-CDSS groups (21, 27, 40, 53, 54, 83, 90, 106, 113, 130, 141, 159, 160).
Modest evidence from academic and community inpatient and ambulatory settings showed that locally and commercially developed CDSSs had lower treatment costs, total costs, and reduced costs compared with control groups and other non-CDSS intervention groups. Most studies were conducted in the academic ambulatory setting and evaluated locally developed, integrated CDSSs in CPOE or EHR systems that automatically delivered system-initiated recommendations synchronously at the point of care and did not require a mandatory clinician response.
Cost-Effectiveness
Six studies (53, 54, 56, 57, 78, 95, 96, 161, 162) examined the cost-effectiveness of CDSSs or their effect on cost-effectiveness of care. These demonstrated conflicting findings, with 3 studies suggesting that CDSSs were cost-effective (53, 54, 78, 95) and 3 reporting that CDSSs were not cost-effective (56, 57, 161, 162). Objectives included diagnosis (95), pharmacotherapy (53, 54), chronic disease management (161, 162), preventive care (53, 54, 56, 57), and immunizations (56, 78).
Use and Implementation Outcomes
Twenty-four studies assessed the effect of provider acceptance of CDSSs (19, 41, 55, 62, 75, 90, 105, 113, 119, 120, 136, 137, 145–147, 149, 158, 159, 164–170). Topics addressed included diagnosis (62, 90, 166), pharmacotherapy (19, 119, 136, 145–147, 149, 164, 165), chronic disease management (19, 113, 120, 168–170), laboratory test ordering (19, 55, 75, 105), preventive care (19, 55, 62, 75, 120, 147, 158, 159, 167), immunizations (19), initiating discussions with patients (159), and additional clinical tasks (41, 90, 137).
Comparators included usual care or no CDSS and direct comparison with the same CDSS with additional features. One half of the studies required a mandatory response (55, 90, 119, 147, 166) or justification (75, 105, 113, 120, 136, 137, 145) for not adhering to the recommendation; however, there was no significant effect on provider acceptance. Limitations included an inconsistent definition of provider acceptance; small sample sizes; and scarce data on clinical outcomes, such as morbidity, length of stay, or adverse events. We rated this level of evidence as low. Most of these studies were fair quality and were evaluated in academic medical settings with established health IT infrastructures and experienced EHR users, which may limit the generalizability of the findings.
Provider satisfaction with CDSSs was examined in 19 studies (14, 23–25, 37, 43, 80, 86, 105, 109, 112, 119, 127, 128, 141, 151–153, 155, 156, 158, 165). Topics addressed included diagnosis (43, 112, 151, 152), pharmacotherapy (25, 80, 119, 127, 128, 141, 153, 165), chronic disease management (14, 43, 112, 141), laboratory test ordering (37, 80, 105), preventive care (37, 43, 80), and initiating discussions with patients (109).
Comparators included usual care or no CDSS and direct comparison with the same CDSS with additional features. Seven CDSSs required a mandatory response (37, 119, 141, 151, 152, 155) or justification (105, 112) for not adhering to the recommendation. Limitations included the narrow assessment of the role of provider satisfaction with CDSSs on patient-specific outcomes and small sample sizes of clinicians.
Twelve studies demonstrated provider satisfaction with CDSSs (25, 37, 80, 109, 112, 119, 127, 128, 141, 155, 156, 158, 165); 4 showed a significant effect of satisfaction among intervention providers compared with control providers (109, 112, 155, 165). Provider dissatisfaction with CDSSs was also reported in 6 studies (14, 43, 86, 105, 151–153). We rated this level of evidence as moderate. Most studies were good quality and evaluated CDSSs integrated into CPOE or EHR systems in multiple interventions outside of environments with an established and robust health IT. However, most CDSSs were locally developed and implemented in the ambulatory setting.
Seventeen studies examined provider use of CDSSs by using such metrics as the number of times the CDSS was accessed by the clinician or provided a recommendation to the clinician (51, 71, 80, 86, 107, 110, 117, 119, 123, 138, 142, 145, 156, 165, 168–172). Objectives included diagnosis (71, 123, 138), pharmacotherapy (80, 110, 117, 119, 123, 138, 142, 145, 165), chronic disease management (51, 110, 123, 168–172), laboratory test ordering (71, 80, 107, 110), preventive care (71, 80), and additional clinical tasks (86, 110, 156).
Comparators included usual care or no CDSS, direct comparison with the same CDSS with additional features, or comparison of the same CDSS for different conditions. Limitations included sparse data demonstrating how provider use translated into more appropriate patient care and small sample sizes of clinicians. We rated this level of evidence as low.
Among the 12 studies (80, 107, 117, 119, 123, 138, 145, 168–172) that provided statistical data about provider use, 8 (80, 110, 119, 123, 145, 165, 168–170) documented low use (<50% of the clinician's time or of patient visits) or that less than 50% of clinicians used the CDSS or received alerts to guide therapeutic action. Most of these studies were fair quality and evaluated locally developed interventions in multiple community and ambulatory settings. Additional results are available from the technical report at www.effectivehealthcare.ahrq.gov.
Discussion
Our systematic review investigated the continuum of information support for clinical care, including traditional CDSSs, as well as information retrieval systems and knowledge resources developed for access at the point of care. Studies were primarily conducted outside of institutions with an established health IT infrastructure. Most interventions targeted specific medical conditions and were evaluated in single settings.
Clinical decision support had a favorable effect on prescribing treatments, facilitating preventive care services, and ordering clinical studies across diverse venues and systems. This finding contrasts with that of another review (2), which showed that most reports of successful CDSS implementation were based on locally developed systems at 4 sites.
Evidence demonstrating positive effects of CDSSs on clinical and economic outcomes remains surprisingly sparse, although this could be because of the relative difficulty of implementing randomized, controlled trials in real clinical settings, as well as the logistics of measuring the direct clinical effect of CDSSs. Evidence was also limited in showing an effect of CDSSs on clinical workload and efficiency. Furthermore, available evidence is insufficient to draw conclusions about the potential negative effect of implementing decision-support tools, which is necessary to truly fulfill the goal of evaluating these interventions and to better address implementation challenges (173).
Our findings are important in light of the increasing political interest and financial investment of the U.S. government in resources for health IT. Meaningful use of CDSSs needs to be objectively informed about the role that they can and should play in reshaping health care delivery. Further understanding is increasingly important to optimally define their role in the context of meaningful use for EHRs.
Our systematic review has several limitations. The heterogeneity of the studies limited general observations about CDSSs. We minimized this limitation in our meta-analyses by including studies that assessed the same outcome in the same manner. Although this investigation was a comprehensive review of randomized, controlled trials that provided the best evidence on CDSS effectiveness, these studies may provide less information about issues related to CDSS implementation, effect on workflow, and factors affecting usability.
Most studies (76%) evaluated the effectiveness of the intervention by using usual care rather than a direct comparator, which may contribute to more positive results. Finally, we acknowledge the possibility of selective reporting or publication bias. However, a recent review by Buntin and colleagues (173) found that 62% of included studies on health IT reported positive effects where the technology was associated with improvement in 1 or more aspects of care (173). Formal assessment by using funnel plots found no consistent bias for most outcomes, except for ordering or completing of clinical studies, where there was a strong suggestion of publication bias.
Significant research is still required to promote widespread use of CDSSs and to augment their clinical effectiveness. Future studies should investigate how to expand CDSS content to accommodate multiple comorbid conditions simultaneously and to determine which members of the care team should receive clinical decision support, what effect CDSSs have on clinical and economic outcomes, and how CDSSs can be most effectively integrated into workflow and deployed across diverse settings. Further work is also needed to understand how CDSSs can aid in the transformation of care delivery models, such as accountable care organizations and patient-centered medical homes; how to incorporate CDSSs into workflow tools, such as medical registries and provider–provider messaging capabilities; and how to integrate CDSSs with workflow-oriented quality improvement programs.
Promoting extensive use of CDSSs will require a better definition of the clinical decision-support infrastructure. Such infrastructure could include consistent underlying frameworks for describing CDSSs, such as the “Clinical Decision Support Five Rights” (174), to aid in the aggregation and synthesis of results; development and evaluation of models for porting CDSSs across settings; and improved identification of characteristics of the environment and workflow into which a CDSS is deployed, as well as characteristics of the intended users.
In summary, evidence demonstrated the efficacy of CDSSs on health care process outcomes across diverse settings by using both commercially and locally developed systems, but data showing an effect on clinical and economic outcomes were sparse. Broad penetration of clinical decision-support tools will require aggressively seeking a better understanding of what the right information is and when and how it should be delivered to the right person, and a critical examination of the unintended consequences of CDSS implementation.
References
1.
Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ. 2005;330:765. [PMID: 15767266]
2.
Chaudhry B, Wang J, Wu S, Maglione M, Mojica W, Roth E, et al. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med. 2006;144:742-52. [PMID: 16702590]
3.
Bryan C, Boren SA. The use and effectiveness of electronic clinical decision support tools in the ambulatory/primary care setting: a systematic review of the literature. Inform Prim Care. 2008;16:79-91. [PMID: 18713524]
4.
Garg AX, Adhikari NK, McDonald H, Rosas-Arellano MP, Devereaux PJ, Beyene J, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA. 2005;293:1223-38. [PMID: 15755945]
5.
Grimshaw J, Freemantle N, Wallace S, Russell I, Hurwitz B, Watt I, et al. Developing and implementing clinical practice guidelines. Qual Health Care. 1995;4:55-64. [PMID: 10142039]
6.
Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA. 1998;280:1339-46. [PMID: 9794315]
7.
Sintchenko V, Magrabi F, Tipper S. Are we measuring the right end-points? Variables that affect the impact of computerised decision support on patient outcomes: a systematic review. Med Inform Internet Med. 2007;32:225-40. [PMID: 17701828]
8.
Shojania KG, Jennings A, Mayhew A, Ramsay C, Eccles M, Grimshaw J. Effect of point-of-care computer reminders on physician behaviour: a systematic review. Can Med Assoc J. 2010;182:216-25.
9.
Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med. 2003;163:1409-16. [PMID: 12824090]
10.
Agency for Healthcare Research and Quality. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. Rockville, MD: Agency for Healthcare Research and Quality; 2011. Accessed at www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productid=318 on 24 January 2012.
11.
Owens DK, Lohr KN, Atkins D, Treadwell JR, Reston JT, Bass EB, et al. AHRQ series paper 5: grading the strength of a body of evidence when comparing medical interventions—agency for healthcare research and quality and the effective health-care program. J Clin Epidemiol. 2010;63:513-23. [PMID: 19595577]
12.
Atkins D, Chang SM, Gartlehner G, Buckley DI, Whitlock EP, Berliner E, et al. Assessing applicability when comparing medical interventions: AHRQ and the Effective Health Care Program. J Clin Epidemiol. 2011;64:1198-207. [PMID: 21463926]
13.
DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials. 1986;7:177-88. [PMID: 3802833]
14.
McCowan C, Neville RG, Ricketts IW, Warner FC, Hoskins G, Thomas GE. Lessons from a randomized controlled trial designed to evaluate computer decision support software to improve the management of asthma. Med Inform Internet Med. 2001;26:191-201. [PMID: 11706929]
15.
Cavalcanti AB, Silva E, Pereira AJ, Caldeira-Filho M, Almeida FP, Westphal GA, et al. A randomized controlled trial comparing a computer-assisted insulin infusion protocol with a strict and a conventional protocol for glucose control in critically ill patients. J Crit Care. 2009;24:371-8. [PMID: 19592202]
16.
Kline JA, Zeitouni RA, Hernandez-Nino J, Jones AE. Randomized trial of computerized quantitative pretest probability in low-risk chest pain patients: effect on safety and resource use. Ann Emerg Med. 2009;53:727-35.e1. [PMID: 19135281]
17.
Kucher N, Koo S, Quiroz R, Cooper JM, Paterno MD, Soukonnikov B, et al. Electronic alerts to prevent venous thromboembolism among hospitalized patients. N Engl J Med. 2005;352:969-77. [PMID: 15758007]
18.
Zanetti G, Flanagan HL Jr, Cohn LH, Giardina R, Platt R. Improvement of intraoperative antibiotic prophylaxis in prolonged cardiac surgery by automated alerts in the operating room. Infect Control Hosp Epidemiol. 2003;24:13-6. [PMID: 12558230]
19.
McDonald CJ, Hui SL, Smith DM, Tierney WM, Cohen SJ, Weinberger M, et al. Reminders to physicians from an introspective computer medical record. A two-year randomized trial. Ann Intern Med. 1984;100:130-8. [PMID: 6691639]
20.
Roumie CL, Elasy TA, Greevy R, Griffin MR, Liu X, Stone WJ, et al. Improving blood pressure control through provider education, provider alerts, and patient education: a cluster randomized trial. Ann Intern Med. 2006;145:165-75. [PMID: 16880458]
21.
Paul M, Andreassen S, Tacconelli E, Nielsen AD, Almanasreh N, Frank U, et al; TREAT Study Group. Improving empirical antibiotic treatment using TREAT, a computerized decision support system: cluster randomized trial. J Antimicrob Chemother. 2006;58:1238-45. [PMID: 16998208]
22.
Ansari M, Shlipak MG, Heidenreich PA, Van Ostaeyen D, Pohl EC, Browner WS, et al. Improving guideline adherence: a randomized trial evaluating strategies to increase beta-blocker use in heart failure. Circulation. 2003;107:2799-804. [PMID: 12756157]
23.
Graumlich JF, Novotny NL, Nace GS, Aldag JC. Patient and physician perceptions after software-assisted hospital discharge: cluster randomized trial. J Hosp Med. 2009;4:356-63. [PMID: 19621342]
24.
Graumlich JF, Novotny NL, Stephen Nace G, Kaushal H, Ibrahim-Ali W, Theivanayagam S, et al. Patient readmissions, emergency visits, and adverse events after software-assisted discharge from hospital: cluster randomized trial. J Hosp Med. 2009;4:11-9. [PMID: 19479782]
25.
Heidenreich PA, Gholami P, Sahay A, Massie B, Goldstein MK. Clinical reminders attached to echocardiography reports of patients with reduced left ventricular ejection fraction increase use of beta-blockers: a randomized trial. Circulation. 2007;115:2829-34. [PMID: 17515459]
26.
Tierney WM, Overhage JM, Murray MD, Harris LE, Zhou XH, Eckert GJ, et al. Can computer-generated evidence-based care suggestions enhance evidence-based management of asthma and chronic obstructive pulmonary disease? A randomized, controlled trial. Health Serv Res. 2005;40:477-97. [PMID: 15762903]
27.
Tierney WM, Overhage JM, Murray MD, Harris LE, Zhou XH, Eckert GJ, et al. Effects of computerized guidelines for managing heart disease in primary care. J Gen Intern Med. 2003;18:967-76. [PMID: 14687254]
28.
Gilutz H, Novack L, Shvartzman P, Zelingher J, Bonneh DY, Henkin Y, et al. Computerized community cholesterol control (4C): meeting the challenge of secondary prevention. Isr Med Assoc J. 2009;11:23-9. [PMID: 19344008]
29.
Holt TA, Thorogood M, Griffiths F, Munday S. Protocol for the ‘e-Nudge trial’: a randomised controlled trial of electronic feedback to reduce the cardiovascular risk of individuals in general practice [ISRCTN64828380]. Trials. 2006;7:11. [PMID: 16646967]
30.
Holt TA, Thorogood M, Griffiths F, Munday S, Friede T, Stables D. Automated electronic reminders to facilitate primary cardiovascular disease prevention: randomised controlled trial. Br J Gen Pract. 2010;60:137-43. [PMID: 20353659]
31.
Khan S, Maclean CD, Littenberg B. The effect of the Vermont Diabetes Information System on inpatient and emergency room use: results from a randomized trial. Health Outcomes Res Med. 2010;1:61-e66. [PMID: 20975923]
32.
Maclean CD, Gagnon M, Callas P, Littenberg B. The Vermont diabetes information system: a cluster randomized trial of a population based decision support system. J Gen Intern Med. 2009;24:1303-10. [PMID: 19862578]
33.
Brier ME, Gaweda AE, Dailey A, Aronoff GR, Jacobs AA. Randomized trial of model predictive control for improved anemia management. Clin J Am Soc Nephrol. 2010;5:814-20. [PMID: 20185598]
34.
Hamilton E, Platt R, Gauthier R, McNamara H, Miner L, Rothenberg S, et al. The effect of computer-assisted evaluation of labor on cesarean rates. J Healthc Qual. 2004;26:37-44. [PMID: 14763319]
35.
McDonald CJ, Hui SL, Tierney WM. Effects of computer reminders for influenza vaccination on morbidity during influenza epidemics. MD Comput. 1992;9:304-12. [PMID: 1522792]
36.
Murray MD, Harris LE, Overhage JM, Zhou XH, Eckert GJ, Smith FE, et al. Failure of computerized treatment suggestions to improve health outcomes of outpatients with uncomplicated hypertension: results of a randomized controlled trial. Pharmacotherapy. 2004;24:324-37. [PMID: 15040645]
37.
Sequist TD, Zaslavsky AM, Marshall R, Fletcher RH, Ayanian JZ. Patient and physician reminders to promote colorectal cancer screening: a randomized controlled trial. Arch Intern Med. 2009;169:364-71. [PMID: 19237720]
38.
Subramanian U, Fihn SD, Weinberger M, Plue L, Smith FE, Udris EM, et al. A controlled trial of including symptom data in computer-based care suggestions for managing patients with chronic heart failure. Am J Med. 2004;116:375-84. [PMID: 15006586]
39.
Kuperman GJ, Teich JM, Tanasijevic MJ, Ma'Luf N, Rittenberg E, Jha A, et al. Improving response to critical laboratory results with automation: results of a randomized controlled trial. J Am Med Inform Assoc. 1999;6:512-22. [PMID: 10579608]
40.
McGregor JC, Weekes E, Forrest GN, Standiford HC, Perencevich EN, Furuno JP, et al. Impact of a computerized clinical decision support system on reducing inappropriate antimicrobial use: a randomized controlled trial. J Am Med Inform Assoc. 2006;13:378-84. [PMID: 16622162]
41.
Fihn SD, McDonell MB, Vermes D, Henikoff JG, Martin DC, Callahan CM, et al. A computerized intervention to improve timing of outpatient follow-up: a multicenter randomized trial in patients treated with warfarin. National Consortium of Anticoagulation Clinics. J Gen Intern Med. 1994;9:131-9. [PMID: 8195911]
42.
Gurwitz JH, Field TS, Rochon P, Judge J, Harrold LR, Bell CM, et al. Effect of computerized provider order entry with clinical decision support on adverse drug events in the long-term care setting. J Am Geriatr Soc. 2008;56:2225-33. [PMID: 19093922]
43.
Apkon M, Mattera JA, Lin Z, Herrin J, Bradley EH, Carbone M, et al. A randomized outpatient trial of a decision-support information technology tool. Arch Intern Med. 2005;165:2388-94. [PMID: 16287768]
44.
Bertoni AG, Bonds DE, Chen H, Hogan P, Crago L, Rosenberger E, et al. Impact of a multifaceted intervention on cholesterol management in primary care practices: guideline adherence for heart health randomized trial. Arch Intern Med. 2009;169:678-86. [PMID: 19364997]
45.
Burack RC, Gimotty PA, George J, McBride S, Moncrease A, Simon MS, et al. How reminders given to patients and physicians affected pap smear use in a health maintenance organization: results of a randomized controlled trial. Cancer. 1998;82:2391-400. [PMID: 9635532]
46.
Burack RC, Gimotty PA, Simon M, Moncrease A, Dews P. The effect of adding Pap smear information to a mammography reminder system in an HMO: results of randomized controlled trial. Prev Med. 2003;36:547-54. [PMID: 12689799]
47.
Cannon DS, Allen SN. A comparison of the effects of computer and manual reminders on compliance with a mental health clinical practice guideline. J Am Med Inform Assoc. 2000;7:196-203. [PMID: 10730603]
48.
Demakis JG, Beauchamp C, Cull WL, Denwood R, Eisen SA, Lofgren R, et al. Improving residents' compliance with standards of ambulatory care: results from the VA Cooperative Study on Computerized Reminders. JAMA. 2000;284:1411-6. [PMID: 10989404]
49.
Dexter PR, Perkins S, Overhage JM, Maharry K, Kohler RB, McDonald CJ. A computerized reminder system to increase the use of preventive care for hospitalized patients. N Engl J Med. 2001;345:965-70. [PMID: 11575289]
50.
Dexter PR, Perkins SM, Maharry KS, Jones K, McDonald CJ. Inpatient computer-based standing orders vs physician reminders to increase influenza and pneumococcal vaccination rates: a randomized trial. JAMA. 2004;292:2366-71. [PMID: 15547164]
51.
Eccles M, McColl E, Steen N, Rousseau N, Grimshaw J, Parkin D, et al. Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial. BMJ. 2002;325:941. [PMID: 12399345]
52.
Frank O, Litt J, Beilby J. Opportunistic electronic reminders. Improving performance of preventive care in general practice. Aust Fam Physician. 2004;33:87-90. [PMID: 14988972]
53.
Fretheim A, Aaserud M, Oxman AD. Rational prescribing in primary care (RaPP): economic evaluation of an intervention to improve professional practice. PLoS Med. 2006;3:216. [PMID: 16737349]
54.
Fretheim A, Oxman AD, Håvelsrud K, Treweek S, Kristoffersen DT, Bjørndal A. Rational prescribing in primary care (RaPP): a cluster randomized trial of a tailored intervention. PLoS Med. 2006;3:134. [PMID: 16737346]
55.
Litzelman DK, Dittus RS, Miller ME, Tierney WM. Requiring physicians to respond to computerized reminders improves their compliance with preventive care protocols. J Gen Intern Med. 1993;8:311-7. [PMID: 8320575]
56.
McDowell I, Newell C, Rosser W. Comparison of three methods of recalling patients for influenza vaccination. CMAJ. 1986;135:991-7. [PMID: 3093045]
57.
McDowell I, Newell C, Rosser W. Computerized reminders to encourage cervical screening in family practice. J Fam Pract. 1989;28:420-4. [PMID: 2495337]
58.
Overhage JM, Tierney WM, McDonald CJ. Computer reminders to implement preventive care guidelines for hospitalized patients. Arch Intern Med. 1996;156:1551-6. [PMID: 8687263]
59.
Price M. Can hand-held computers improve adherence to guidelines? A (Palm) Pilot study of family doctors in British Columbia. Can Fam Physician. 2005;51:1506-7. [PMID: 16926943]
60.
Taylor V, Thompson B, Lessler D, Yasui Y, Montano D, Johnson KM, et al. A clinic-based mammography intervention targeting inner-city women. J Gen Intern Med. 1999;14:104-11. [PMID: 10051781]
61.
Unrod M, Smith M, Spring B, DePue J, Redd W, Winkel G. Randomized controlled trial of a computer-based, tailored intervention to increase smoking cessation counseling by primary care physicians. J Gen Intern Med. 2007;22:478-84. [PMID: 17372796]
62.
Dykes PC, Carroll DL, Hurley A, Lipsitz S, Benoit A, Chang F, et al. Fall prevention in acute care hospitals: a randomized trial. JAMA. 2010;304:1912-8. [PMID: 21045097]
63.
Chambers CV, Balaban DJ, Carlson BL, Ungemack JA, Grasberger DM. Microcomputer-generated reminders. Improving the compliance of primary care physicians with mammography screening guidelines. J Fam Pract. 1989;29:273-80. [PMID: 2769192]
64.
Burack RC, Gimotty PA. Promoting screening mammography in inner-city settings. The sustained effectiveness of computerized reminders in a randomized controlled trial. Med Care. 1997;35:921-31. [PMID: 9298081]
65.
Burack RC, Gimotty PA, George J, Stengle W, Warbasse L, Moncrease A. Promoting screening mammography in inner-city settings: a randomized controlled trial of computerized reminders as a component of a program to facilitate mammography. Med Care. 1994;32:609-24. [PMID: 8189778]
66.
Fiks AG, Hunter KF, Localio AR, Grundmeier RW, Bryant-Stephens T, Luberti AA, et al. Impact of electronic health record-based alerts on influenza vaccination for children with asthma. Pediatrics. 2009;124:159-69. [PMID: 19564296]
67.
Flanagan JR, Doebbeling BN, Dawson J, Beekmann S. Randomized study of online vaccine reminders in adult primary care. Proc AMIA Symp. 1999;:755-9. [PMID: 10566461]
68.
Fordham D, McPhee SJ, Bird JA, Rodnick JE, Detmer WM. The Cancer Prevention Reminder System. MD Comput. 1990;7:289-95. [PMID: 2243544]
69.
McPhee SJ, Bird JA, Jenkins CN, Fordham D. Promoting cancer screening. A randomized, controlled trial of three interventions. Arch Intern Med. 1989;149:1866-72. [PMID: 2764657]
70.
Gill JM, Chen YX, Glutting JJ, Diamond JJ, Lieberman MI. Impact of decision support in electronic medical records on lipid management in primary care. Popul Health Manag. 2009;12:221-6. [PMID: 19848563]
71.
Hobbs FD, Delaney BC, Carson A, Kenkre JE. A prospective controlled trial of computerized decision support for lipid management in primary care. Fam Pract. 1996;13:133-7. [PMID: 8732323]
72.
Holbrook A, Thabane L, Keshavjee K, Dolovich L, Bernstein B, Chan D, et al; COMPETE II Investigators. Individualized electronic decision support and reminders to improve diabetes care in the community: COMPETE II randomized trial. CMAJ. 2009;181:37-44. [PMID: 19581618]
73.
Kenealy T, Arroll B, Petrie KJ. Patients and computers as reminders to screen for diabetes in family practice. Randomized-controlled trial. J Gen Intern Med. 2005;20:916-21. [PMID: 16191138]
74.
Lobach DF, Hammond WE. Development and evaluation of a Computer-Assisted Management Protocol (CAMP): improved compliance with care guidelines for diabetes mellitus. Proc Annu Symp Comput Appl Med Care. 1994;:787-91. [PMID: 7950032]
75.
Ornstein SM, Garr DR, Jenkins RG, Rust PF, Arnon A. Computer-generated physician and patient reminders. Tools to improve population adherence to selected preventive services. J Fam Pract. 1991;32:82-90. [PMID: 1985140]
76.
Peterson KA, Radosevich DM, O'Connor PJ, Nyman JA, Prineas RJ, Smith SA, et al. Improving diabetes care in practice: findings from the TRANSLATE trial. Diabetes Care. 2008;31:2238-43. [PMID: 18809622]
77.
Reeve JF, Tenni PC, Peterson GM. An electronic prompt in dispensing software to promote clinical interventions by community pharmacists: a randomized controlled trial. Br J Clin Pharmacol. 2008;65:377-85. [PMID: 17764471]
78.
Rosser WW, Hutchison BG, McDowell I, Newell C. Use of reminders to increase compliance with tetanus booster vaccination. CMAJ. 1992;146:911-7. [PMID: 1544078]
79.
Rosser WW, McDowell I, Newell C. Use of reminders for preventive procedures in family medicine. CMAJ. 1991;145:807-14. [PMID: 1913409]
80.
Sequist TD, Gandhi TK, Karson AS, Fiskio JM, Bugbee D, Sperling M, et al. A randomized trial of electronic clinical reminders to improve quality of care for diabetes and coronary artery disease. J Am Med Inform Assoc. 2005;12:431-7. [PMID: 15802479]
81.
Tierney WM, Hui SL, McDonald CJ. Delayed feedback of physician performance versus immediate reminders to perform preventive care. Effects on physician compliance. Med Care. 1986;24:659-66. [PMID: 3736141]
82.
van Wyk JT, van Wijk MA, Sturkenboom MC, Mosseveld M, Moorman PW, van der Lei J. Electronic alerts versus on-demand decision support to improve dyslipidemia treatment: a cluster randomized controlled trial. Circulation. 2008;117:371-8. [PMID: 18172036]
83.
Bates DW, Kuperman GJ, Rittenberg E, Teich JM, Fiskio J, Ma'luf N, et al. A randomized trial of a computer-based intervention to reduce utilization of redundant laboratory tests. Am J Med. 1999;106:144-50. [PMID: 10230742]
84.
Bell LM, Grundmeier R, Localio R, Zorc J, Fiks AG, Zhang X, et al. Electronic health record-based decision support to improve asthma care: a cluster-randomized trial. Pediatrics. 2010;125:770-7. [PMID: 20231191]
85.
Downs M, Turner S, Bryans M, Wilcock J, Keady J, Levin E, et al. Effectiveness of educational interventions in improving detection and management of dementia in primary care: cluster randomised controlled study. BMJ. 2006;332:692-6. [PMID: 16565124]
86.
Emery J, Morris H, Goodchild R, Fanshawe T, Prevost AT, Bobrow M, et al. The GRAIDS Trial: a cluster randomised controlled trial of computer decision support for the management of familial cancer risk in primary care. Br J Cancer. 2007;97:486-93. [PMID: 17700548]
87.
Feldstein AC, Smith DH, Perrin N, Yang X, Rix M, Raebel MA, et al. Improved therapeutic monitoring with several interventions: a randomized trial. Arch Intern Med. 2006;166:1848-54. [PMID: 17000941]
88.
Flottorp S, Oxman AD, Håvelsrud K, Treweek S, Herrin J. Cluster randomised controlled trial of tailored interventions to improve the management of urinary tract infections in women and sore throat. BMJ. 2002;325:367. [PMID: 12183309]
89.
Greiver M, Drummond N, White D, Weshler J, Moineddin R; North Toronto Primary Care Research Network (Nortren). Angina on the Palm: randomized controlled pilot trial of Palm PDA software for referrals for cardiac testing. Can Fam Physician. 2005;51:382-3. [PMID: 16926933]
90.
Harpole LH, Khorasani R, Fiskio J, Kuperman GJ, Bates DW. Automated evidence-based critiquing of orders for abdominal radiographs: impact on utilization and appropriateness. J Am Med Inform Assoc. 1997;4:511-21. [PMID: 9391938]
91.
Lee NJ, Chen ES, Currie LM, Donovan M, Hall EK, Jia H, et al. The effect of a mobile clinical decision support system on the diagnosis of obesity and overweight in acute and primary care encounters. ANS Adv Nurs Sci. 2009;32:211-21. [PMID: 19707090]
92.
Lo HG, Matheny ME, Seger DL, Bates DW, Gandhi TK. Impact of non-interruptive medication laboratory monitoring alerts in ambulatory care. J Am Med Inform Assoc. 2009;16:66-71. [PMID: 18952945]
93.
Matheny ME, Sequist TD, Seger AC, Fiskio JM, Sperling M, Bugbee D, et al. A randomized trial of electronic clinical reminders to improve medication laboratory monitoring. J Am Med Inform Assoc. 2008;15:424-9. [PMID: 18436905]
94.
McDonald CJ. Use of a computer to detect and respond to clinical events: its effect on clinician behavior. Ann Intern Med. 1976;84:162-7. [PMID: 1252043]
95.
McDowell I, Newell C, Rosser W. A randomized trial of computerized reminders for blood pressure screening in primary care. Med Care. 1989;27:297-305. [PMID: 2494397]
96.
Palen TE, Raebel M, Lyons E, Magid DM. Evaluation of laboratory monitoring alerts within a computerized physician order entry system for medication orders. Am J Manag Care. 2006;12:389-95. [PMID: 16834525]
97.
Palen TE, Price DW, Snyder AJ, Shetterly SM. Computerized alert reduced D-dimer testing in the elderly. Am J Manag Care. 2010;16:267-75. [PMID: 21087072]
98.
Player MS, Gill JM, Mainous AG 3rd, Everett CJ, Koopman RJ, Diamond JJ, et al. An electronic medical record-based intervention to improve quality of care for gastro-esophageal reflux disease (GERD) and atypical presentations of GERD. Qual Prim Care. 2010;18:223-9. [PMID: 20836938]
99.
Raebel MA, Lyons EE, Chester EA, Bodily MA, Kelleher JA, Long CL, et al. Improving laboratory monitoring at initiation of drug therapy in ambulatory care: a randomized trial. Arch Intern Med. 2005;165:2395-401. [PMID: 16287769]
100.
Raebel MA, Chester EA, Newsom EE, Lyons EE, Kelleher JA, Long C, et al. Randomized trial to improve laboratory safety monitoring of ongoing drug therapy in ambulatory patients. Pharmacotherapy. 2006;26:619-26. [PMID: 16637791]
101.
Roukema J, Steyerberg EW, van der Lei J, Moll HA. Randomized trial of a clinical decision support system: impact on the management of children with fever without apparent source. J Am Med Inform Assoc. 2008;15:107-13. [PMID: 17947627]
102.
Roy PM, Durieux P, Gillaizeau F, Legall C, Armand-Perroux A, Martino L, et al. A computerized handheld decision-support system to improve pulmonary embolism diagnosis: a randomized trial. Ann Intern Med. 2009;151:677-86. [PMID: 19920268]
103.
Schriefer SP, Landis SE, Turbow DJ, Patch SC. Effect of a computerized body mass index prompt on diagnosis and treatment of adult obesity. Fam Med. 2009;41:502-7. [PMID: 19582636]
104.
Stiell IG, Clement CM, Grimshaw J, Brison RJ, Rowe BH, Schull MJ, et al. Implementation of the Canadian C-Spine Rule: prospective 12 centre cluster randomised trial. BMJ. 2009;339:b4146. [PMID: 19875425]
105.
Sundaram V, Lazzeroni LC, Douglass LR, Sanders GD, Tempio P, Owens DK. A randomized trial of computer-based reminders and audit and feedback to improve HIV screening in a primary care setting. Int J STD AIDS. 2009;20:527-33. [PMID: 19625582]
106.
Tierney WM, McDonald CJ, Martin DK, Rogers MP. Computerized display of past test results. Effect on outpatient testing. Ann Intern Med. 1987;107:569-74. [PMID: 3631792]
107.
van Wijk MA, van der Lei J, Mosseveld M, Bohnen AM, van Bemmel JH. Assessment of decision support for blood test ordering in primary care. a randomized trial. Ann Intern Med. 2001;134:274-81. [PMID: 11182837]
108.
Walker J, Fairley CK, Walker SM, Gurrin LC, Gunn JM, Pirotta MV, et al. Computer reminders for Chlamydia screening in general practice: a randomized controlled trial. Sex Transm Dis. 2010;37:445-50. [PMID: 20375930]
109.
Wilson BJ, Torrance N, Mollison J, Watson MS, Douglas A, Miedzybrodzka Z, et al. Cluster randomized trial of a multifaceted primary care decision-support intervention for inherited breast cancer risk. Fam Pract. 2006;23:537-44. [PMID: 16787957]
110.
Bourgeois FC, Linder J, Johnson SA, Co JP, Fiskio J, Ferris TG. Impact of a computerized template on antibiotic prescribing for acute respiratory infections in children and adolescents. Clin Pediatr (Phila). 2010;49:976-83. [PMID: 20724348]
111.
Christakis DA, Zimmerman FJ, Wright JA, Garrison MM, Rivara FP, Davis RL. A randomized controlled trial of point-of-care evidence to improve the antibiotic prescribing practices for otitis media in children. Pediatrics. 2001;107:15. [PMID: 11158489]
112.
Co JP, Johnson SA, Poon EG, Fiskio J, Rao SR, Van Cleave J, et al. Electronic health record decision support and quality of care for children with ADHD. Pediatrics. 2010;126:239-46. [PMID: 20643719]
113.
Cobos A, Vilaseca J, Asenjo C, Pedro-Botet J, Sánchez E, Val A, et al. Cost effectiveness of a clinical decision support system based on the recommendations of the European Society of Cardiology and other societies for the management of hypercholesterolemia: report of a cluster-randomized trial. Disease Management & Health Outcomes. 2005;13:421-32.
114.
Davis RL, Wright J, Chalmers F, Levenson L, Brown JC, Lozano P, et al. A cluster randomized clinical trial to improve prescribing patterns in ambulatory pediatrics. PLoS Clin Trials. 2007;2:25. [PMID: 17525793]
115.
Feldstein A, Elmer PJ, Smith DH, Herson M, Orwoll E, Chen C, et al. Electronic medical record reminder improves osteoporosis management after a fracture: a randomized, controlled trial. J Am Geriatr Soc. 2006;54:450-7. [PMID: 16551312]
116.
Field TS, Rochon P, Lee M, Gavendo L, Baril JL, Gurwitz JH. Computerized clinical decision support during medication ordering for long-term care residents with renal insufficiency. J Am Med Inform Assoc. 2009;16:480-5. [PMID: 19390107]
117.
Filippi A, Sabatini A, Badioli L, Samani F, Mazzaglia G, Catapano A, et al. Effects of an automated electronic reminder in changing the antiplatelet drug-prescribing behavior among Italian general practitioners in diabetic patients: an intervention trial. Diabetes Care. 2003;26:1497-500. [PMID: 12716811]
118.
Fitzmaurice DA, Hobbs FD, Murray ET, Holder RL, Allan TF, Rose PE. Oral anticoagulation management in primary care with the use of computerized decision support and near-patient testing: a randomized, controlled trial. Arch Intern Med. 2000;160:2343-8. [PMID: 10927732]
119.
Fortuna RJ, Zhang F, Ross-Degnan D, Campion FX, Finkelstein JA, Kotch JB, et al. Reducing the prescribing of heavily marketed medications: a randomized controlled trial. J Gen Intern Med. 2009;24:897-903. [PMID: 19475459]
120.
Goud R, de Keizer NF, ter Riet G, Wyatt JC, Hasman A, Hellemans IM, et al. Effect of guideline based computerised decision support on decision making of multidisciplinary teams: cluster randomised trial in cardiac rehabilitation. BMJ. 2009;338:b1440. [PMID: 19398471]
121.
Hicks LS, Sequist TD, Ayanian JZ, Shaykevich S, Fairchild DG, Orav EJ, et al. Impact of computerized decision support on blood pressure management and control: a randomized controlled trial. J Gen Intern Med. 2008;23:429-41. [PMID: 18373141]
122.
Krall MA, Traunweiser K, Towery W. Effectiveness of an electronic medical record clinical quality alert prepared by off-line data analysis. Stud Health Technol Inform. 2004;107:135-9. [PMID: 15360790]
123.
Linder JA, Rigotti NA, Schneider LI, Kelley JH, Brawarsky P, Haas JS. An electronic health record-based intervention to improve tobacco treatment in primary care: a cluster-randomized controlled trial. Arch Intern Med. 2009;169:781-7. [PMID: 19398690]
124.
Locatelli F, Covic A, Macdougall IC, Scherhag A, Wiecek A; ORAMA Study Group. Effect of computer-assisted European Best Practice Guideline implementation on adherence and target attainment: ORAMA results. J Nephrol. 2009;22:662-74. [PMID: 19810000]
125.
Manotti C, Moia M, Palareti G, Pengo V, Ria L, Dettori AG. Effect of computer-aided management on the quality of treatment in anticoagulated patients: a prospective, randomized, multicenter trial of APROAT (Automated PRogram for Oral Anticoagulant Treatment). Haematologica. 2001;86:1060-70. [PMID: 11602412]
126.
Marco F, Sedano C, Bermúdez A, López-Duarte M, Fernández-Fontecha E, Zubizarreta A. A prospective controlled study of a computer-assisted acenocoumarol dosage program. Pathophysiol Haemost Thromb. 2003;33:59-63. [PMID: 14624045]
127.
Martens JD, van der Aa A, Panis B, van der Weijden T, Winkens RA, Severens JL. Design and evaluation of a computer reminder system to improve prescribing behaviour of GPs. Stud Health Technol Inform. 2006;124:617-23. [PMID: 17108585]
128.
Martens JD, van der Weijden T, Severens JL, de Clercq PA, de Bruijn DP, Kester AD, et al. The effect of computer reminders on GPs' prescribing behaviour: a cluster-randomised trial. Int J Med Inform. 2007;76 Suppl 3 S403-16. [PMID: 17569575]
129.
Montgomery AA, Fahey T, Peters TJ, MacIntosh C, Sharp DJ. Evaluation of computer based clinical decision support system and risk chart for management of hypertension in primary care: randomised controlled trial. BMJ. 2000;320:686-90. [PMID: 10710578]
130.
Overhage JM, Tierney WM, Zhou XH, McDonald CJ. A randomized trial of “corollary orders” to prevent errors of omission. J Am Med Inform Assoc. 1997;4:364-75. [PMID: 9292842]
131.
Peterson JF, Rosenbaum BP, Waitman LR, Habermann R, Powers J, Harrell D, et al. Physicians' response to guided geriatric dosing: initial results from a randomized trial. Stud Health Technol Inform. 2007;129:1037-40. [PMID: 17911873]
132.
Phillips LS, Ziemer DC, Doyle JP, Barnes CS, Kolm P, Branch WT, et al. An endocrinologist-supported intervention aimed at providers improves diabetes management in a primary care site: improving primary care of African Americans with diabetes (IPCAAD) 7. Diabetes Care. 2005;28:2352-60. [PMID: 16186262]
133.
Ziemer DC, Doyle JP, Barnes CS, Branch WT Jr, Cook CB, El-Kebbi IM, et al. An intervention to overcome clinical inertia and improve diabetes mellitus control in a primary care setting: Improving Primary Care of African Americans with Diabetes (IPCAAD) 8. Arch Intern Med. 2006;166:507-13. [PMID: 16534036]
134.
Raebel MA, Charles J, Dugan J, Carroll NM, Korner EJ, Brand DW, et al. Randomized trial to improve prescribing safety in ambulatory elderly patients. J Am Geriatr Soc. 2007;55:977-85. [PMID: 17608868]
135.
Rood E, Bosman RJ, van der Spoel JI, Taylor P, Zandstra DF. Use of a computerized guideline for glucose regulation in the intensive care unit improved both guideline adherence and glucose regulation. J Am Med Inform Assoc. 2005;12:172-80. [PMID: 15561795]
136.
Rossi RA, Every NR. A computerized intervention to decrease the use of calcium channel blockers in hypertension. J Gen Intern Med. 1997;12:672-8. [PMID: 9383135]
137.
Rothschild JM, McGurk S, Honour M, Lu L, McClendon AA, Srivastava P, et al. Assessment of education and computerized decision support interventions for improving transfusion practice. Transfusion. 2007;47:228-39. [PMID: 17302768]
138.
Samore MH, Bateman K, Alder SC, Hannah E, Donnelly S, Stoddard GJ, et al. Clinical decision support and appropriateness of antimicrobial prescribing: a randomized trial. JAMA. 2005;294:2305-14. [PMID: 16278358]
139.
Shojania KG, Yokoe D, Platt R, Fiskio J, Ma'luf N, Bates DW. Reducing vancomycin use utilizing a computer guideline: results of a randomized controlled trial. J Am Med Inform Assoc. 1998;5:554-62. [PMID: 9824802]
140.
Simon SR, Smith DH, Feldstein AC, Perrin N, Yang X, Zhou Y, et al. Computerized prescribing alerts and group academic detailing to reduce the use of potentially inappropriate medications in older people. J Am Geriatr Soc. 2006;54:963-8. [PMID: 16776793]
141.
Smith SA, Shah ND, Bryant SC, Christianson TJ, Bjornsen SS, Giesler PD, et al; Evidens Research Group. Chronic care model and shared care in diabetes: randomized trial of an electronic decision support system. Mayo Clin Proc. 2008;83:747-57. [PMID: 18613991]
142.
Strom BL, Schinnar R, Aberra F, Bilker W, Hennessy S, Leonard CE, et al. Unintended effects of a computerized physician order entry nearly hard-stop alert to prevent a drug interaction: a randomized controlled trial. Arch Intern Med. 2010;170:1578-83. [PMID: 20876410]
143.
Strom BL, Schinnar R, Bilker W, Hennessy S, Leonard CE, Pifer E. Randomized clinical trial of a customized electronic alert requiring an affirmative response compared to a control group receiving a commercial passive CPOE alert: NSAID—warfarin co-prescribing as a test case. J Am Med Inform Assoc. 2010;17:411-5. [PMID: 20595308]
144.
Tamblyn R, Huang A, Perreault R, Jacques A, Roy D, Hanley J, et al. The medical office of the 21st century (MOXXI): effectiveness of computerized decision-making support in reducing inappropriate prescribing in primary care. CMAJ. 2003;169:549-56. [PMID: 12975221]
145.
Tamblyn R, Huang A, Taylor L, Kawasumi Y, Bartlett G, Grad R, et al. A randomized trial of the effectiveness of on-demand versus computer-triggered drug decision support in primary care. J Am Med Inform Assoc. 2008;15:430-8. [PMID: 18436904]
146.
Tamblyn R, Reidel K, Huang A, Taylor L, Winslade N, Bartlett G, et al. Increasing the detection and response to adherence problems with cardiovascular medication in primary care through computerized drug management systems: a randomized controlled trial. Med Decis Making. 2010;30:176-88. [PMID: 19675319]
147.
Terrell KM, Perkins AJ, Dexter PR, Hui SL, Callahan CM, Miller DK. Computerized decision support to reduce potentially inappropriate prescribing to older emergency department patients: a randomized, controlled trial. J Am Geriatr Soc. 2009;57:1388-94. [PMID: 19549022]
148.
Terrell KM, Perkins AJ, Hui SL, Callahan CM, Dexter PR, Miller DK. Computerized decision support for medication dosing in renal insufficiency: a randomized, controlled trial. Ann Emerg Med. 2010;56:623-9. [PMID: 20452703]
149.
Vadher BD, Patterson DL, Leaning M. Comparison of oral anticoagulant control by a nurse-practitioner using a computer decision-support system with that by clinicians. Clin Lab Haematol. 1997;19:203-7. [PMID: 9352146]
150.
Vadher B, Patterson DL, Leaning M. Evaluation of a decision support system for initiation and control of oral anticoagulation in a randomised trial. BMJ. 1997;314:1252-6. [PMID: 9154031]
151.
Vissers MC, Biert J, van der Linden CJ, Hasman A. Effects of a supportive protocol processing system (ProtoVIEW) on clinical behaviour of residents in the accident and emergency department. Comput Methods Programs Biomed. 1996;49:177-84. [PMID: 8735024]
152.
Vissers MC, Hasman A, van der Linden CJ. Protocol processing system (ProtoVIEW) to support residents at the emergency ward. Comput Methods Programs Biomed. 1995;48:53-8. [PMID: 8846712]
153.
Weir CJ, Lees KR, MacWalter RS, Muir KW, Wallesch CW, McLelland EV, et al; PRISM Study Group. Cluster-randomized, controlled trial of computer-based decision support for selecting long-term anti-thrombotic therapy after acute ischaemic stroke. QJM. 2003;96:143-53. [PMID: 12589012]
154.
White KS, Lindsay A, Pryor TA, Brown WF, Walsh K. Application of a computerized medical decision-making process to the problem of digoxin intoxication. J Am Coll Cardiol. 1984;4:571-6. [PMID: 6381570]
155.
Alper BS, White DS, Ge B. Physicians answer more clinical questions and change clinical decisions more often with synthesized evidence: a randomized trial in primary care. Ann Fam Med. 2005;3:507-13. [PMID: 16338914]
156.
Del Fiol G, Haug PJ, Cimino JJ, Narus SP, Norlin C, Mitchell JA. Effectiveness of topic-specific infobuttons: a randomized controlled trial. J Am Med Inform Assoc. 2008;15:752-9. [PMID: 18755999]
157.
Etchells E, Adhikari NK, Cheung C, Fowler R, Kiss A, Quan S, et al. Real-time clinical alerting: effect of an automated paging system on response time to critical laboratory values—a randomised controlled trial. Qual Saf Health Care. 2010;19:99-102. [PMID: 20351157]
158.
Bird JA, McPhee SJ, Jenkins C, Fordham D. Three strategies to promote cancer screening. How feasible is wide-scale implementation? Med Care. 1990;28:1005-12. [PMID: 2250488]
159.
Frame PS, Zimmer JG, Werth PL, Hall WJ, Eberly SW. Computer-based vs manual health maintenance tracking. A controlled trial. Arch Fam Med. 1994;3:581-8. [PMID: 7921293]
160.
Tierney WM, McDonald CJ, Hui SL, Martin DK. Computer predictions of abnormal test results. Effects on outpatient testing. JAMA. 1988;259:1194-8. [PMID: 3339821]
161.
Cleveringa FG, Gorter KJ, van den Donk M, Rutten GE. Combined task delegation, computerized decision support, and feedback improve cardiovascular risk for type 2 diabetic patients: a cluster randomized trial in primary care. Diabetes Care. 2008;31:2273-5. [PMID: 18796619]
162.
Cleveringa FG, Welsing PM, van den Donk M, Gorter KJ, Niessen LW, Rutten GE, et al. Cost-effectiveness of the diabetes care protocol, a multifaceted computerized decision support diabetes management intervention that reduces cardiovascular risk. Diabetes Care. 2010;33:258-63. [PMID: 19933991]
163.
Smith DH, Feldstein AC, Perrin NA, Yang X, Rix MM, Raebel MA, et al. Improving laboratory monitoring of medications: an economic analysis alongside a clinical trial. Am J Manag Care. 2009;15:281-9. [PMID: 19435396]
164.
Judge J, Field TS, DeFlorio M, Laprino J, Auger J, Rochon P, et al. Prescribers' responses to alerts during medication ordering in the long term care setting. J Am Med Inform Assoc. 2006;13:385-90. [PMID: 16622171]
165.
Maviglia SM, Yoon CS, Bates DW, Kuperman G. KnowledgeLink: impact of context-sensitive information retrieval on clinicians' information needs. J Am Med Inform Assoc. 2006;13:67-73. [PMID: 16221942]
166.
Rollman BL, Hanusa BH, Gilbert T, Lowe HJ, Kapoor WN, Schulberg HC. The electronic medical record. A randomized trial of its impact on primary care physicians' initial management of major depression [corrected]. Arch Intern Med. 2001;161:189-97. [PMID: 11176732]
167.
McLaughlin D, Hayes JR, Kelleher K. Office-based interventions for recognizing abnormal pediatric blood pressures. Clin Pediatr (Phila). 2010;49:355-62.
168.
Hetlevik I, Holmen J, Krüger O. Implementing clinical guidelines in the treatment of hypertension in general practice. Evaluation of patient outcome related to implementation of a computer-based clinical decision support system. Scand J Prim Health Care. 1999;17:35-40. [PMID: 10229991]
169.
Hetlevik I, Holmen J, Krüger O, Kristensen P, Iversen H. Implementing clinical guidelines in the treatment of hypertension in general practice. Blood Press. 1998;7:270-6. [PMID: 10321438]
170.
Hetlevik I, Holmen J, Krüger O, Kristensen P, Iversen H, Furuseth K. Implementing clinical guidelines in the treatment of diabetes mellitus in general practice. Evaluation of effort, process, and patient outcome related to implementation of a computer-based decision support system. Int J Technol Assess Health Care. 2000;16:210-27. [PMID: 10815366]
171.
Bosworth HB, Olsen MK, Dudley T, Orr M, Goldstein MK, Datta SK, et al. Patient education and provider decision support to control blood pressure in primary care: a cluster randomized trial. Am Heart J. 2009;157:450-6. [PMID: 19249414]
172.
Bosworth HB, Olsen MK, Goldstein MK, Orr M, Dudley T, McCant F, et al. The veterans' study to improve the control of hypertension (V-STITCH): design and methodology. Contemp Clin Trials. 2005;26:155-68. [PMID: 15837438]
173.
Buntin MB, Burke MF, Hoaglin MC, Blumenthal D. The benefits of health information technology: a review of the recent literature shows predominantly positive results. Health Aff (Millwood). 2011;30:464-71. [PMID: 21383365]
174.
Osheroff JA; Healthcare Information and Management Systems Society. Improving Outcomes with Clinical Decision Support: An Implementer's Guide. 2nd ed. Chicago: Healthcare Information and Management Systems Society; 2005.
Comments
Sign In to Submit A CommentInformation & Authors
Information
Published In
Annals of Internal Medicine
Volume 157 • Number 1 • 3 July 2012
Pages: 29 - 43
History
Published online: 3 July 2012
Published in issue: 3 July 2012
Keywords
Copyright
© 2012 American College of Physicians.
Authors
Metrics & Citations
Metrics
Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. For an editable text file, please select Medlars format which will download as a .txt file. Simply select your manager software from the list below and click Download.
For more information or tips please see 'Downloading to a citation manager' in the Help menu.
Download article citation data for:
Effect of Clinical Decision-Support Systems: A Systematic Review. Ann Intern Med.2012;157:29-43. [Epub 3 July 2012]. doi:10.7326/0003-4819-157-1-201207030-00450
Effect of Clinical Decision-Support Systems: A Systematic Review. Ann Intern Med.2012;157:29-43. [Epub 3 July 2012]. doi:10.7326/0003-4819-157-1-201207030-00450
View More
Login Options:
Purchase
You will be redirected to acponline.org to sign-in to Annals to complete your purchase.
Access to EPUBs and PDFs for FREE Annals content requires users to be registered and logged in. A subscription is not required. You can create a free account below or from the following link. You will be redirected to acponline.org to create an account that will provide access to Annals. If you are accessing the Free Annals content via your institution's access, registration is not required.
Create your Free Account
You will be redirected to acponline.org to create an account that will provide access to Annals.
To the Editor
The article by Bright et al on Clinical Decision-Support Systems (CDSS) (1) was timely and interesting, but had an important omission. No mention was made of CDSS that focused on helping doctors make the correct diagnosis. Such systems have been broadly available for some 25 years, including QMR/Internist-I, ILIAD and DXplain. Currently, at least two general-purpose Diagnostic Decision Support systems, DXplain and ISABEL, are widely available. Reviews (2,3) have shown that these systems suggest important diseases the clinician had not previously considered, and a study (4) from the Mayo Clinic found that use of DXplain lowered length-of-stay and hospital costs.
1. Bright TJ et al “Effect of Clinical Decision-Support Systems” Ann Intern Med 2012;157:29-432.
2. Berner ES et al “Performance of Four Computer-Based Diagnostic Systems” New Engl J Med 1994; 330:1792-63.
3. Bond WF et al “Differential Diagnosis Generators: an Evaluation of Currently Available Computer Programs” J Gen Intern Med 2011;27:213-94.
4. Elkin PL et al “The Introduction of a Diagnostic Decision Support System (DXplain) into the Workflow of a Teaching Hospital Service can Decrease the Cost of Service for Diagnostically Challenging DRGs” Internat Jl Med Informatics 2010; 79:772-