Academia and Clinic
7 February 2006

Translating Clinical Research into Clinical Practice: Impact of Using Prediction Rules To Make Decisions

Publication: Annals of Internal Medicine
Volume 144, Number 3

Abstract

Clinical prediction rules, sometimes called clinical decision rules, have proliferated in recent years. However, very few have undergone formal impact analysis, the standard of evidence to assess their impact on patient care. Without impact analysis, clinicians cannot know whether using a prediction rule will be beneficial or harmful. This paper reviews standards of evidence for developing and evaluating prediction rules; important differences between prediction rules and decision rules; how to assess the potential clinical impact of a prediction rule before translating it into a decision rule; methodologic issues critical to successful impact analysis, including defining outcome measures and estimating sample size; the importance of close collaboration between clinical investigators and practicing clinicians before, during, and after impact analysis; and the need to measure both efficacy and effectiveness when analyzing a decision rule's clinical impact. These considerations should inform future development, evaluation, and use of all clinical prediction or decision rules.

Get full access to this article

View all available purchase options and get full access to this article.

References

1.
Wasson JHSox HCNeff RKGoldman L. Clinical prediction rules. Applications and methodological standards. N Engl J Med. 1985;313:793-9. [PMID: 3897864]
2.
Laupacis ASekar NStiell IG. Clinical prediction rules. A review and suggested modifications of methodological standards. JAMA. 1997;277:488-94. [PMID: 9020274]
3.
Wells PSAnderson DRBormanis JGuy FMitchell MGray Let al. Value of assessment of pretest probability of deep venous thrombosis in clinical management. Lancet. 1997;350:1795-8. [PMID: 9428249]
4.
Goldman LCook EFJohnson PABrand DARouan GWLee TH. Prediction of the need for intensive care in patients who come to the emergency departments with acute chest pain. N Engl J Med. 1996;334:1498-504. [PMID: 8618604]
5.
Stiell IGMcKnight RDGreenberg GHMcDowell INair RCWells GAet al. Implementation of the Ottawa ankle rules. JAMA. 1994;271:827-32. [PMID: 8114236]
6.
McGinn TGGuyatt GHWyer PCNaylor CDStiell IGRichardson WS. Users' guides to the medical literature: XXII: how to use articles about clinical decision rules. Evidence-Based Medicine Working Group. JAMA. 2000;284:79-84. [PMID: 10872017]
7.
Lee THPearson SDJohnson PAGarcia TBWeisberg MCGuadagnoli Eet al. Failure of information as an intervention to modify clinical management. A time-series trial in patients with acute chest pain. Ann Intern Med. 1995;122:434-7. [PMID: 7856992]
8.
Selker HPBeshansky JRGriffith JLAufderheide TPBallin DSBernard SAet al. Use of the acute cardiac ischemia time-insensitive predictive instrument (ACI-TIPI) to assist with triage of patients with chest pain or other symptoms suggestive of acute cardiac ischemia. A multicenter, controlled clinical trial. Ann Intern Med. 1998;129:845-55. [PMID: 9867725]
9.
Brand DAFrazier WHKohlhepp WCShea KMHoefer AMEcker MDet al. A protocol for selecting patients with injured extremities who need x-rays. N Engl J Med. 1982;306:333-9. [PMID: 7054709]
10.
Cameron CNaylor CD. No impact from active dissemination of the Ottawa Ankle Rules: further evidence of the need for local implementation of practice guidelines. CMAJ. 1999;160:1165-8. [PMID: 10234347]
11.
Lee TH. Evaluating decision aids: the next painful step [Editorial]. J Gen Intern Med. 1990;5:528-9. [PMID: 2266437]
12.
Wang TJMassaro JMLevy DVasan RSWolf PAD'Agostino RBet al. A risk score for predicting stroke or death in individuals with new-onset atrial fibrillation in the community: the Framingham Heart Study. JAMA. 2003;290:1049-56. [PMID: 12941677]
13.
Stiell IGWells GAVandemheen KLClement CMLesiuk HDe Maio VJet al. The Canadian C-spine rule for radiography in alert and stable trauma patients. JAMA. 2001;286:1841-8. [PMID: 11597285]
14.
Mehr DRBinder EFKruse RLZweig SCMadsen RPopejoy Let al. Predicting mortality in nursing home residents with lower respiratory tract infection: The Missouri LRI Study. JAMA. 2001;286:2427-36. [PMID: 11712938]
15.
Collins SMyatt M. Short-term prognosis in severe adult and adolescent malnutrition during famine: use of a simple prognostic model based on counting clinical signs. JAMA. 2000;284:621-6. [PMID: 10918712]
16.
Johnston SCGress DRBrowner WSSidney S. Short-term prognosis after emergency department diagnosis of TIA. JAMA. 2000;284:2901-6. [PMID: 11147987]
17.
Boersma EPoldermans DBax JJSteyerberg EWThomson IRBanga JDet al. Predictors of cardiac events after major vascular surgery: role of clinical characteristics, dobutamine echocardiography, and beta-blocker therapy. JAMA. 2001;285:1865-73. [PMID: 11308400]
18.
Ferreira FLBota DPBross AMélot CVincent JL. Serial evaluation of the SOFA score to predict outcome in critically ill patients. JAMA. 2001;286:1754-8. [PMID: 11594901]
19.
Nielsen OWHansen JFHilden JLarsen CTSvanegaard J. Risk assessment of left ventricular systolic dysfunction in primary care: cross sectional study evaluating a range of diagnostic tests. BMJ. 2000;320:220-4. [PMID: 10642232]
20.
Hasbun RAbrahams JJekel JQuagliarello VJ. Computed tomography of the head before lumbar puncture in adults with suspected meningitis. N Engl J Med. 2001;345:1727-33. [PMID: 11742046]
21.
Benavente OEliasziw MStreifler JYFox AJBarnett HJMeldrum Het al. Prognosis after transient monocular blindness associated with carotid-artery stenosis. N Engl J Med. 2001;345:1084-90. [PMID: 11596587]
22.
Bailey WCGerald LBKimerling MERedden DBrook NBruce Fet al. Predictive model to identify positive tuberculosis skin test results during contact investigations. JAMA. 2002;287:996-1002. [PMID: 11866647]
23.
Walter LCBrand RJCounsell SRPalmer RMLandefeld CSFortinsky RHet al. Development and validation of a prognostic index for 1-year mortality in older adults after hospitalization. JAMA. 2001;285:2987-94. [PMID: 11410097]
24.
van Walraven CForster AJParish DCDane FCChandra KMDurham MDet al. Validation of a clinical decision aid to discontinue in-hospital cardiac arrest resuscitations. JAMA. 2001;285:1602-6. [PMID: 11268268]
25.
Ozuah POOzuah TPStein REBurton WMulvihill M. Evaluation of a risk assessment questionnaire used to target tuberculin skin testing in children. JAMA. 2001;285:451-3. [PMID: 11242430]
26.
Berkley JARoss AMwangi IOsier FHMohammed MShebbe Met al. Prognostic indicators of early and late death in children admitted to district hospital in Kenya: cohort study. BMJ. 2003;326:361. [PMID: 12586667]
27.
Imperiale TFWagner DRLin CYLarkin GNRogge JDRansohoff DF. Using risk for advanced proximal colonic neoplasia to tailor endoscopic screening for colorectal cancer. Ann Intern Med. 2003;139:959-65. [PMID: 14678915]
28.
Haydel MJPreston CAMills TJLuber SBlaudeau EDeBlieux PM. Indications for computed tomography in patients with minor head injury. N Engl J Med. 2000;343:100-5. [PMID: 10891517]
29.
Casey BMMcIntire DDLeveno KJ. The continuing value of the Apgar score for the assessment of newborn infants. N Engl J Med. 2001;344:467-71. [PMID: 11172187]
30.
Miller STSleeper LAPegelow CHEnos LEWang WCWeiner SJet al. Prediction of adverse outcomes in children with sickle cell disease. N Engl J Med. 2000;342:83-9. [PMID: 10631276]
31.
Weber JEShofer FSLarkin GLKalaria ASHollander JE. Validation of a brief observation period for patients with cocaine-associated chest pain. N Engl J Med. 2003;348:510-7. [PMID: 12571258]
32.
Lee DSAustin PCRouleau JLLiu PPNaimark DTu JV. Predicting mortality among patients hospitalized for heart failure: derivation and validation of a clinical model. JAMA. 2003;290:2581-7. [PMID: 14625335]
33.
Morrow DAAntman EMParsons Lde Lemos JACannon CPGiugliano RPet al. Application of the TIMI risk score for ST-elevation MI in the National Registry of Myocardial Infarction 3. JAMA. 2001;286:1356-9. [PMID: 11560541]
34.
Cadarette SMJaglal SBMurray TMMcIsaac WJJoseph LBrown JPet al. Evaluation of decision rules for referring women for bone densitometry by dual-energy x-ray absorptiometry. JAMA. 2001;286:57-63. [PMID: 11434827]
35.
Hasbun RVikram HRBarakat LABuenconsejo JQuagliarello VJ. Complicated left-sided native valve endocarditis in adults: risk classification for mortality. JAMA. 2003;289:1933-40. [PMID: 12697795]
36.
D'Agostino RB SrGrundy SSullivan LMWilson P. Validation of the Framingham coronary heart disease prediction scores: results of a multiple ethnic groups investigation. JAMA. 2001;286:180-7. [PMID: 11448281]
37.
Gage BFWaterman ADShannon WBoechler MRich MWRadford MJ. Validation of clinical classification schemes for predicting stroke: results from the National Registry of Atrial Fibrillation. JAMA. 2001;285:2864-70. [PMID: 11401607]
38.
Antman EMCohen MBernink PJMcCabe CHHoracek TPapuchis Get al. The TIMI risk score for unstable angina/non-ST elevation MI: a method for prognostication and therapeutic decision making. JAMA. 2000;284:835-42. [PMID: 10938172]
39.
Brindle PEmberson JLampe FWalker MWhincup PFahey Tet al. Predictive accuracy of the Framingham coronary risk score in British men: prospective cohort study. BMJ. 2003;327:1267. [PMID: 14644971]
40.
Daly KBeale RChang RW. Reduction in mortality after inappropriate early discharge from intensive care unit: logistic regression triage model. BMJ. 2001;322:1274-6. [PMID: 11375229]
41.
Schoonhoven LHaalboom JRBousema MTAlgra AGrobbee DEGrypdonck MHet al. Prospective cohort study of routine use of risk assessment scales for prediction of pressure ulcers. BMJ. 2002;325:797. [PMID: 12376437]
42.
Ramachandran SFrench JMVanderpump MPCroft PNeary RH. Using the Framingham model to predict heart disease in the United Kingdom: retrospective study. BMJ. 2000;320:676-7. [PMID: 10710574]
43.
Arozullah AMKhuri SFHenderson WGDaley J. Development and validation of a multifactorial risk index for predicting postoperative pneumonia after major noncardiac surgery. Ann Intern Med. 2001;135:847-57. [PMID: 11712875]
44.
Gilbert KLarocque BJPatrick LT. Prospective evaluation of cardiac risk indices for patients undergoing noncardiac surgery. Ann Intern Med. 2000;133:356-9. [PMID: 10979880]
45.
Wells PSAnderson DRRodger MStiell IDreyer JFBarnes Det al. Excluding pulmonary embolism at the bedside without diagnostic imaging: management of patients with suspected pulmonary embolism presenting to the emergency department by using a simple clinical model and d-dimer. Ann Intern Med. 2001;135:98-107. [PMID: 11453709]
46.
Hoffman JRMower WRWolfson ABTodd KHZucker MI. Validity of a set of clinical criteria to rule out injury to the cervical spine in patients with blunt trauma. National Emergency X-Radiography Utilization Study Group. N Engl J Med. 2000;343:94-9. [PMID: 10891516]
47.
Stiell IGClement CMMcKnight RDBrison RSchull MJRowe BHet al. The Canadian C-spine rule versus the NEXUS low-risk criteria in patients with trauma. N Engl J Med. 2003;349:2510-8. [PMID: 14695411]
48.
Wells PSAnderson DRRodger MForgie MKearon CDreyer Jet al. Evaluation of D-dimer in the diagnosis of suspected deep-vein thrombosis. N Engl J Med. 2003;349:1227-35. [PMID: 14507948]
49.
Marrie TJLau CYWheeler SLWong CJVandervoort MKFeagan BG. A controlled trial of a critical pathway for treatment of community-acquired pneumonia. CAPITAL Study Investigators. Community-Acquired Pneumonia Intervention Trial Assessing Levofloxacin. JAMA. 2000;283:749-55. [PMID: 10683053]
50.
English DRBurton RCdel Mar CBDonovan RJIreland PDEmery G. Evaluation of aid to diagnosis of pigmented skin lesions in general practice: controlled trial randomised by practice. BMJ. 2003;327:375. [PMID: 12919990]
51.
Montgomery AAFahey TPeters TJMacIntosh CSharp DJ. Evaluation of computer based clinical decision support system and risk chart for management of hypertension in primary care: randomised controlled trial. BMJ. 2000;320:686-90. [PMID: 10710578]
52.
Reilly BMEvans ATSchaider JJDas KCalvin JEMoran LAet al. Impact of a clinical decision rule on hospital triage of patients with suspected acute cardiac ischemia in the emergency department. JAMA. 2002;288:342-50. [PMID: 12117399]
53.
Pozen MWD'Agostino RBSelker HPSytkowski PAHood WB Jr. A predictive instrument to improve coronary-care-unit admission practices in acute ischemic heart disease. A prospective multicenter clinical trial. N Engl J Med. 1984;310:1273-8. [PMID: 6371525]
54.
Brush JE JrBrand DAAcampora DChalmer BWackers FJ. Use of the initial electrocardiogram to predict in-hospital complications of acute myocardial infarction. N Engl J Med. 1985;312:1137-41. [PMID: 3920520]
55.
Goldman LCook EFBrand DALee THRouan GWWeisberg MCet al. A computer protocol to predict myocardial infarction in emergency department patients with chest pain. N Engl J Med. 1988;318:797-803. [PMID: 3280998]
56.
Reilly BDurairaj LHusain SAcob CEvans AHu TCet al. Performance and potential impact of a chest pain prediction rule in a large public hospital. Am J Med. 1999;106:285-91. [PMID: 10190376]
57.
Durairaj LReilly BDas KSmith CAcob CHusain Set al. Emergency department admissions to inpatient cardiac telemetry beds: a prospective cohort study of risk stratification and outcomes. Am J Med. 2001;110:7-11. [PMID: 11152858]
58.
Reilly BMEvans ATSchaider JJWang Y. Triage of patients with chest pain in the emergency department: a comparative study of physicians' decisions. Am J Med. 2002;112:95-103. [PMID: 11835946]
59.
Grol RDalhuijsen JThomas SVeld CRutten GMokkink H. Attributes of clinical guidelines that influence use of guidelines in general practice: observational study. BMJ. 1998;317:858-61. [PMID: 9748183]
60.
Grimshaw JMShirran LThomas RMowatt GFraser CBero Let al. Changing provider behavior: an overview of systematic reviews of interventions. Med Care. 2001;39:II2-45. [PMID: 11583120]
61.
Rutter DQuine L. Changing Health Behavior: Intervention and Research with Social Cognition Models. Buckingham, United Kingdom: Open University Pr; 2002.
62.
Michie SJohnston M. Changing clinical behaviour by making guidelines specific. BMJ. 2004;328:343-5. [PMID: 14764503]
63.
Kawamoto KHoulihan CABalas EALobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ. 2005;330:765. [PMID: 15767266]
64.
Goldman LKirtane AJ. Triage of patients with acute chest pain and possible cardiac ischemia: the elusive search for diagnostic perfection. Ann Intern Med. 2003;139:987-95. [PMID: 14678918]
65.
Lucchesi GMJackson REPeacock WFCerasani CSwor RA. Sensitivity of the Ottawa rules. Ann Emerg Med. 1995;26:1-5. [PMID: 7793707]
66.
Verma SHamilton KHawkins HHKothari RSingal BBuncher Ret al. Clinical application of the Ottawa ankle rules for the use of radiography in acute ankle injuries: an independent site assessment. AJR Am J Roentgenol. 1997;169:825-7. [PMID: 9275906]
67.
Hurwitz B. How does evidence based guidance influence determinations of medical negligence? BMJ. 2004;329:1024-8. [PMID: 15514351]
68.
Genuis SJGenuis SK. Resisting cookbook medicine. BMJ. 2004;329:179.
69.
Cabana MDRand CSPowe NRWu AWWilson MHAbboud PAet al. Why don't physicians follow clinical practice guidelines? A framework for improvement. JAMA. 1999;282:1458-65. [PMID: 10535437]
70.
Grol R. Improving the quality of medical care: building bridges among professional pride, payer profit, and patient satisfaction. JAMA. 2001;286:2578-85. [PMID: 11722272]
71.
Casalino LP. The unintended consequences of measuring quality on the quality of medical care. N Engl J Med. 1999;341:1147-50. [PMID: 10511617]
72.
James BC. Making it easy to do it right [Editorial]. N Engl J Med. 2001;345:991-3. [PMID: 11575294]
73.
Weinstein MCStason WB. Foundations of cost-effectiveness analysis for health and medical practices. N Engl J Med. 1977;296:716-21. [PMID: 402576]
74.
Eddy DM. Clinical decision making: from theory to practice. Cost-effectiveness analysis. Is it up to the task? JAMA. 1992;267:3342-8. [PMID: 1597918]

Information & Authors

Information

Published In

cover image Annals of Internal Medicine
Annals of Internal Medicine
Volume 144Number 37 February 2006
Pages: 201 - 209

History

Published online: 7 February 2006
Published in issue: 7 February 2006

Keywords

Authors

Affiliations

Brendan M. Reilly, MD
From Cook County (Stroger) Hospital and Rush Medical College, Chicago, Illinois.
Arthur T. Evans, MD, MPH
From Cook County (Stroger) Hospital and Rush Medical College, Chicago, Illinois.
Disclosures: None disclosed.
Corresponding Author: Brendan M. Reilly, MD, Cook County (Stroger) Hospital, Administration Building, Room 1528, 1900 West Polk Street, Chicago, IL 60612; e-mail, [email protected].
Current Author Addresses: Dr. Reilly: Cook County (Stroger) Hospital, Administration Building, Room 1528, 1900 West Polk Street, Chicago, IL 60612.
Dr. Evans: Cook County (Stroger) Hospital, Administration Building, Room 1600, 1900 West Polk Street, Chicago, IL 60612.

Metrics & Citations

Metrics

Citations

If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. For an editable text file, please select Medlars format which will download as a .txt file. Simply select your manager software from the list below and click Download.

For more information or tips please see 'Downloading to a citation manager' in the Help menu.

Format





Download article citation data for:
Brendan M. Reilly, Arthur T. Evans. Translating Clinical Research into Clinical Practice: Impact of Using Prediction Rules To Make Decisions. Ann Intern Med.2006;144:201-209. [Epub 7 February 2006]. doi:10.7326/0003-4819-144-3-200602070-00009

View More

Get Access

Login Options:
Purchase

You will be redirected to acponline.org to sign-in to Annals to complete your purchase.

Create your Free Account

You will be redirected to acponline.org to create an account that will provide access to Annals.

View options

PDF/ePub

View PDF/ePub

Related in ACP Journals

Full Text

View Full Text

Media

Figures

Other

Tables

Share

Share

Copy the content Link

Share on social media