Advanced searchCitation search
Skip main navigation
Close Drawer MenuOpen Drawer Menu
Academia and Clinic7 February 2006

Translating Clinical Research into Clinical Practice: Impact of Using Prediction Rules To Make Decisions

    Author, Article, and Disclosure Information
    https://doi.org/10.7326/0003-4819-144-3-200602070-00009

    Abstract

    Clinical prediction rules, sometimes called clinical decision rules, have proliferated in recent years. However, very few have undergone formal impact analysis, the standard of evidence to assess their impact on patient care. Without impact analysis, clinicians cannot know whether using a prediction rule will be beneficial or harmful. This paper reviews standards of evidence for developing and evaluating prediction rules; important differences between prediction rules and decision rules; how to assess the potential clinical impact of a prediction rule before translating it into a decision rule; methodologic issues critical to successful impact analysis, including defining outcome measures and estimating sample size; the importance of close collaboration between clinical investigators and practicing clinicians before, during, and after impact analysis; and the need to measure both efficacy and effectiveness when analyzing a decision rule's clinical impact. These considerations should inform future development, evaluation, and use of all clinical prediction or decision rules.

    References

    • 1. Wasson JH, Sox HC, Neff RK, Goldman L. Clinical prediction rules. Applications and methodological standards. N Engl J Med. 1985;313:793-9. [PMID: 3897864] CrossrefMedlineGoogle Scholar
    • 2. Laupacis A, Sekar N, Stiell IG. Clinical prediction rules. A review and suggested modifications of methodological standards. JAMA. 1997;277:488-94. [PMID: 9020274] CrossrefMedlineGoogle Scholar
    • 3. Wells PS, Anderson DR, Bormanis J, Guy F, Mitchell M, Gray L, et al. Value of assessment of pretest probability of deep venous thrombosis in clinical management. Lancet. 1997;350:1795-8. [PMID: 9428249] CrossrefMedlineGoogle Scholar
    • 4. Goldman L, Cook EF, Johnson PA, Brand DA, Rouan GW, Lee TH. Prediction of the need for intensive care in patients who come to the emergency departments with acute chest pain. N Engl J Med. 1996;334:1498-504. [PMID: 8618604] CrossrefMedlineGoogle Scholar
    • 5. Stiell IG, McKnight RD, Greenberg GH, McDowell I, Nair RC, Wells GA, et al. Implementation of the Ottawa ankle rules. JAMA. 1994;271:827-32. [PMID: 8114236] CrossrefMedlineGoogle Scholar
    • 6. McGinn TG, Guyatt GH, Wyer PC, Naylor CD, Stiell IG, Richardson WS. Users' guides to the medical literature: XXII: how to use articles about clinical decision rules. Evidence-Based Medicine Working Group. JAMA. 2000;284:79-84. [PMID: 10872017] CrossrefMedlineGoogle Scholar
    • 7. Lee TH, Pearson SD, Johnson PA, Garcia TB, Weisberg MC, Guadagnoli E, et al. Failure of information as an intervention to modify clinical management. A time-series trial in patients with acute chest pain. Ann Intern Med. 1995;122:434-7. [PMID: 7856992] LinkGoogle Scholar
    • 8. Selker HP, Beshansky JR, Griffith JL, Aufderheide TP, Ballin DS, Bernard SA, et al. Use of the acute cardiac ischemia time-insensitive predictive instrument (ACI-TIPI) to assist with triage of patients with chest pain or other symptoms suggestive of acute cardiac ischemia. A multicenter, controlled clinical trial. Ann Intern Med. 1998;129:845-55. [PMID: 9867725] LinkGoogle Scholar
    • 9. Brand DA, Frazier WH, Kohlhepp WC, Shea KM, Hoefer AM, Ecker MD, et al. A protocol for selecting patients with injured extremities who need x-rays. N Engl J Med. 1982;306:333-9. [PMID: 7054709] CrossrefMedlineGoogle Scholar
    • 10. Cameron C, Naylor CD. No impact from active dissemination of the Ottawa Ankle Rules: further evidence of the need for local implementation of practice guidelines. CMAJ. 1999;160:1165-8. [PMID: 10234347] MedlineGoogle Scholar
    • 11. Lee TH. Evaluating decision aids: the next painful step [Editorial]. J Gen Intern Med. 1990;5:528-9. [PMID: 2266437] CrossrefMedlineGoogle Scholar
    • 12. Wang TJ, Massaro JM, Levy D, Vasan RS, Wolf PA, D'Agostino RB, et al. A risk score for predicting stroke or death in individuals with new-onset atrial fibrillation in the community: the Framingham Heart Study. JAMA. 2003;290:1049-56. [PMID: 12941677] CrossrefMedlineGoogle Scholar
    • 13. Stiell IG, Wells GA, Vandemheen KL, Clement CM, Lesiuk H, De Maio VJ, et al. The Canadian C-spine rule for radiography in alert and stable trauma patients. JAMA. 2001;286:1841-8. [PMID: 11597285] CrossrefMedlineGoogle Scholar
    • 14. Mehr DR, Binder EF, Kruse RL, Zweig SC, Madsen R, Popejoy L, et al. Predicting mortality in nursing home residents with lower respiratory tract infection: The Missouri LRI Study. JAMA. 2001;286:2427-36. [PMID: 11712938] CrossrefMedlineGoogle Scholar
    • 15. Collins S, Myatt M. Short-term prognosis in severe adult and adolescent malnutrition during famine: use of a simple prognostic model based on counting clinical signs. JAMA. 2000;284:621-6. [PMID: 10918712] CrossrefMedlineGoogle Scholar
    • 16. Johnston SC, Gress DR, Browner WS, Sidney S. Short-term prognosis after emergency department diagnosis of TIA. JAMA. 2000;284:2901-6. [PMID: 11147987] CrossrefMedlineGoogle Scholar
    • 17. Boersma E, Poldermans D, Bax JJ, Steyerberg EW, Thomson IR, Banga JD, et al. Predictors of cardiac events after major vascular surgery: role of clinical characteristics, dobutamine echocardiography, and beta-blocker therapy. JAMA. 2001;285:1865-73. [PMID: 11308400] CrossrefMedlineGoogle Scholar
    • 18. Ferreira FL, Bota DP, Bross A, Mélot C, Vincent JL. Serial evaluation of the SOFA score to predict outcome in critically ill patients. JAMA. 2001;286:1754-8. [PMID: 11594901] CrossrefMedlineGoogle Scholar
    • 19. Nielsen OW, Hansen JF, Hilden J, Larsen CT, Svanegaard J. Risk assessment of left ventricular systolic dysfunction in primary care: cross sectional study evaluating a range of diagnostic tests. BMJ. 2000;320:220-4. [PMID: 10642232] CrossrefMedlineGoogle Scholar
    • 20. Hasbun R, Abrahams J, Jekel J, Quagliarello VJ. Computed tomography of the head before lumbar puncture in adults with suspected meningitis. N Engl J Med. 2001;345:1727-33. [PMID: 11742046] CrossrefMedlineGoogle Scholar
    • 21. Benavente O, Eliasziw M, Streifler JY, Fox AJ, Barnett HJ, Meldrum H, et al. Prognosis after transient monocular blindness associated with carotid-artery stenosis. N Engl J Med. 2001;345:1084-90. [PMID: 11596587] CrossrefMedlineGoogle Scholar
    • 22. Bailey WC, Gerald LB, Kimerling ME, Redden D, Brook N, Bruce F, et al. Predictive model to identify positive tuberculosis skin test results during contact investigations. JAMA. 2002;287:996-1002. [PMID: 11866647] CrossrefMedlineGoogle Scholar
    • 23. Walter LC, Brand RJ, Counsell SR, Palmer RM, Landefeld CS, Fortinsky RH, et al. Development and validation of a prognostic index for 1-year mortality in older adults after hospitalization. JAMA. 2001;285:2987-94. [PMID: 11410097] CrossrefMedlineGoogle Scholar
    • 24. van Walraven C, Forster AJ, Parish DC, Dane FC, Chandra KM, Durham MD, et al. Validation of a clinical decision aid to discontinue in-hospital cardiac arrest resuscitations. JAMA. 2001;285:1602-6. [PMID: 11268268] CrossrefMedlineGoogle Scholar
    • 25. Ozuah PO, Ozuah TP, Stein RE, Burton W, Mulvihill M. Evaluation of a risk assessment questionnaire used to target tuberculin skin testing in children. JAMA. 2001;285:451-3. [PMID: 11242430] CrossrefMedlineGoogle Scholar
    • 26. Berkley JA, Ross A, Mwangi I, Osier FH, Mohammed M, Shebbe M, et al. Prognostic indicators of early and late death in children admitted to district hospital in Kenya: cohort study. BMJ. 2003;326:361. [PMID: 12586667] CrossrefMedlineGoogle Scholar
    • 27. Imperiale TF, Wagner DR, Lin CY, Larkin GN, Rogge JD, Ransohoff DF. Using risk for advanced proximal colonic neoplasia to tailor endoscopic screening for colorectal cancer. Ann Intern Med. 2003;139:959-65. [PMID: 14678915] LinkGoogle Scholar
    • 28. Haydel MJ, Preston CA, Mills TJ, Luber S, Blaudeau E, DeBlieux PM. Indications for computed tomography in patients with minor head injury. N Engl J Med. 2000;343:100-5. [PMID: 10891517] CrossrefMedlineGoogle Scholar
    • 29. Casey BM, McIntire DD, Leveno KJ. The continuing value of the Apgar score for the assessment of newborn infants. N Engl J Med. 2001;344:467-71. [PMID: 11172187] CrossrefMedlineGoogle Scholar
    • 30. Miller ST, Sleeper LA, Pegelow CH, Enos LE, Wang WC, Weiner SJ, et al. Prediction of adverse outcomes in children with sickle cell disease. N Engl J Med. 2000;342:83-9. [PMID: 10631276] CrossrefMedlineGoogle Scholar
    • 31. Weber JE, Shofer FS, Larkin GL, Kalaria AS, Hollander JE. Validation of a brief observation period for patients with cocaine-associated chest pain. N Engl J Med. 2003;348:510-7. [PMID: 12571258] CrossrefMedlineGoogle Scholar
    • 32. Lee DS, Austin PC, Rouleau JL, Liu PP, Naimark D, Tu JV. Predicting mortality among patients hospitalized for heart failure: derivation and validation of a clinical model. JAMA. 2003;290:2581-7. [PMID: 14625335] CrossrefMedlineGoogle Scholar
    • 33. Morrow DA, Antman EM, Parsons L, de Lemos JA, Cannon CP, Giugliano RP, et al. Application of the TIMI risk score for ST-elevation MI in the National Registry of Myocardial Infarction 3. JAMA. 2001;286:1356-9. [PMID: 11560541] CrossrefMedlineGoogle Scholar
    • 34. Cadarette SM, Jaglal SB, Murray TM, McIsaac WJ, Joseph L, Brown JP, et al. Evaluation of decision rules for referring women for bone densitometry by dual-energy x-ray absorptiometry. JAMA. 2001;286:57-63. [PMID: 11434827] CrossrefMedlineGoogle Scholar
    • 35. Hasbun R, Vikram HR, Barakat LA, Buenconsejo J, Quagliarello VJ. Complicated left-sided native valve endocarditis in adults: risk classification for mortality. JAMA. 2003;289:1933-40. [PMID: 12697795] CrossrefMedlineGoogle Scholar
    • 36. D'Agostino RB, Grundy S, Sullivan LM, Wilson P. Validation of the Framingham coronary heart disease prediction scores: results of a multiple ethnic groups investigation. JAMA. 2001;286:180-7. [PMID: 11448281] CrossrefMedlineGoogle Scholar
    • 37. Gage BF, Waterman AD, Shannon W, Boechler M, Rich MW, Radford MJ. Validation of clinical classification schemes for predicting stroke: results from the National Registry of Atrial Fibrillation. JAMA. 2001;285:2864-70. [PMID: 11401607] CrossrefMedlineGoogle Scholar
    • 38. Antman EM, Cohen M, Bernink PJ, McCabe CH, Horacek T, Papuchis G, et al. The TIMI risk score for unstable angina/non-ST elevation MI: a method for prognostication and therapeutic decision making. JAMA. 2000;284:835-42. [PMID: 10938172] CrossrefMedlineGoogle Scholar
    • 39. Brindle P, Emberson J, Lampe F, Walker M, Whincup P, Fahey T, et al. Predictive accuracy of the Framingham coronary risk score in British men: prospective cohort study. BMJ. 2003;327:1267. [PMID: 14644971] CrossrefMedlineGoogle Scholar
    • 40. Daly K, Beale R, Chang RW. Reduction in mortality after inappropriate early discharge from intensive care unit: logistic regression triage model. BMJ. 2001;322:1274-6. [PMID: 11375229] CrossrefMedlineGoogle Scholar
    • 41. Schoonhoven L, Haalboom JR, Bousema MT, Algra A, Grobbee DE, Grypdonck MH, et al. Prospective cohort study of routine use of risk assessment scales for prediction of pressure ulcers. BMJ. 2002;325:797. [PMID: 12376437] CrossrefMedlineGoogle Scholar
    • 42. Ramachandran S, French JM, Vanderpump MP, Croft P, Neary RH. Using the Framingham model to predict heart disease in the United Kingdom: retrospective study. BMJ. 2000;320:676-7. [PMID: 10710574] CrossrefMedlineGoogle Scholar
    • 43. Arozullah AM, Khuri SF, Henderson WG, Daley J. Development and validation of a multifactorial risk index for predicting postoperative pneumonia after major noncardiac surgery. Ann Intern Med. 2001;135:847-57. [PMID: 11712875] LinkGoogle Scholar
    • 44. Gilbert K, Larocque BJ, Patrick LT. Prospective evaluation of cardiac risk indices for patients undergoing noncardiac surgery. Ann Intern Med. 2000;133:356-9. [PMID: 10979880] LinkGoogle Scholar
    • 45. Wells PS, Anderson DR, Rodger M, Stiell I, Dreyer JF, Barnes D, et al. Excluding pulmonary embolism at the bedside without diagnostic imaging: management of patients with suspected pulmonary embolism presenting to the emergency department by using a simple clinical model and d-dimer. Ann Intern Med. 2001;135:98-107. [PMID: 11453709] LinkGoogle Scholar
    • 46. Hoffman JR, Mower WR, Wolfson AB, Todd KH, Zucker MI. Validity of a set of clinical criteria to rule out injury to the cervical spine in patients with blunt trauma. National Emergency X-Radiography Utilization Study Group. N Engl J Med. 2000;343:94-9. [PMID: 10891516] CrossrefMedlineGoogle Scholar
    • 47. Stiell IG, Clement CM, McKnight RD, Brison R, Schull MJ, Rowe BH, et al. The Canadian C-spine rule versus the NEXUS low-risk criteria in patients with trauma. N Engl J Med. 2003;349:2510-8. [PMID: 14695411] CrossrefMedlineGoogle Scholar
    • 48. Wells PS, Anderson DR, Rodger M, Forgie M, Kearon C, Dreyer J, et al. Evaluation of D-dimer in the diagnosis of suspected deep-vein thrombosis. N Engl J Med. 2003;349:1227-35. [PMID: 14507948] CrossrefMedlineGoogle Scholar
    • 49. Marrie TJ, Lau CY, Wheeler SL, Wong CJ, Vandervoort MK, Feagan BG. A controlled trial of a critical pathway for treatment of community-acquired pneumonia. CAPITAL Study Investigators. Community-Acquired Pneumonia Intervention Trial Assessing Levofloxacin. JAMA. 2000;283:749-55. [PMID: 10683053] CrossrefMedlineGoogle Scholar
    • 50. English DR, Burton RC, del Mar CB, Donovan RJ, Ireland PD, Emery G. Evaluation of aid to diagnosis of pigmented skin lesions in general practice: controlled trial randomised by practice. BMJ. 2003;327:375. [PMID: 12919990] CrossrefMedlineGoogle Scholar
    • 51. Montgomery AA, Fahey T, Peters TJ, MacIntosh C, Sharp DJ. Evaluation of computer based clinical decision support system and risk chart for management of hypertension in primary care: randomised controlled trial. BMJ. 2000;320:686-90. [PMID: 10710578] CrossrefMedlineGoogle Scholar
    • 52. Reilly BM, Evans AT, Schaider JJ, Das K, Calvin JE, Moran LA, et al. Impact of a clinical decision rule on hospital triage of patients with suspected acute cardiac ischemia in the emergency department. JAMA. 2002;288:342-50. [PMID: 12117399] CrossrefMedlineGoogle Scholar
    • 53. Pozen MW, D'Agostino RB, Selker HP, Sytkowski PA, Hood WB. A predictive instrument to improve coronary-care-unit admission practices in acute ischemic heart disease. A prospective multicenter clinical trial. N Engl J Med. 1984;310:1273-8. [PMID: 6371525] CrossrefMedlineGoogle Scholar
    • 54. Brush JE, Brand DA, Acampora D, Chalmer B, Wackers FJ. Use of the initial electrocardiogram to predict in-hospital complications of acute myocardial infarction. N Engl J Med. 1985;312:1137-41. [PMID: 3920520] CrossrefMedlineGoogle Scholar
    • 55. Goldman L, Cook EF, Brand DA, Lee TH, Rouan GW, Weisberg MC, et al. A computer protocol to predict myocardial infarction in emergency department patients with chest pain. N Engl J Med. 1988;318:797-803. [PMID: 3280998] CrossrefMedlineGoogle Scholar
    • 56. Reilly B, Durairaj L, Husain S, Acob C, Evans A, Hu TC, et al. Performance and potential impact of a chest pain prediction rule in a large public hospital. Am J Med. 1999;106:285-91. [PMID: 10190376] CrossrefMedlineGoogle Scholar
    • 57. Durairaj L, Reilly B, Das K, Smith C, Acob C, Husain S, et al. Emergency department admissions to inpatient cardiac telemetry beds: a prospective cohort study of risk stratification and outcomes. Am J Med. 2001;110:7-11. [PMID: 11152858] CrossrefMedlineGoogle Scholar
    • 58. Reilly BM, Evans AT, Schaider JJ, Wang Y. Triage of patients with chest pain in the emergency department: a comparative study of physicians' decisions. Am J Med. 2002;112:95-103. [PMID: 11835946] CrossrefMedlineGoogle Scholar
    • 59. Grol R, Dalhuijsen J, Thomas S, Veld C, Rutten G, Mokkink H. Attributes of clinical guidelines that influence use of guidelines in general practice: observational study. BMJ. 1998;317:858-61. [PMID: 9748183] CrossrefMedlineGoogle Scholar
    • 60. Grimshaw JM, Shirran L, Thomas R, Mowatt G, Fraser C, Bero L, et al. Changing provider behavior: an overview of systematic reviews of interventions. Med Care. 2001;39:II2-45. [PMID: 11583120] CrossrefMedlineGoogle Scholar
    • 61. Rutter D, Quine L. Changing Health Behavior: Intervention and Research with Social Cognition Models. Buckingham, United Kingdom: Open University Pr; 2002. Google Scholar
    • 62. Michie S, Johnston M. Changing clinical behaviour by making guidelines specific. BMJ. 2004;328:343-5. [PMID: 14764503] CrossrefMedlineGoogle Scholar
    • 63. Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ. 2005;330:765. [PMID: 15767266] CrossrefMedlineGoogle Scholar
    • 64. Goldman L, Kirtane AJ. Triage of patients with acute chest pain and possible cardiac ischemia: the elusive search for diagnostic perfection. Ann Intern Med. 2003;139:987-95. [PMID: 14678918] LinkGoogle Scholar
    • 65. Lucchesi GM, Jackson RE, Peacock WF, Cerasani C, Swor RA. Sensitivity of the Ottawa rules. Ann Emerg Med. 1995;26:1-5. [PMID: 7793707] CrossrefMedlineGoogle Scholar
    • 66. Verma S, Hamilton K, Hawkins HH, Kothari R, Singal B, Buncher R, et al. Clinical application of the Ottawa ankle rules for the use of radiography in acute ankle injuries: an independent site assessment. AJR Am J Roentgenol. 1997;169:825-7. [PMID: 9275906] CrossrefMedlineGoogle Scholar
    • 67. Hurwitz B. How does evidence based guidance influence determinations of medical negligence? BMJ. 2004;329:1024-8. [PMID: 15514351] CrossrefMedlineGoogle Scholar
    • 68. Genuis SJ, Genuis SK. Resisting cookbook medicine. BMJ. 2004;329:179. CrossrefGoogle Scholar
    • 69. Cabana MD, Rand CS, Powe NR, Wu AW, Wilson MH, Abboud PA, et al. Why don't physicians follow clinical practice guidelines? A framework for improvement. JAMA. 1999;282:1458-65. [PMID: 10535437] CrossrefMedlineGoogle Scholar
    • 70. Grol R. Improving the quality of medical care: building bridges among professional pride, payer profit, and patient satisfaction. JAMA. 2001;286:2578-85. [PMID: 11722272] CrossrefMedlineGoogle Scholar
    • 71. Casalino LP. The unintended consequences of measuring quality on the quality of medical care. N Engl J Med. 1999;341:1147-50. [PMID: 10511617] CrossrefMedlineGoogle Scholar
    • 72. James BC. Making it easy to do it right [Editorial]. N Engl J Med. 2001;345:991-3. [PMID: 11575294] CrossrefMedlineGoogle Scholar
    • 73. Weinstein MC, Stason WB. Foundations of cost-effectiveness analysis for health and medical practices. N Engl J Med. 1977;296:716-21. [PMID: 402576] CrossrefMedlineGoogle Scholar
    • 74. Eddy DM. Clinical decision making: from theory to practice. Cost-effectiveness analysis. Is it up to the task? JAMA. 1992;267:3342-8. [PMID: 1597918] CrossrefMedlineGoogle Scholar
    Previous
    Next

    Submit a Comment

    Contributors must reveal any conflict of interest. Comments are moderated. Please see our information for authorsregarding comments on an Annals publication.
    *All comments submitted after October 1, 2021 and selected for publication will be published online only.
    Cancel