Research and Reporting Methods21 January 2014
A Literature Review and Clinician's Guide
Abstract
The net reclassification improvement (NRI) is an increasingly popular measure for evaluating improvements in risk predictions. This article details a review of 67 publications in high-impact general clinical journals that considered the NRI. Incomplete reporting of NRI methods, incorrect calculation, and common misinterpretations were found. To aid improved applications of the NRI, the article elaborates on several aspects of the computation and interpretation in various settings. Limitations and controversies are discussed, including the effect of miscalibration of prediction models, the use of the continuous NRI and “clinical NRI,” and the relation with decision analytic measures. A systematic approach toward presenting NRI analysis is proposed: Detail and motivate the methods used for computation of the NRI, use clinically meaningful risk cutoffs for the category-based NRI, report both NRI components, address issues of calibration, and do not interpret the overall NRI as a percentage of the study population reclassified. Promising NRI findings need to be followed with decision analytic or formal cost-effectiveness evaluations.
References
- 1.
Kannel WB ,Dawber TR ,Kagan A ,Revotskie N ,Stokes J . Factors of risk in the development of coronary heart disease—six year follow-up experience. The Framingham Study. Ann Intern Med. 1961;55:33-50. [PMID: 13751193] LinkGoogle Scholar - 2.
Expert Panel on Detection, Evaluation .Treatment of High Blood Cholesterol in Adults . Executive Summary of The Third Report of The National Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation, And Treatment of High Blood Cholesterol In Adults (Adult Treatment Panel III). JAMA. 2001;285:2486-97. [PMID: 11368702] CrossrefMedlineGoogle Scholar - 3.
Perk J ,De Backer G ,Gohlke H ,Graham I ,Reiner Z ,Verschuren M ,et al ;European Association for Cardiovascular Prevention & Rehabilitation (EACPR) . European Guidelines on cardiovascular disease prevention in clinical practice (version 2012). The Fifth Joint Task Force of the European Society of Cardiology and Other Societies on Cardiovascular Disease Prevention in Clinical Practice (constituted by representatives of nine societies and by invited experts). Eur Heart J. 2012;33:1635-701. [PMID: 22555213] MedlineGoogle Scholar - 4.
Hamm CW ,Bassand JP ,Agewall S ,Bax J ,Boersma E ,Bueno H ,et al ;ESC Committee for Practice Guidelines . ESC Guidelines for the management of acute coronary syndromes in patients presenting without persistent ST-segment elevation: The Task Force for the management of acute coronary syndromes (ACS) in patients presenting without persistent ST-segment elevation of the European Society of Cardiology (ESC). Eur Heart J. 2011;32:2999-3054. [PMID: 21873419] CrossrefMedlineGoogle Scholar - 5.
Visvanathan K ,Chlebowski RT ,Hurley P ,Col NF ,Ropka M ,Collyar D ,et al ;American Society of Clinical Oncology . American Society of Clinical Oncology clinical practice guideline update on the use of pharmacologic interventions including tamoxifen, raloxifene, and aromatase inhibition for breast cancer risk reduction. J Clin Oncol. 2009;27:3235-58. [PMID: 19470930] CrossrefMedlineGoogle Scholar - 6.
Munshi NC ,Anderson KC ,Bergsagel PL ,Shaughnessy J ,Palumbo A ,Durie B ,et al ;International Myeloma Workshop Consensus Panel 2 . Consensus recommendations for risk stratification in multiple myeloma: report of the International Myeloma Workshop Consensus Panel 2. Blood. 2011;117:4696-700. [PMID: 21292777] CrossrefMedlineGoogle Scholar - 7.
Worth LJ ,Lingaratnam S ,Taylor A ,Hayward AM ,Morrissey S ,Cooney J ,et al ;Australian Consensus Guidelines 2011 Steering Committee . Use of risk stratification to guide ambulatory management of neutropenic fever. Australian Consensus Guidelines 2011 Steering Committee. Intern Med J. 2011;41:82-9. [PMID: 21272172] CrossrefMedlineGoogle Scholar - 8.
Bates SM ,Jaeschke R ,Stevens SM ,Goodacre S ,Wells PS ,Stevenson MD ,et al ;American College of Chest Physicians . Diagnosis of DVT: Antithrombotic Therapy and Prevention of Thrombosis, 9th ed: American College of Chest Physicians Evidence-Based Clinical Practice Guidelines. Chest. 2012;141:351S-418S. [PMID: 22315267] CrossrefMedlineGoogle Scholar - 9.
Pepe MS ,Janes H ,Longton G ,Leisenring W ,Newcomb P . Limitations of the odds ratio in gauging the performance of a diagnostic, prognostic, or screening marker. Am J Epidemiol. 2004;159:882-90. [PMID: 15105181] CrossrefMedlineGoogle Scholar - 10.
Cook NR . Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation. 2007;115:928-35. [PMID: 17309939] CrossrefMedlineGoogle Scholar - 11.
Pencina MJ ,D'Agostino RB ,Pencina KM ,Janssens ACJW ,Greenland P . Interpreting incremental value of markers added to risk prediction models. Am J Epidemiol. 2012;176:473-81. [PMID: 22875755] CrossrefMedlineGoogle Scholar - 12.
Austin PC ,Steyerberg EW . Predictive accuracy of risk factors and markers: a simulation study of the effect of novel markers on different performance measures for logistic regression models. Stat Med. 2013;32:661-72. [PMID: 22961910] CrossrefMedlineGoogle Scholar - 13.
Hlatky MA ,Greenland P ,Arnett DK ,Ballantyne CM ,Criqui MH ,Elkind MSV ,et al ;American Heart Association Expert Panel on Subclinical Atherosclerotic Diseases and Emerging Risk Factors and the Stroke Council . Criteria for evaluation of novel markers of cardiovascular risk: a scientific statement from the American Heart Association. Circulation. 2009;119:2408-16. [PMID: 19364974] CrossrefMedlineGoogle Scholar - 14.
Cook NR ,Buring JE ,Ridker PM . The effect of including C-reactive protein in cardiovascular risk prediction models for women. Ann Intern Med. 2006;145:21-9. [PMID: 16818925] LinkGoogle Scholar - 15.
Pencina MJ ,D'Agostino RB ,D'Agostino RB ,Vasan RS . Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med. 2008;27:157-72. [PMID: 17569110] CrossrefMedlineGoogle Scholar - 16.
Tzoulaki I ,Liberopoulos G ,Ioannidis JPA . Use of reclassification for assessment of improved prediction: an empirical evaluation. Int J Epidemiol. 2011;40:1094-105. [PMID: 21325392] CrossrefMedlineGoogle Scholar - 17.
Leening MJG ,Steyerberg EW . Fibrosis and mortality in patients with dilated cardiomyopathy [Letter]. JAMA. 2013;309:2547-8. [PMID: 23800926] CrossrefMedlineGoogle Scholar - 18.
Lilford RJ ,Richardson A ,Stevens A ,Fitzpatrick R ,Edwards S ,Rock F ,et al . Issues in methodological research: perspectives from researchers and commissioners. Health Technol Assess. 2001;5:1-57. [PMID: 11368832] CrossrefMedlineGoogle Scholar - 19.
Pencina MJ ,D'Agostino RB ,Vasan RS . Statistical methods for assessment of added usefulness of new biomarkers. Clin Chem Lab Med. 2010;48:1703-11. [PMID: 20716010] CrossrefMedlineGoogle Scholar - 20.
Pencina MJ ,D'Agostino RB ,Steyerberg EW . Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Stat Med. 2011;30:11-21. [PMID: 21204120] CrossrefMedlineGoogle Scholar - 21.
Pencina MJ ,D'Agostino RB ,Demler OV . Novel metrics for evaluating improvement in discrimination: net reclassification and integrated discrimination improvement for normal variables and nested models. Stat Med. 2012;31:101-13. [PMID: 22147389] CrossrefMedlineGoogle Scholar - 22.
Cook NR ,Ridker PM . Advances in measuring the effect of individual predictors of cardiovascular risk: the role of reclassification measures. Ann Intern Med. 2009;150:795-802. [PMID: 19487714] LinkGoogle Scholar - 23.
Adabag AS ,Therneau TM ,Gersh BJ ,Weston SA ,Roger VL . Sudden death after myocardial infarction. JAMA. 2008;300:2022-9. [PMID: 18984889] CrossrefMedlineGoogle Scholar - 24.
Auer R ,Bauer DC ,Marques-Vidal P ,Butler J ,Min LJ ,Cornuz J ,et al ;Health ABC Study . Association of major and minor ECG abnormalities with coronary heart disease events. JAMA. 2012;307:1497-505. [PMID: 22496264] CrossrefMedlineGoogle Scholar - 25.
Breteler MMB . Mapping out biomarkers for Alzheimer disease [Editorial]. JAMA. 2011;305:304-5. [PMID: 21245188] CrossrefMedlineGoogle Scholar - 26.
Buckley DI ,Fu R ,Freeman M ,Rogers K ,Helfand M . C-reactive protein as a risk factor for coronary heart disease: a systematic review and meta-analyses for the U.S. Preventive Services Task Force. Ann Intern Med. 2009;151:483-95. [PMID: 19805771] LinkGoogle Scholar - 27.
Chou R ,Arora B ,Dana T ,Fu R ,Walker M ,Humphrey L . Screening asymptomatic adults with resting or exercise electrocardiography: a review of the evidence for the U.S. Preventive Services Task Force. Ann Intern Med. 2011;155:375-85. [PMID: 21930855] LinkGoogle Scholar - 28.
Cook NR . Biomarkers for prediction of cardiovascular events [Letter]. JAMA. 2009;302:2089. [PMID: 19920231] CrossrefMedlineGoogle Scholar - 29.
Cornelis MC ,Qi L ,Zhang C ,Kraft P ,Manson J ,Cai T ,et al . Joint effects of common genetic variants on the risk for type 2 diabetes in U.S. men and women of European ancestry. Ann Intern Med. 2009;150:541-50. [PMID: 19380854] LinkGoogle Scholar - 30.
de Boer IH ,Levin G ,Robinson-Cohen C ,Biggs ML ,Hoofnagle AN ,Siscovick DS ,et al . Serum 25-hydroxyvitamin D concentration and risk for major clinical disease events in a community-based population of older adults: a cohort study. Ann Intern Med. 2012;156:627-34. [PMID: 22547472] LinkGoogle Scholar - 31.
de Lemos JA ,Drazner MH ,Omland T ,Ayers CR ,Khera A ,Rohatgi A ,et al . Association of troponin T detected with a highly sensitive assay and cardiac structure and mortality risk in the general population. JAMA. 2010;304:2503-12. [PMID: 21139111] CrossrefMedlineGoogle Scholar - 32.
deFilippi CR ,de Lemos JA ,Christenson RH ,Gottdiener JS ,Kop WJ ,Zhan M ,et al . Association of serial measures of cardiac troponin T using a sensitive assay with incident heart failure and cardiovascular mortality in older adults. JAMA. 2010;304:2494-502. [PMID: 21078811] CrossrefMedlineGoogle Scholar - 33.
den Ruijter HM ,Peters SAE ,Anderson TJ ,Britton AR ,Dekker JM ,Eijkemans MJ ,et al . Common carotid intima-media thickness measurements in cardiovascular risk prediction: a meta-analysis. JAMA. 2012;308:796-803. [PMID: 22910757] CrossrefMedlineGoogle Scholar - 34.
Devereaux PJ ,Chan MTV ,Alonso-Coello P ,Walsh M ,Berwanger O ,Villar JC ,et al ;Vascular Events In Noncardiac Surgery Patients Cohort Evaluation (VISION) Study Investigators . Association between postoperative troponin levels and 30-day mortality among patients undergoing noncardiac surgery. JAMA. 2012;307:2295-304. [PMID: 22706835] CrossrefMedlineGoogle Scholar - 35.
Di Angelantonio E ,Gao P ,Pennells L ,Kaptoge S ,Caslake M ,Thompson A ,et al ;Emerging Risk Factors Collaboration . Lipid-related markers and cardiovascular disease prediction. JAMA. 2012;307:2499-506. [PMID: 22797450] MedlineGoogle Scholar - 36.
Eddy DM ,Adler J ,Patterson B ,Lucas D ,Smith KA ,Morris M . Individualized guidelines: the potential for increasing quality and reducing costs. Ann Intern Med. 2011;154:627-34. [PMID: 21536939] LinkGoogle Scholar - 37.
Farooq V ,van Klaveren D ,Steyerberg EW ,Meliga E ,Vergouwe Y ,Chieffo A ,et al . Anatomical and clinical characteristics to guide decision making between coronary artery bypass surgery and percutaneous coronary intervention for individual patients: development and validation of SYNTAX Score II. Lancet. 2013;381:639-50. [PMID: 23439103] CrossrefMedlineGoogle Scholar - 38.
Fonarow GC ,Pan W ,Saver JL ,Smith EE ,Reeves MJ ,Broderick JP ,et al . Comparison of 30-day mortality models for profiling hospital performance in acute ischemic stroke with vs without adjustment for stroke severity. JAMA. 2012;308:257-64. [PMID: 22797643] CrossrefMedlineGoogle Scholar - 39.
Gulati A ,Jabbour A ,Ismail TF ,Guha K ,Khwaja J ,Raza S ,et al . Association of fibrosis with mortality and sudden cardiac death in patients with nonischemic dilated cardiomyopathy. JAMA. 2013;309:896-908. [PMID: 23462786] CrossrefMedlineGoogle Scholar - 40.
Helfand M ,Buckley DI ,Freeman M ,Fu R ,Rogers K ,Fleming C ,et al . Emerging risk factors for coronary heart disease: a summary of systematic reviews conducted for the U.S. Preventive Services Task Force. Ann Intern Med. 2009;151:496-507. [PMID: 19805772] LinkGoogle Scholar - 41.
Hingorani AD ,Psaty BM . Primary prevention of cardiovascular disease: time to get more or less personal? JAMA. 2009;302:2144-5. [PMID: 19920239] CrossrefMedlineGoogle Scholar - 42.
Hlatky MA . Framework for evaluating novel risk markers [Editorial]. Ann Intern Med. 2012;156:468-9. [PMID: 22431679] LinkGoogle Scholar - 43.
Janes H ,Pepe MS ,Gu W . Assessing the value of risk predictions by using risk stratification tables. Ann Intern Med. 2008;149:751-60. [PMID: 19017593] LinkGoogle Scholar - 44.
Janssens ACJW ,Ioannidis JPA ,van Duijn CM ,Little J ,Khoury MJ ;GRIPS Group . Strengthening the reporting of genetic risk prediction studies: The GRIPS Statement. Ann Intern Med. 2011;154:421-5. [PMID: 21403077] LinkGoogle Scholar - 45.
Kaptoge S ,Di Angelantonio E ,Lowe G ,Pepys MB ,Thompson SG ,Collins R ,et al ;Emerging Risk Factors Collaboration . C-reactive protein concentration and risk of coronary heart disease, stroke, and mortality: an individual participant meta-analysis. Lancet. 2010;375:132-40. [PMID: 20031199] CrossrefMedlineGoogle Scholar - 46.
Kaptoge S ,Di Angelantonio E ,Pennells L ,Wood AM ,White IR ,Gao P ,et al ;Emerging Risk Factors Collaboration . C-reactive protein, fibrinogen, and cardiovascular disease prediction. N Engl J Med. 2012;367:1310-20. [PMID: 23034020] CrossrefMedlineGoogle Scholar - 47.
Kathiresan S ,Melander O ,Anevski D ,Guiducci C ,Burtt NP ,Roos C ,et al . Polymorphisms associated with cholesterol and risk of cardiovascular events. N Engl J Med. 2008;358:1240-9. [PMID: 18354102] CrossrefMedlineGoogle Scholar - 48.
Kavousi M ,Elias-Smale S ,Rutten JHW ,Leening MJG ,Vliegenthart R ,Verwoert GC ,et al . Evaluation of newer risk markers for coronary heart disease risk classification: a cohort study. Ann Intern Med. 2012;156:438-44. [PMID: 22431676] LinkGoogle Scholar - 49.
Keller T ,Zeller T ,Ojeda F ,Tzikas S ,Lillpopp L ,Sinning C ,et al . Serial changes in highly sensitive troponin I assay and early diagnosis of myocardial infarction. JAMA. 2011;306:2684-93. [PMID: 22203537] CrossrefMedlineGoogle Scholar - 50.
Kengne AP ,Echouffo-Tcheugui JB ,Sobngwi E . Coronary artery calcium for guiding statin treatment [Letter]. Lancet. 2012;379:312. [PMID: 22284653] CrossrefMedlineGoogle Scholar - 51.
Khera AV ,Cuchel M ,de la Llera-Moya M ,Rodrigues A ,Burke MF ,Jafri K ,et al . Cholesterol efflux capacity, high-density lipoprotein function, and atherosclerosis. N Engl J Med. 2011;364:127-35. [PMID: 21226578] CrossrefMedlineGoogle Scholar - 52.
Kim WR ,Biggins SW ,Kremers WK ,Wiesner RH ,Kamath PS ,Benson JT ,et al . Hyponatremia and mortality among patients on the liver-transplant waiting list. N Engl J Med. 2008;359:1018-26. [PMID: 18768945] CrossrefMedlineGoogle Scholar - 53.
Kivimäki M ,Batty GD ,Hamer M ,Ferrie JE ,Vahtera J ,Virtanen M ,et al . Using additional information on working hours to predict coronary heart disease: a cohort study. Ann Intern Med. 2011;154:457-63. [PMID: 21464347] LinkGoogle Scholar - 54.
Koller MT ,Leening MJG ,Wolbers M ,Steyerberg EW ,Hunink MGM ,Schoop R ,et al . Development and validation of a coronary risk prediction model for older U.S. and European persons in the Cardiovascular Health Study and the Rotterdam Study. Ann Intern Med. 2012;157:389-97. [PMID: 22986376] LinkGoogle Scholar - 55.
Lubitz SA ,Yin X ,Fontes JD ,Magnani JW ,Rienstra M ,Pai M ,et al . Association between familial atrial fibrillation and risk of new-onset atrial fibrillation. JAMA. 2010;304:2263-9. [PMID: 21076174] CrossrefMedlineGoogle Scholar - 56.
Lyssenko V ,Jonsson A ,Almgren P ,Pulizzi N ,Isomaa B ,Tuomi T ,et al . Clinical risk factors, DNA variants, and the development of type 2 diabetes. N Engl J Med. 2008;359:2220-32. [PMID: 19020324] CrossrefMedlineGoogle Scholar - 57.
Manolio TA . Genomewide association studies and assessment of the risk of disease. N Engl J Med. 2010;363:166-76. [PMID: 20647212] CrossrefMedlineGoogle Scholar - 58.
Martínez ME ,Thompson P ,Messer K ,Ashbeck EL ,Lieberman DA ,Baron JA ,et al . One-year risk for advanced colorectal neoplasia: U.S. versus U.K. risk-stratification guidelines. Ann Intern Med. 2012;157:856-64. [PMID: 23247939] LinkGoogle Scholar - 59.
Matsushita K ,Mahmoodi BK ,Woodward M ,Emberson JR ,Jafar TH ,Jee SH ,et al ;Chronic Kidney Disease Prognosis Consortium . Comparison of risk prediction using the CKD-EPI equation and the MDRD study equation for estimated glomerular filtration rate. JAMA. 2012;307:1941-51. [PMID: 22570462] CrossrefMedlineGoogle Scholar - 60.
McEvoy JW . Coronary artery calcium score and cardiovascular event prediction [Letter]. JAMA. 2010;304:741-2. [PMID: 20716732] CrossrefMedlineGoogle Scholar - 61.
Meigs JB ,Shrader P ,Sullivan LM ,McAteer JB ,Fox CS ,Dupuis J ,et al . Genotype score in addition to common risk factors for prediction of type 2 diabetes. N Engl J Med. 2008;359:2208-19. [PMID: 19020323] CrossrefMedlineGoogle Scholar - 62.
Melander O ,Newton-Cheh C ,Almgren P ,Hedblad B ,Berglund G ,Engström G ,et al . Novel and conventional biomarkers for prediction of incident cardiovascular events in the community. JAMA. 2009;302:49-57. [PMID: 19567439] CrossrefMedlineGoogle Scholar - 63.
Melander O ,Newton-Cheh C ,Wang TJ . Biomarkers for prediction of cardiovascular events—reply. JAMA. 2009;302:2090. MedlineGoogle Scholar - 64.
Omland T ,de Lemos JA ,Sabatine MS ,Christophi CA ,Rice MM ,Jablonski KA ,et al ;Prevention of Events with Angiotensin Converting Enzyme Inhibition (PEACE) Trial Investigators . A sensitive cardiac troponin T assay in stable coronary artery disease. N Engl J Med. 2009;361:2538-47. [PMID: 19940289] CrossrefMedlineGoogle Scholar - 65.
Palomaki GE ,Melillo S ,Bradley LA . Association between 9p21 genomic markers and heart disease: a meta-analysis. JAMA. 2010;303:648-56. [PMID: 20159873] CrossrefMedlineGoogle Scholar - 66.
Paynter NP ,Chasman DI ,Buring JE ,Shiffman D ,Cook NR ,Ridker PM . Cardiovascular disease risk prediction with and without knowledge of genetic variation at chromosome 9p21.3. Ann Intern Med. 2009;150:65-72. [PMID: 19153409] LinkGoogle Scholar - 67.
Paynter NP ,Chasman DI ,Paré G ,Buring JE ,Cook NR ,Miletich JP ,et al . Association between a literature-based genetic risk score and cardiovascular events in women. JAMA. 2010;303:631-7. [PMID: 20159871] CrossrefMedlineGoogle Scholar - 68.
Peralta CA ,Shlipak MG ,Judd S ,Cushman M ,McClellan W ,Zakai NA ,et al . Detection of chronic kidney disease with creatinine, cystatin C, and urine albumin-to-creatinine ratio and association with progression to end-stage renal disease and mortality. JAMA. 2011;305:1545-52. [PMID: 21482744] CrossrefMedlineGoogle Scholar - 69.
Pischon T ,Boeing H ,Hoffmann K ,Bergmann M ,Schulze MB ,Overvad K ,et al . General and abdominal adiposity and risk of death in Europe. N Engl J Med. 2008;359:2105-20. [PMID: 19005195] CrossrefMedlineGoogle Scholar - 70.
Pletcher MJ ,Tice JA ,Pignone M . Modeling cardiovascular disease prevention [Letter]. JAMA. 2010;303:835. [PMID: 20197528] CrossrefMedlineGoogle Scholar - 71.
Polak JF ,Pencina MJ ,Pencina KM ,O'Donnell CJ ,Wolf PA ,D'Agostino RB . Carotid-wall intima-media thickness and cardiovascular events. N Engl J Med. 2011;365:213-21. [PMID: 21774709] CrossrefMedlineGoogle Scholar - 72.
Polonsky TS ,McClelland RL ,Jorgensen NW ,Bild DE ,Burke GL ,Guerci AD ,et al . Coronary artery calcium score and risk classification for coronary heart disease prediction. JAMA. 2010;303:1610-6. [PMID: 20424251] CrossrefMedlineGoogle Scholar - 73.
Ripatti S ,Tikkanen E ,Orho-Melander M ,Havulinna AS ,Silander K ,Sharma A ,et al . A multilocus genetic risk score for coronary heart disease: case-control and prospective cohort analyses. Lancet. 2010;376:1393-400. [PMID: 20971364] CrossrefMedlineGoogle Scholar - 74.
Rosenberg S ,Elashoff MR ,Beineke P ,Daniels SE ,Wingrove JA ,Tingley WG ,et al ;PREDICT (Personalized Risk Evaluation and Diagnosis in the Coronary Tree) Investigators . Multicenter validation of the diagnostic accuracy of a blood-based gene expression test for assessing obstructive coronary artery disease in nondiabetic patients. Ann Intern Med. 2010;153:425-34. [PMID: 20921541] LinkGoogle Scholar - 75.
Schelbert EB ,Cao JJ ,Sigurdsson S ,Aspelund T ,Kellman P ,Aletras AH ,et al . Prevalence and prognosis of unrecognized myocardial infarction determined by cardiac magnetic resonance in older adults. JAMA. 2012;308:890-6. [PMID: 22948699] CrossrefMedlineGoogle Scholar - 76.
Schnabel RB ,Sullivan LM ,Levy D ,Pencina MJ ,Massaro JM ,D'Agostino RB ,et al . Development of a risk score for atrial fibrillation (Framingham Heart Study): a community-based cohort study. Lancet. 2009;373:739-45. [PMID: 19249635] CrossrefMedlineGoogle Scholar - 77.
Selvin E ,Steffes MW ,Zhu H ,Matsushita K ,Wagenknecht L ,Pankow J ,et al . Glycated hemoglobin, diabetes, and cardiovascular risk in nondiabetic adults. N Engl J Med. 2010;362:800-11. [PMID: 20200384] CrossrefMedlineGoogle Scholar - 78.
Steyerberg EW ,Pencina MJ . Reclassification calculations for persons with incomplete follow-up [Letter]. Ann Intern Med. 2010;152:195-6. [PMID: 20124243] LinkGoogle Scholar - 79.
Tammemägi MC ,Katki HA ,Hocking WG ,Church TR ,Caporaso N ,Kvale PA ,et al . Selection criteria for lung-cancer screening. N Engl J Med. 2013;368:728-36. [PMID: 23425165] CrossrefMedlineGoogle Scholar - 80.
Tangri N ,Stevens LA ,Griffith J ,Tighiouart H ,Djurdjev O ,Naimark D ,et al . A predictive model for progression of chronic kidney disease to kidney failure. JAMA. 2011;305:1553-9. [PMID: 21482743] CrossrefMedlineGoogle Scholar - 81.
Tzoulaki I ,Liberopoulos G ,Ioannidis JPA . Assessment of claims of improved prediction beyond the Framingham risk score. JAMA. 2009;302:2345-52. [PMID: 19952321] CrossrefMedlineGoogle Scholar - 82.
Wacholder S ,Hartge P ,Prentice R ,Garcia-Closas M ,Feigelson HS ,Diver WR ,et al . Performance of common genetic variants in breast-cancer risk models. N Engl J Med. 2010;362:986-93. [PMID: 20237344] CrossrefMedlineGoogle Scholar - 83.
Wilson PWF . Challenges to improve coronary heart disease risk assessment [Editorial]. JAMA. 2009;302:2369-70. [PMID: 19952326] CrossrefMedlineGoogle Scholar - 84.
Wormser D ,Kaptoge S ,Di Angelantonio E ,Wood AM ,Pennells L ,Thompson A ,et al ;Emerging Risk Factors Collaboration . Separate and combined associations of body-mass index and abdominal adiposity with cardiovascular disease: collaborative analysis of 58 prospective studies. Lancet. 2011;377:1085-95. [PMID: 21397319] CrossrefMedlineGoogle Scholar - 85.
Wormser D ,Di Angelantonio E ,Sattar N ,Collins R ,Thompson S ,Danesh J ,et al . Body-mass index, abdominal adiposity, and cardiovascular risk - reply. Lancet. 2011;378:228. CrossrefGoogle Scholar - 86.
Yeboah J ,McClelland RL ,Polonsky TS ,Burke GL ,Sibley CT ,O'Leary D ,et al . Comparison of novel risk markers for improvement in cardiovascular risk assessment in intermediate-risk individuals. JAMA. 2012;308:788-95. [PMID: 22910756] CrossrefMedlineGoogle Scholar - 87.
Zethelius B ,Berglund L ,Sundström J ,Ingelsson E ,Basu S ,Larsson A ,et al . Use of multiple biomarkers to improve the prediction of death from cardiovascular causes. N Engl J Med. 2008;358:2107-16. [PMID: 18480203] CrossrefMedlineGoogle Scholar - 88.
Zoungas S ,Patel A ,Chalmers J ,de Galan BE ,Li Q ,Billot L ,et al ;ADVANCE Collaborative Group . Severe hypoglycemia and risks of vascular events and death. N Engl J Med. 2010;363:1410-8. [PMID: 20925543] CrossrefMedlineGoogle Scholar - 89.
Greenland P ,Alpert JS ,Beller GA ,Benjamin EJ ,Budoff MJ ,Fayad ZA ,et al ;American College of Cardiology Foundation .American Heart Association Task Force on Practice Guidelines . 2010 ACCF/AHA guideline for assessment of cardiovascular risk in asymptomatic adults: a report of the American College of Cardiology Foundation/American Heart Association Task Force on Practice Guidelines. Circulation. 2010;122:584-636. [PMID: 21098428] MedlineGoogle Scholar - 90.
Mosca L ,Benjamin EJ ,Berra K ,Bezanson JL ,Dolor RJ ,Lloyd-Jones DM ,et al . Effectiveness-based guidelines for the prevention of cardiovascular disease in women—2011 update: a guideline from the American Heart Association. Circulation. 2011;123:1243-62. [PMID: 21325087] CrossrefMedlineGoogle Scholar - 91.
Cai T ,Tian L ,Lloyd-Jones DM . Comparing costs associated with risk stratification rules for t-year survival. Biostatistics. 2011;12:597-609. [PMID: 21415016] CrossrefMedlineGoogle Scholar - 92.
Leening MJG ,Cook NR . Net reclassification improvement: a link between statistics and clinical practice. Eur J Epidemiol. 2013;28:21-3. [PMID: 23291877] CrossrefMedlineGoogle Scholar - 93.
Pepe MS . Problems with risk reclassification methods for evaluating prediction models. Am J Epidemiol. 2011;173:1327-35. [PMID: 21555714] CrossrefMedlineGoogle Scholar - 94.
Mihaescu R ,van Zitteren M ,van Hoek M ,Sijbrands EJ ,Uitterlinden AG ,Witteman JCM ,et al . Improvement of risk prediction by genomic profiling: reclassification measures versus the area under the receiver operating characteristic curve. Am J Epidemiol. 2010;172:353-61. [PMID: 20562194] CrossrefMedlineGoogle Scholar - 95.
Mühlenbruch K ,Heraclides A ,Steyerberg EW ,Joost HG ,Boeing H ,Schulze MB . Assessing improvement in disease prediction using net reclassification improvement: impact of risk cut-offs and number of risk categories. Eur J Epidemiol. 2013;28:25-33. [PMID: 23179629] CrossrefMedlineGoogle Scholar - 96.
Pepe MS ,Janes H . Commentary: Reporting standards are needed for evaluations of risk reclassification. Int J Epidemiol. 2011;40:1106-8. [PMID: 21571811] CrossrefMedlineGoogle Scholar - 97.
McGeechan K ,Macaskill P ,Irwig L ,Liew G ,Wong TY . Assessing new biomarkers and predictive models for use in clinical practice: a clinician's guide. Arch Intern Med. 2008;168:2304-10. [PMID: 19029492] CrossrefMedlineGoogle Scholar - 98.
Greenland S . The need for reorientation toward cost-effective prediction: comments on ‘Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond’ by M. J. Pencina et al., Statistics in Medicine (DOI: 10.1002/sim.2929). Stat Med. 2008;27:199-206. [PMID: 17729377] MedlineGoogle Scholar - 99.
Cook NR ,Paynter NP . Performance of reclassification statistics in comparing risk prediction models. Biom J. 2011;53:237-58. [PMID: 21294152] CrossrefMedlineGoogle Scholar - 100.
National Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation .Treatment of High Blood Cholesterol in Adults (Adult Treatment Panel III) . Third Report of the National Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults (Adult Treatment Panel III) final report. Circulation. 2002;106:3143-421. [PMID: 12485966] CrossrefMedlineGoogle Scholar - 101.
Cook NR ,Paynter NP ,Eaton CB ,Manson JE ,Martin LW ,Robinson JG ,et al . Comparison of the Framingham and Reynolds Risk scores for global cardiovascular risk prediction in the multiethnic Women's Health Initiative. Circulation. 2012;125:1748-56, S1-11. [PMID: 22399535] CrossrefMedlineGoogle Scholar - 102.
Pencina MJ ,D'Agostino RB ,Larson MG ,Massaro JM ,Vasan RS . Predicting the 30-year risk of cardiovascular disease: the Framingham Heart Study. Circulation. 2009;119:3078-84. [PMID: 19506114] CrossrefMedlineGoogle Scholar - 103.
Takahara M ,Katakami N ,Kaneto H ,Shimomura I . Risk categorization for calculating net reclassification improvement [Letter]. Eur J Epidemiol. 2013;28:607-9. [PMID: 23839540] CrossrefMedlineGoogle Scholar - 104.
Ganna A ,Reilly M ,de Faire U ,Pedersen N ,Magnusson P ,Ingelsson E . Risk prediction measures for case-cohort and nested case-control designs: an application to cardiovascular disease. Am J Epidemiol. 2012;175:715-24. [PMID: 22396388] CrossrefMedlineGoogle Scholar - 105.
Pepe MS ,Fan J ,Seymour CW ,Li C ,Huang Y ,Feng Z . Biases introduced by choosing controls to match risk factors of cases in biomarker research. Clin Chem. 2012;58:1242-51. [PMID: 22730452] CrossrefMedlineGoogle Scholar - 106.
Rothman KJ ,Greenland S ,Lash TL . Modern Epidemiology. 3rd ed. Philadelphia: Lippincott Williams & Wilkins; 2008. Google Scholar - 107.
Kavousi M ,Leening MJG ,Witteman JCM . Markers for prediction of cardiovascular disease risk [Letter]. JAMA. 2012;308:2561. [PMID: 23268505] CrossrefMedlineGoogle Scholar - 108.
Pencina MJ ,D'Agostino RB ,D'Agostino RB ,Vasan RS . Comments on ‘Integrated discrimination and net reclassification improvements—Practical advice’. Stat Med. 2008;27:207-12. [PMID: 17569110] CrossrefMedlineGoogle Scholar - 109.
Pepe MS ,Feng Z ,Gu JW . Comments on ‘Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond’ by M. J. Pencina et al., Statistics in Medicine (DOI: 10.1002/sim.2929). Stat Med. 2008;27:173-81. [PMID: 17671958] CrossrefMedlineGoogle Scholar - 110.
Hilden J ,Gerds TA . A note on the evaluation of novel biomarkers: do not rely on integrated discrimination improvement and net reclassification index. Stat Med. 2013;:. [PMID: 23553436] MedlineGoogle Scholar - 111.
Brindle P ,Emberson J ,Lampe F ,Walker M ,Whincup P ,Fahey T ,et al . Predictive accuracy of the Framingham coronary risk score in British men: prospective cohort study. BMJ. 2003;327:1267. [PMID: 14644971] CrossrefMedlineGoogle Scholar - 112.
Hense HW ,Schulte H ,Löwel H ,Assmann G ,Keil U . Framingham risk function overestimates risk of coronary heart disease in men and women from Germany—results from the MONICA Augsburg and the PROCAM cohorts. Eur Heart J. 2003;24:937-45. [PMID: 12714025] CrossrefMedlineGoogle Scholar - 113.
Koller MT ,Steyerberg EW ,Wolbers M ,Stijnen T ,Bucher HC ,Hunink MGM ,et al . Validity of the Framingham point scores in the elderly: results from the Rotterdam study. Am Heart J. 2007;154:87-93. [PMID: 17584559] CrossrefMedlineGoogle Scholar - 114.
Merry AH ,Boer JM ,Schouten LJ ,Ambergen T ,Steyerberg EW ,Feskens EJM ,et al . Risk prediction of incident coronary heart disease in The Netherlands: re-estimation and improvement of the SCORE risk function. Eur J Prev Cardiol. 2012;19:840-8. [PMID: 21551214] CrossrefMedlineGoogle Scholar - 115.
Siontis GCM ,Tzoulaki I ,Siontis KC ,Ioannidis JPA . Comparisons of established risk prediction models for cardiovascular disease: systematic review. BMJ. 2012;344:3318. [PMID: 22628003] CrossrefMedlineGoogle Scholar - 116.
Steyerberg EW . Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. New York: Springer; 2009. Google Scholar - 117.
Kramer AA ,Zimmerman JE . Assessing the calibration of mortality benchmarks in critical care: The Hosmer-Lemeshow test revisited. Crit Care Med. 2007;35:2052-6. [PMID: 17568333] CrossrefMedlineGoogle Scholar - 118.
Cox DR . Two further applications of a model for binary regression. Biometrika. 1958;45:562-5. CrossrefGoogle Scholar - 119.
Cook NR ,Paynter NP . Comments on ‘Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers’ by M. J. Pencina, R. B. D'Agostino, Sr. and E. W. Steyerberg [Letter]. Stat Med. 2012;31:93-5. [PMID: 21344474] CrossrefMedlineGoogle Scholar - 120.
Vergouwe Y ,Moons KGM ,Steyerberg EW . External validity of risk models: Use of benchmark values to disentangle a case-mix effect from incorrect coefficients. Am J Epidemiol. 2010;172:971-80. [PMID: 20807737] CrossrefMedlineGoogle Scholar - 121.
Cook NR . Comments on ‘Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond’ by M. J. Pencina et al., Statistics in Medicine (DOI: 10.1002/sim.2929). Stat Med. 2008;27:191-5. [PMID: 17671959] CrossrefMedlineGoogle Scholar - 122.
Paynter NP ,Cook NR . A bias-corrected net reclassification improvement for clinical subgroups. Med Decis Making. 2013;33:154-62. [PMID: 23042826] CrossrefMedlineGoogle Scholar - 123.
Vickers AJ ,Elkin EB ,Steyerberg E . Net reclassification improvement and decision theory [Letter]. Stat Med. 2009;28:525-6. [PMID: 17907248] CrossrefMedlineGoogle Scholar - 124.
Van Calster B ,Vickers AJ ,Pencina MJ ,Baker SG ,Timmerman D ,Steyerberg EW . Evaluation of markers and risk prediction models: overview of relationships between NRI and decision-analytic measures. Med Decis Making. 2013;33:490-501. [PMID: 23313931] CrossrefMedlineGoogle Scholar - 125.
Localio AR ,Goodman S . Beyond the usual prediction accuracy metrics: reporting results for clinical decision making [Editorial]. Ann Intern Med. 2012;157:294-5. [PMID: 22910942] LinkGoogle Scholar - 126.
Steyerberg EW ,Vickers AJ ,Cook NR ,Gerds T ,Gonen M ,Obuchowski N ,et al . Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21:128-38. [PMID: 20010215] CrossrefMedlineGoogle Scholar - 127.
Vickers AJ ,Elkin EB . Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making. 2006;26:565-74. [PMID: 17099194] CrossrefMedlineGoogle Scholar - 128.
Vickers AJ ,Cronin AM ,Begg CB . One statistical test is sufficient for assessing new predictive markers. BMC Med Res Methodol. 2011;11:13. [PMID: 21276237] CrossrefMedlineGoogle Scholar - 129.
Pepe MS ,Kerr KF ,Longton G ,Wang Z . Testing for improvement in prediction model performance. Stat Med. 2013;32:1467-82. [PMID: 23296397] CrossrefMedlineGoogle Scholar - 130.
McGeechan K ,Liew G ,Macaskill P ,Irwig L ,Klein R ,Sharrett AR ,et al . Risk prediction of coronary heart disease based on retinal vascular caliber (from the Atherosclerosis Risk In Communities [ARIC] Study). Am J Cardiol. 2008;102:58-63. [PMID: 18572036] CrossrefMedlineGoogle Scholar
Author, Article, and Disclosure Information
Authors:
Maarten J.G. Leening,
From Erasmus MC - University Medical Center Rotterdam, Rotterdam, the Netherlands, and Duke University, Durham, North Carolina.
Grant Support: By the Netherlands Organisation for Health Research and Development (ZonMw) and the Netherlands Organisation for Scientific Research (NWO) (ZonMw HTA grant 80-82500-98-10208; Vici grant 918-76-619) and the Center for Translational Molecular Medicine (PCMM project grant). The funding sources had no role in the design or conduct of the study; the collection, management, analysis, or interpretation of the data; or the preparation, review, or approval of the manuscript.
Disclosures: Disclosures can be viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum=M13-1522.
Corresponding Author: Maarten J.G. Leening, MD, MSc, Department of Epidemiology, Erasmus MC - University Medical Center Rotterdam, PO Box 2040, 3000 CA Rotterdam, the Netherlands; e-mail, m.
Current Author Addresses: Drs. Leening and Witteman: Department of Epidemiology, Erasmus MC - University Medical Center Rotterdam, Dr. Molenwaterplein 50, 3015 GE Rotterdam, the Netherlands.
Ms. Vedder and Dr. Steyerberg: Department of Public Health, Erasmus MC - University Medical Center Rotterdam, Dr. Molenwaterplein 50, 3015 GE Rotterdam, the Netherlands.
Dr. Pencina: Department of Biostatistics and Bioinformatics, Duke Clinical Research Institute, Duke University, 2400 Pratt Street, Durham, NC 27715.
Author Contributions: Conception and design: M.J.G. Leening, E.W. Steyerberg.
Analysis and interpretation of the data: M.J.G. Leening, M.M. Vedder, E.W. Steyerberg.
Drafting of the article: M.J.G. Leening.
Critical revision of the article for important intellectual content: M.J.G. Leening, M.M. Vedder, J.C.M. Witteman, M.J. Pencina, E.W. Steyerberg.
Final approval of the article: M.J.G. Leening, M.M. Vedder, J.C.M. Witteman, M.J. Pencina, E.W. Steyerberg.
Statistical expertise: M.J.G. Leening, M.J. Pencina, E.W. Steyerberg.
Obtaining of funding: E.W. Steyerberg.
Administrative, technical, or logistic support: M.J.G. Leening.
Collection and assembly of data: M.J.G. Leening, M.M. Vedder.
Submit a Comment
Contributors must reveal any conflict of interest. Comments are moderated. Please see our information for authorsregarding comments on an Annals publication.
*All comments submitted after October 1, 2021 and selected for publication will be published online only.