Net Reclassification Improvement: Computation, Interpretation, and Controversies: A Literature Review and Clinician's Guide
Submit a Comment
Contributors must reveal any conflict of interest. Comments are moderated. Please see our information for authorsregarding comments on an Annals publication.
Abstract
The net reclassification improvement (NRI) is an increasingly popular measure for evaluating improvements in risk predictions. This article details a review of 67 publications in high-impact general clinical journals that considered the NRI. Incomplete reporting of NRI methods, incorrect calculation, and common misinterpretations were found. To aid improved applications of the NRI, the article elaborates on several aspects of the computation and interpretation in various settings. Limitations and controversies are discussed, including the effect of miscalibration of prediction models, the use of the continuous NRI and “clinical NRI,” and the relation with decision analytic measures. A systematic approach toward presenting NRI analysis is proposed: Detail and motivate the methods used for computation of the NRI, use clinically meaningful risk cutoffs for the category-based NRI, report both NRI components, address issues of calibration, and do not interpret the overall NRI as a percentage of the study population reclassified. Promising NRI findings need to be followed with decision analytic or formal cost-effectiveness evaluations.
Get full access to this article
View all available purchase options and get full access to this article.
References
1.
Kannel WB, Dawber TR, Kagan A, Revotskie N, Stokes J 3rd. Factors of risk in the development of coronary heart disease—six year follow-up experience. The Framingham Study. Ann Intern Med. 1961;55:33-50. [PMID: 13751193]
2.
Expert Panel on Detection, Evaluation. Treatment of High Blood Cholesterol in Adults. Executive Summary of The Third Report of The National Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation, And Treatment of High Blood Cholesterol In Adults (Adult Treatment Panel III). JAMA. 2001;285:2486-97. [PMID: 11368702]
3.
Perk J, De Backer G, Gohlke H, Graham I, Reiner Z, Verschuren M, et al; European Association for Cardiovascular Prevention & Rehabilitation (EACPR). European Guidelines on cardiovascular disease prevention in clinical practice (version 2012). The Fifth Joint Task Force of the European Society of Cardiology and Other Societies on Cardiovascular Disease Prevention in Clinical Practice (constituted by representatives of nine societies and by invited experts). Eur Heart J. 2012;33:1635-701. [PMID: 22555213]
4.
Hamm CW, Bassand JP, Agewall S, Bax J, Boersma E, Bueno H, et al; ESC Committee for Practice Guidelines. ESC Guidelines for the management of acute coronary syndromes in patients presenting without persistent ST-segment elevation: The Task Force for the management of acute coronary syndromes (ACS) in patients presenting without persistent ST-segment elevation of the European Society of Cardiology (ESC). Eur Heart J. 2011;32:2999-3054. [PMID: 21873419]
5.
Visvanathan K, Chlebowski RT, Hurley P, Col NF, Ropka M, Collyar D, et al; American Society of Clinical Oncology. American Society of Clinical Oncology clinical practice guideline update on the use of pharmacologic interventions including tamoxifen, raloxifene, and aromatase inhibition for breast cancer risk reduction. J Clin Oncol. 2009;27:3235-58. [PMID: 19470930]
6.
Munshi NC, Anderson KC, Bergsagel PL, Shaughnessy J, Palumbo A, Durie B, et al; International Myeloma Workshop Consensus Panel 2. Consensus recommendations for risk stratification in multiple myeloma: report of the International Myeloma Workshop Consensus Panel 2. Blood. 2011;117:4696-700. [PMID: 21292777]
7.
Worth LJ, Lingaratnam S, Taylor A, Hayward AM, Morrissey S, Cooney J, et al; Australian Consensus Guidelines 2011 Steering Committee. Use of risk stratification to guide ambulatory management of neutropenic fever. Australian Consensus Guidelines 2011 Steering Committee. Intern Med J. 2011;41:82-9. [PMID: 21272172]
8.
Bates SM, Jaeschke R, Stevens SM, Goodacre S, Wells PS, Stevenson MD, et al; American College of Chest Physicians. Diagnosis of DVT: Antithrombotic Therapy and Prevention of Thrombosis, 9th ed: American College of Chest Physicians Evidence-Based Clinical Practice Guidelines. Chest. 2012;141:351S-418S. [PMID: 22315267]
9.
Pepe MS, Janes H, Longton G, Leisenring W, Newcomb P. Limitations of the odds ratio in gauging the performance of a diagnostic, prognostic, or screening marker. Am J Epidemiol. 2004;159:882-90. [PMID: 15105181]
10.
Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation. 2007;115:928-35. [PMID: 17309939]
11.
Pencina MJ, D'Agostino RB, Pencina KM, Janssens ACJW, Greenland P. Interpreting incremental value of markers added to risk prediction models. Am J Epidemiol. 2012;176:473-81. [PMID: 22875755]
12.
Austin PC, Steyerberg EW. Predictive accuracy of risk factors and markers: a simulation study of the effect of novel markers on different performance measures for logistic regression models. Stat Med. 2013;32:661-72. [PMID: 22961910]
13.
Hlatky MA, Greenland P, Arnett DK, Ballantyne CM, Criqui MH, Elkind MSV, et al; American Heart Association Expert Panel on Subclinical Atherosclerotic Diseases and Emerging Risk Factors and the Stroke Council. Criteria for evaluation of novel markers of cardiovascular risk: a scientific statement from the American Heart Association. Circulation. 2009;119:2408-16. [PMID: 19364974]
14.
Cook NR, Buring JE, Ridker PM. The effect of including C-reactive protein in cardiovascular risk prediction models for women. Ann Intern Med. 2006;145:21-9. [PMID: 16818925]
15.
Pencina MJ, D'Agostino RB Sr, D'Agostino RB Jr, Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med. 2008;27:157-72. [PMID: 17569110]
16.
Tzoulaki I, Liberopoulos G, Ioannidis JPA. Use of reclassification for assessment of improved prediction: an empirical evaluation. Int J Epidemiol. 2011;40:1094-105. [PMID: 21325392]
17.
Leening MJG, Steyerberg EW. Fibrosis and mortality in patients with dilated cardiomyopathy [Letter]. JAMA. 2013;309:2547-8. [PMID: 23800926]
18.
Lilford RJ, Richardson A, Stevens A, Fitzpatrick R, Edwards S, Rock F, et al. Issues in methodological research: perspectives from researchers and commissioners. Health Technol Assess. 2001;5:1-57. [PMID: 11368832]
19.
Pencina MJ, D'Agostino RB, Vasan RS. Statistical methods for assessment of added usefulness of new biomarkers. Clin Chem Lab Med. 2010;48:1703-11. [PMID: 20716010]
20.
Pencina MJ, D'Agostino RB Sr, Steyerberg EW. Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Stat Med. 2011;30:11-21. [PMID: 21204120]
21.
Pencina MJ, D'Agostino RB Sr, Demler OV. Novel metrics for evaluating improvement in discrimination: net reclassification and integrated discrimination improvement for normal variables and nested models. Stat Med. 2012;31:101-13. [PMID: 22147389]
22.
Cook NR, Ridker PM. Advances in measuring the effect of individual predictors of cardiovascular risk: the role of reclassification measures. Ann Intern Med. 2009;150:795-802. [PMID: 19487714]
23.
Adabag AS, Therneau TM, Gersh BJ, Weston SA, Roger VL. Sudden death after myocardial infarction. JAMA. 2008;300:2022-9. [PMID: 18984889]
24.
Auer R, Bauer DC, Marques-Vidal P, Butler J, Min LJ, Cornuz J, et al; Health ABC Study. Association of major and minor ECG abnormalities with coronary heart disease events. JAMA. 2012;307:1497-505. [PMID: 22496264]
25.
Breteler MMB. Mapping out biomarkers for Alzheimer disease [Editorial]. JAMA. 2011;305:304-5. [PMID: 21245188]
26.
Buckley DI, Fu R, Freeman M, Rogers K, Helfand M. C-reactive protein as a risk factor for coronary heart disease: a systematic review and meta-analyses for the U.S. Preventive Services Task Force. Ann Intern Med. 2009;151:483-95. [PMID: 19805771]
27.
Chou R, Arora B, Dana T, Fu R, Walker M, Humphrey L. Screening asymptomatic adults with resting or exercise electrocardiography: a review of the evidence for the U.S. Preventive Services Task Force. Ann Intern Med. 2011;155:375-85. [PMID: 21930855]
28.
Cook NR. Biomarkers for prediction of cardiovascular events [Letter]. JAMA. 2009;302:2089. [PMID: 19920231]
29.
Cornelis MC, Qi L, Zhang C, Kraft P, Manson J, Cai T, et al. Joint effects of common genetic variants on the risk for type 2 diabetes in U.S. men and women of European ancestry. Ann Intern Med. 2009;150:541-50. [PMID: 19380854]
30.
de Boer IH, Levin G, Robinson-Cohen C, Biggs ML, Hoofnagle AN, Siscovick DS, et al. Serum 25-hydroxyvitamin D concentration and risk for major clinical disease events in a community-based population of older adults: a cohort study. Ann Intern Med. 2012;156:627-34. [PMID: 22547472]
31.
de Lemos JA, Drazner MH, Omland T, Ayers CR, Khera A, Rohatgi A, et al. Association of troponin T detected with a highly sensitive assay and cardiac structure and mortality risk in the general population. JAMA. 2010;304:2503-12. [PMID: 21139111]
32.
deFilippi CR, de Lemos JA, Christenson RH, Gottdiener JS, Kop WJ, Zhan M, et al. Association of serial measures of cardiac troponin T using a sensitive assay with incident heart failure and cardiovascular mortality in older adults. JAMA. 2010;304:2494-502. [PMID: 21078811]
33.
den Ruijter HM, Peters SAE, Anderson TJ, Britton AR, Dekker JM, Eijkemans MJ, et al. Common carotid intima-media thickness measurements in cardiovascular risk prediction: a meta-analysis. JAMA. 2012;308:796-803. [PMID: 22910757]
34.
Devereaux PJ, Chan MTV, Alonso-Coello P, Walsh M, Berwanger O, Villar JC, et al; Vascular Events In Noncardiac Surgery Patients Cohort Evaluation (VISION) Study Investigators. Association between postoperative troponin levels and 30-day mortality among patients undergoing noncardiac surgery. JAMA. 2012;307:2295-304. [PMID: 22706835]
35.
Di Angelantonio E, Gao P, Pennells L, Kaptoge S, Caslake M, Thompson A, et al; Emerging Risk Factors Collaboration. Lipid-related markers and cardiovascular disease prediction. JAMA. 2012;307:2499-506. [PMID: 22797450]
36.
Eddy DM, Adler J, Patterson B, Lucas D, Smith KA, Morris M. Individualized guidelines: the potential for increasing quality and reducing costs. Ann Intern Med. 2011;154:627-34. [PMID: 21536939]
37.
Farooq V, van Klaveren D, Steyerberg EW, Meliga E, Vergouwe Y, Chieffo A, et al. Anatomical and clinical characteristics to guide decision making between coronary artery bypass surgery and percutaneous coronary intervention for individual patients: development and validation of SYNTAX Score II. Lancet. 2013;381:639-50. [PMID: 23439103]
38.
Fonarow GC, Pan W, Saver JL, Smith EE, Reeves MJ, Broderick JP, et al. Comparison of 30-day mortality models for profiling hospital performance in acute ischemic stroke with vs without adjustment for stroke severity. JAMA. 2012;308:257-64. [PMID: 22797643]
39.
Gulati A, Jabbour A, Ismail TF, Guha K, Khwaja J, Raza S, et al. Association of fibrosis with mortality and sudden cardiac death in patients with nonischemic dilated cardiomyopathy. JAMA. 2013;309:896-908. [PMID: 23462786]
40.
Helfand M, Buckley DI, Freeman M, Fu R, Rogers K, Fleming C, et al. Emerging risk factors for coronary heart disease: a summary of systematic reviews conducted for the U.S. Preventive Services Task Force. Ann Intern Med. 2009;151:496-507. [PMID: 19805772]
41.
Hingorani AD, Psaty BM. Primary prevention of cardiovascular disease: time to get more or less personal? JAMA. 2009;302:2144-5. [PMID: 19920239]
42.
Hlatky MA. Framework for evaluating novel risk markers [Editorial]. Ann Intern Med. 2012;156:468-9. [PMID: 22431679]
43.
Janes H, Pepe MS, Gu W. Assessing the value of risk predictions by using risk stratification tables. Ann Intern Med. 2008;149:751-60. [PMID: 19017593]
44.
Janssens ACJW, Ioannidis JPA, van Duijn CM, Little J, Khoury MJ; GRIPS Group. Strengthening the reporting of genetic risk prediction studies: The GRIPS Statement. Ann Intern Med. 2011;154:421-5. [PMID: 21403077]
45.
Kaptoge S, Di Angelantonio E, Lowe G, Pepys MB, Thompson SG, Collins R, et al; Emerging Risk Factors Collaboration. C-reactive protein concentration and risk of coronary heart disease, stroke, and mortality: an individual participant meta-analysis. Lancet. 2010;375:132-40. [PMID: 20031199]
46.
Kaptoge S, Di Angelantonio E, Pennells L, Wood AM, White IR, Gao P, et al; Emerging Risk Factors Collaboration. C-reactive protein, fibrinogen, and cardiovascular disease prediction. N Engl J Med. 2012;367:1310-20. [PMID: 23034020]
47.
Kathiresan S, Melander O, Anevski D, Guiducci C, Burtt NP, Roos C, et al. Polymorphisms associated with cholesterol and risk of cardiovascular events. N Engl J Med. 2008;358:1240-9. [PMID: 18354102]
48.
Kavousi M, Elias-Smale S, Rutten JHW, Leening MJG, Vliegenthart R, Verwoert GC, et al. Evaluation of newer risk markers for coronary heart disease risk classification: a cohort study. Ann Intern Med. 2012;156:438-44. [PMID: 22431676]
49.
Keller T, Zeller T, Ojeda F, Tzikas S, Lillpopp L, Sinning C, et al. Serial changes in highly sensitive troponin I assay and early diagnosis of myocardial infarction. JAMA. 2011;306:2684-93. [PMID: 22203537]
50.
Kengne AP, Echouffo-Tcheugui JB, Sobngwi E. Coronary artery calcium for guiding statin treatment [Letter]. Lancet. 2012;379:312. [PMID: 22284653]
51.
Khera AV, Cuchel M, de la Llera-Moya M, Rodrigues A, Burke MF, Jafri K, et al. Cholesterol efflux capacity, high-density lipoprotein function, and atherosclerosis. N Engl J Med. 2011;364:127-35. [PMID: 21226578]
52.
Kim WR, Biggins SW, Kremers WK, Wiesner RH, Kamath PS, Benson JT, et al. Hyponatremia and mortality among patients on the liver-transplant waiting list. N Engl J Med. 2008;359:1018-26. [PMID: 18768945]
53.
Kivimäki M, Batty GD, Hamer M, Ferrie JE, Vahtera J, Virtanen M, et al. Using additional information on working hours to predict coronary heart disease: a cohort study. Ann Intern Med. 2011;154:457-63. [PMID: 21464347]
54.
Koller MT, Leening MJG, Wolbers M, Steyerberg EW, Hunink MGM, Schoop R, et al. Development and validation of a coronary risk prediction model for older U.S. and European persons in the Cardiovascular Health Study and the Rotterdam Study. Ann Intern Med. 2012;157:389-97. [PMID: 22986376]
55.
Lubitz SA, Yin X, Fontes JD, Magnani JW, Rienstra M, Pai M, et al. Association between familial atrial fibrillation and risk of new-onset atrial fibrillation. JAMA. 2010;304:2263-9. [PMID: 21076174]
56.
Lyssenko V, Jonsson A, Almgren P, Pulizzi N, Isomaa B, Tuomi T, et al. Clinical risk factors, DNA variants, and the development of type 2 diabetes. N Engl J Med. 2008;359:2220-32. [PMID: 19020324]
57.
Manolio TA. Genomewide association studies and assessment of the risk of disease. N Engl J Med. 2010;363:166-76. [PMID: 20647212]
58.
Martínez ME, Thompson P, Messer K, Ashbeck EL, Lieberman DA, Baron JA, et al. One-year risk for advanced colorectal neoplasia: U.S. versus U.K. risk-stratification guidelines. Ann Intern Med. 2012;157:856-64. [PMID: 23247939]
59.
Matsushita K, Mahmoodi BK, Woodward M, Emberson JR, Jafar TH, Jee SH, et al; Chronic Kidney Disease Prognosis Consortium. Comparison of risk prediction using the CKD-EPI equation and the MDRD study equation for estimated glomerular filtration rate. JAMA. 2012;307:1941-51. [PMID: 22570462]
60.
McEvoy JW. Coronary artery calcium score and cardiovascular event prediction [Letter]. JAMA. 2010;304:741-2. [PMID: 20716732]
61.
Meigs JB, Shrader P, Sullivan LM, McAteer JB, Fox CS, Dupuis J, et al. Genotype score in addition to common risk factors for prediction of type 2 diabetes. N Engl J Med. 2008;359:2208-19. [PMID: 19020323]
62.
Melander O, Newton-Cheh C, Almgren P, Hedblad B, Berglund G, Engström G, et al. Novel and conventional biomarkers for prediction of incident cardiovascular events in the community. JAMA. 2009;302:49-57. [PMID: 19567439]
63.
Melander O, Newton-Cheh C, Wang TJ. Biomarkers for prediction of cardiovascular events—reply. JAMA. 2009;302:2090.
64.
Omland T, de Lemos JA, Sabatine MS, Christophi CA, Rice MM, Jablonski KA, et al; Prevention of Events with Angiotensin Converting Enzyme Inhibition (PEACE) Trial Investigators. A sensitive cardiac troponin T assay in stable coronary artery disease. N Engl J Med. 2009;361:2538-47. [PMID: 19940289]
65.
Palomaki GE, Melillo S, Bradley LA. Association between 9p21 genomic markers and heart disease: a meta-analysis. JAMA. 2010;303:648-56. [PMID: 20159873]
66.
Paynter NP, Chasman DI, Buring JE, Shiffman D, Cook NR, Ridker PM. Cardiovascular disease risk prediction with and without knowledge of genetic variation at chromosome 9p21.3. Ann Intern Med. 2009;150:65-72. [PMID: 19153409]
67.
Paynter NP, Chasman DI, Paré G, Buring JE, Cook NR, Miletich JP, et al. Association between a literature-based genetic risk score and cardiovascular events in women. JAMA. 2010;303:631-7. [PMID: 20159871]
68.
Peralta CA, Shlipak MG, Judd S, Cushman M, McClellan W, Zakai NA, et al. Detection of chronic kidney disease with creatinine, cystatin C, and urine albumin-to-creatinine ratio and association with progression to end-stage renal disease and mortality. JAMA. 2011;305:1545-52. [PMID: 21482744]
69.
Pischon T, Boeing H, Hoffmann K, Bergmann M, Schulze MB, Overvad K, et al. General and abdominal adiposity and risk of death in Europe. N Engl J Med. 2008;359:2105-20. [PMID: 19005195]
70.
Pletcher MJ, Tice JA, Pignone M. Modeling cardiovascular disease prevention [Letter]. JAMA. 2010;303:835. [PMID: 20197528]
71.
Polak JF, Pencina MJ, Pencina KM, O'Donnell CJ, Wolf PA, D'Agostino RB Sr. Carotid-wall intima-media thickness and cardiovascular events. N Engl J Med. 2011;365:213-21. [PMID: 21774709]
72.
Polonsky TS, McClelland RL, Jorgensen NW, Bild DE, Burke GL, Guerci AD, et al. Coronary artery calcium score and risk classification for coronary heart disease prediction. JAMA. 2010;303:1610-6. [PMID: 20424251]
73.
Ripatti S, Tikkanen E, Orho-Melander M, Havulinna AS, Silander K, Sharma A, et al. A multilocus genetic risk score for coronary heart disease: case-control and prospective cohort analyses. Lancet. 2010;376:1393-400. [PMID: 20971364]
74.
Rosenberg S, Elashoff MR, Beineke P, Daniels SE, Wingrove JA, Tingley WG, et al; PREDICT (Personalized Risk Evaluation and Diagnosis in the Coronary Tree) Investigators. Multicenter validation of the diagnostic accuracy of a blood-based gene expression test for assessing obstructive coronary artery disease in nondiabetic patients. Ann Intern Med. 2010;153:425-34. [PMID: 20921541]
75.
Schelbert EB, Cao JJ, Sigurdsson S, Aspelund T, Kellman P, Aletras AH, et al. Prevalence and prognosis of unrecognized myocardial infarction determined by cardiac magnetic resonance in older adults. JAMA. 2012;308:890-6. [PMID: 22948699]
76.
Schnabel RB, Sullivan LM, Levy D, Pencina MJ, Massaro JM, D'Agostino RB Sr, et al. Development of a risk score for atrial fibrillation (Framingham Heart Study): a community-based cohort study. Lancet. 2009;373:739-45. [PMID: 19249635]
77.
Selvin E, Steffes MW, Zhu H, Matsushita K, Wagenknecht L, Pankow J, et al. Glycated hemoglobin, diabetes, and cardiovascular risk in nondiabetic adults. N Engl J Med. 2010;362:800-11. [PMID: 20200384]
78.
Steyerberg EW, Pencina MJ. Reclassification calculations for persons with incomplete follow-up [Letter]. Ann Intern Med. 2010;152:195-6. [PMID: 20124243]
79.
Tammemägi MC, Katki HA, Hocking WG, Church TR, Caporaso N, Kvale PA, et al. Selection criteria for lung-cancer screening. N Engl J Med. 2013;368:728-36. [PMID: 23425165]
80.
Tangri N, Stevens LA, Griffith J, Tighiouart H, Djurdjev O, Naimark D, et al. A predictive model for progression of chronic kidney disease to kidney failure. JAMA. 2011;305:1553-9. [PMID: 21482743]
81.
Tzoulaki I, Liberopoulos G, Ioannidis JPA. Assessment of claims of improved prediction beyond the Framingham risk score. JAMA. 2009;302:2345-52. [PMID: 19952321]
82.
Wacholder S, Hartge P, Prentice R, Garcia-Closas M, Feigelson HS, Diver WR, et al. Performance of common genetic variants in breast-cancer risk models. N Engl J Med. 2010;362:986-93. [PMID: 20237344]
83.
Wilson PWF. Challenges to improve coronary heart disease risk assessment [Editorial]. JAMA. 2009;302:2369-70. [PMID: 19952326]
84.
Wormser D, Kaptoge S, Di Angelantonio E, Wood AM, Pennells L, Thompson A, et al; Emerging Risk Factors Collaboration. Separate and combined associations of body-mass index and abdominal adiposity with cardiovascular disease: collaborative analysis of 58 prospective studies. Lancet. 2011;377:1085-95. [PMID: 21397319]
85.
Wormser D, Di Angelantonio E, Sattar N, Collins R, Thompson S, Danesh J, et al. Body-mass index, abdominal adiposity, and cardiovascular risk - reply. Lancet. 2011;378:228.
86.
Yeboah J, McClelland RL, Polonsky TS, Burke GL, Sibley CT, O'Leary D, et al. Comparison of novel risk markers for improvement in cardiovascular risk assessment in intermediate-risk individuals. JAMA. 2012;308:788-95. [PMID: 22910756]
87.
Zethelius B, Berglund L, Sundström J, Ingelsson E, Basu S, Larsson A, et al. Use of multiple biomarkers to improve the prediction of death from cardiovascular causes. N Engl J Med. 2008;358:2107-16. [PMID: 18480203]
88.
Zoungas S, Patel A, Chalmers J, de Galan BE, Li Q, Billot L, et al; ADVANCE Collaborative Group. Severe hypoglycemia and risks of vascular events and death. N Engl J Med. 2010;363:1410-8. [PMID: 20925543]
89.
Greenland P, Alpert JS, Beller GA, Benjamin EJ, Budoff MJ, Fayad ZA, et al; American College of Cardiology Foundation. American Heart Association Task Force on Practice Guidelines. 2010 ACCF/AHA guideline for assessment of cardiovascular risk in asymptomatic adults: a report of the American College of Cardiology Foundation/American Heart Association Task Force on Practice Guidelines. Circulation. 2010;122:584-636. [PMID: 21098428]
90.
Mosca L, Benjamin EJ, Berra K, Bezanson JL, Dolor RJ, Lloyd-Jones DM, et al. Effectiveness-based guidelines for the prevention of cardiovascular disease in women—2011 update: a guideline from the American Heart Association. Circulation. 2011;123:1243-62. [PMID: 21325087]
91.
Cai T, Tian L, Lloyd-Jones DM. Comparing costs associated with risk stratification rules for t-year survival. Biostatistics. 2011;12:597-609. [PMID: 21415016]
92.
Leening MJG, Cook NR. Net reclassification improvement: a link between statistics and clinical practice. Eur J Epidemiol. 2013;28:21-3. [PMID: 23291877]
93.
Pepe MS. Problems with risk reclassification methods for evaluating prediction models. Am J Epidemiol. 2011;173:1327-35. [PMID: 21555714]
94.
Mihaescu R, van Zitteren M, van Hoek M, Sijbrands EJ, Uitterlinden AG, Witteman JCM, et al. Improvement of risk prediction by genomic profiling: reclassification measures versus the area under the receiver operating characteristic curve. Am J Epidemiol. 2010;172:353-61. [PMID: 20562194]
95.
Mühlenbruch K, Heraclides A, Steyerberg EW, Joost HG, Boeing H, Schulze MB. Assessing improvement in disease prediction using net reclassification improvement: impact of risk cut-offs and number of risk categories. Eur J Epidemiol. 2013;28:25-33. [PMID: 23179629]
96.
Pepe MS, Janes H. Commentary: Reporting standards are needed for evaluations of risk reclassification. Int J Epidemiol. 2011;40:1106-8. [PMID: 21571811]
97.
McGeechan K, Macaskill P, Irwig L, Liew G, Wong TY. Assessing new biomarkers and predictive models for use in clinical practice: a clinician's guide. Arch Intern Med. 2008;168:2304-10. [PMID: 19029492]
98.
Greenland S. The need for reorientation toward cost-effective prediction: comments on ‘Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond’ by M. J. Pencina et al., Statistics in Medicine (DOI: 10.1002/sim.2929). Stat Med. 2008;27:199-206. [PMID: 17729377] doi: 10.1002/sim.2929)
99.
Cook NR, Paynter NP. Performance of reclassification statistics in comparing risk prediction models. Biom J. 2011;53:237-58. [PMID: 21294152]
100.
National Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation. Treatment of High Blood Cholesterol in Adults (Adult Treatment Panel III). Third Report of the National Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults (Adult Treatment Panel III) final report. Circulation. 2002;106:3143-421. [PMID: 12485966]
101.
Cook NR, Paynter NP, Eaton CB, Manson JE, Martin LW, Robinson JG, et al. Comparison of the Framingham and Reynolds Risk scores for global cardiovascular risk prediction in the multiethnic Women's Health Initiative. Circulation. 2012;125:1748-56, S1-11. [PMID: 22399535]
102.
Pencina MJ, D'Agostino RB Sr, Larson MG, Massaro JM, Vasan RS. Predicting the 30-year risk of cardiovascular disease: the Framingham Heart Study. Circulation. 2009;119:3078-84. [PMID: 19506114]
103.
Takahara M, Katakami N, Kaneto H, Shimomura I. Risk categorization for calculating net reclassification improvement [Letter]. Eur J Epidemiol. 2013;28:607-9. [PMID: 23839540]
104.
Ganna A, Reilly M, de Faire U, Pedersen N, Magnusson P, Ingelsson E. Risk prediction measures for case-cohort and nested case-control designs: an application to cardiovascular disease. Am J Epidemiol. 2012;175:715-24. [PMID: 22396388]
105.
Pepe MS, Fan J, Seymour CW, Li C, Huang Y, Feng Z. Biases introduced by choosing controls to match risk factors of cases in biomarker research. Clin Chem. 2012;58:1242-51. [PMID: 22730452]
106.
Rothman KJ, Greenland S, Lash TL. Modern Epidemiology. 3rd ed. Philadelphia: Lippincott Williams & Wilkins; 2008.
107.
Kavousi M, Leening MJG, Witteman JCM. Markers for prediction of cardiovascular disease risk [Letter]. JAMA. 2012;308:2561. [PMID: 23268505]
108.
Pencina MJ, D'Agostino RB Sr, D'Agostino RB Jr, Vasan RS. Comments on ‘Integrated discrimination and net reclassification improvements—Practical advice’. Stat Med. 2008;27:207-12. [PMID: 17569110]
109.
Pepe MS, Feng Z, Gu JW. Comments on ‘Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond’ by M. J. Pencina et al., Statistics in Medicine (DOI: 10.1002/sim.2929). Stat Med. 2008;27:173-81. [PMID: 17671958] doi: 10.1002/sim.2929)
110.
Hilden J, Gerds TA. A note on the evaluation of novel biomarkers: do not rely on integrated discrimination improvement and net reclassification index. Stat Med. 2013;:. [PMID: 23553436]
111.
Brindle P, Emberson J, Lampe F, Walker M, Whincup P, Fahey T, et al. Predictive accuracy of the Framingham coronary risk score in British men: prospective cohort study. BMJ. 2003;327:1267. [PMID: 14644971]
112.
Hense HW, Schulte H, Löwel H, Assmann G, Keil U. Framingham risk function overestimates risk of coronary heart disease in men and women from Germany—results from the MONICA Augsburg and the PROCAM cohorts. Eur Heart J. 2003;24:937-45. [PMID: 12714025]
113.
Koller MT, Steyerberg EW, Wolbers M, Stijnen T, Bucher HC, Hunink MGM, et al. Validity of the Framingham point scores in the elderly: results from the Rotterdam study. Am Heart J. 2007;154:87-93. [PMID: 17584559]
114.
Merry AH, Boer JM, Schouten LJ, Ambergen T, Steyerberg EW, Feskens EJM, et al. Risk prediction of incident coronary heart disease in The Netherlands: re-estimation and improvement of the SCORE risk function. Eur J Prev Cardiol. 2012;19:840-8. [PMID: 21551214]
115.
Siontis GCM, Tzoulaki I, Siontis KC, Ioannidis JPA. Comparisons of established risk prediction models for cardiovascular disease: systematic review. BMJ. 2012;344:3318. [PMID: 22628003]
116.
Steyerberg EW. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. New York: Springer; 2009.
117.
Kramer AA, Zimmerman JE. Assessing the calibration of mortality benchmarks in critical care: The Hosmer-Lemeshow test revisited. Crit Care Med. 2007;35:2052-6. [PMID: 17568333]
118.
Cox DR. Two further applications of a model for binary regression. Biometrika. 1958;45:562-5.
119.
Cook NR, Paynter NP. Comments on ‘Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers’ by M. J. Pencina, R. B. D'Agostino, Sr. and E. W. Steyerberg [Letter]. Stat Med. 2012;31:93-5. [PMID: 21344474]
120.
Vergouwe Y, Moons KGM, Steyerberg EW. External validity of risk models: Use of benchmark values to disentangle a case-mix effect from incorrect coefficients. Am J Epidemiol. 2010;172:971-80. [PMID: 20807737]
121.
Cook NR. Comments on ‘Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond’ by M. J. Pencina et al., Statistics in Medicine (DOI: 10.1002/sim.2929). Stat Med. 2008;27:191-5. [PMID: 17671959] doi: 10.1002/sim.2929)
122.
Paynter NP, Cook NR. A bias-corrected net reclassification improvement for clinical subgroups. Med Decis Making. 2013;33:154-62. [PMID: 23042826]
123.
Vickers AJ, Elkin EB, Steyerberg E. Net reclassification improvement and decision theory [Letter]. Stat Med. 2009;28:525-6. [PMID: 17907248]
124.
Van Calster B, Vickers AJ, Pencina MJ, Baker SG, Timmerman D, Steyerberg EW. Evaluation of markers and risk prediction models: overview of relationships between NRI and decision-analytic measures. Med Decis Making. 2013;33:490-501. [PMID: 23313931]
125.
Localio AR, Goodman S. Beyond the usual prediction accuracy metrics: reporting results for clinical decision making [Editorial]. Ann Intern Med. 2012;157:294-5. [PMID: 22910942]
126.
Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21:128-38. [PMID: 20010215]
127.
Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making. 2006;26:565-74. [PMID: 17099194]
128.
Vickers AJ, Cronin AM, Begg CB. One statistical test is sufficient for assessing new predictive markers. BMC Med Res Methodol. 2011;11:13. [PMID: 21276237]
129.
Pepe MS, Kerr KF, Longton G, Wang Z. Testing for improvement in prediction model performance. Stat Med. 2013;32:1467-82. [PMID: 23296397]
130.
McGeechan K, Liew G, Macaskill P, Irwig L, Klein R, Sharrett AR, et al. Risk prediction of coronary heart disease based on retinal vascular caliber (from the Atherosclerosis Risk In Communities [ARIC] Study). Am J Cardiol. 2008;102:58-63. [PMID: 18572036]
Information & Authors
Information
Published In
Annals of Internal Medicine
Volume 160 • Number 2 • 21 January 2014
Pages: 122 - 131
History
Published online: 21 January 2014
Published in issue: 21 January 2014
Keywords
Copyright
© 2014 American College of Physicians.
Authors
Metrics & Citations
Metrics
Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. For an editable text file, please select Medlars format which will download as a .txt file. Simply select your manager software from the list below and click Download.
For more information or tips please see 'Downloading to a citation manager' in the Help menu.
Download article citation data for:
Net Reclassification Improvement: Computation, Interpretation, and Controversies: A Literature Review and Clinician's Guide. Ann Intern Med.2014;160:122-131. [Epub 21 January 2014]. doi:10.7326/M13-1522
Net Reclassification Improvement: Computation, Interpretation, and Controversies: A Literature Review and Clinician's Guide. Ann Intern Med.2014;160:122-131. [Epub 21 January 2014]. doi:10.7326/M13-1522
View More
Login Options:
Purchase
You will be redirected to acponline.org to sign-in to Annals to complete your purchase.
Access to EPUBs and PDFs for FREE Annals content requires users to be registered and logged in. A subscription is not required. You can create a free account below or from the following link. You will be redirected to acponline.org to create an account that will provide access to Annals. If you are accessing the Free Annals content via your institution's access, registration is not required.
Create your Free Account
You will be redirected to acponline.org to create an account that will provide access to Annals.
Comments
0 Comments