Free access
Research and Reporting Methods
1 January 2019

PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model StudiesFREE

Publication: Annals of Internal Medicine
Volume 170, Number 1

Abstract

Clinical prediction models combine multiple predictors to estimate risk for the presence of a particular condition (diagnostic models) or the occurrence of a certain event in the future (prognostic models).
PROBAST (Prediction model Risk Of Bias ASsessment Tool), a tool for assessing the risk of bias (ROB) and applicability of diagnostic and prognostic prediction model studies, was developed by a steering group that considered existing ROB tools and reporting guidelines. The tool was informed by a Delphi procedure involving 38 experts and was refined through piloting.
PROBAST is organized into the following 4 domains: participants, predictors, outcome, and analysis. These domains contain a total of 20 signaling questions to facilitate structured judgment of ROB, which was defined to occur when shortcomings in study design, conduct, or analysis lead to systematically distorted estimates of model predictive performance. PROBAST enables a focused and transparent approach to assessing the ROB and applicability of studies that develop, validate, or update prediction models for individualized predictions.
Although PROBAST was designed for systematic reviews, it can be used more generally in critical appraisal of prediction model studies. Potential users include organizations supporting decision making, researchers and clinicians who are interested in evidence-based medicine or involved in guideline development, journal editors, and manuscript reviewers.
Prediction relates to estimating the probability of something currently unknown. In the context of medical research, prediction typically concerns either diagnosis (probability of a certain condition being present but not yet detected) or prognosis (probability of an outcome developing in the future) (1–3). Prognosis applies not only to sick persons or those with an established diagnosis but also to, for example, pregnant women at risk for diabetes (4). Prediction research includes predictor finding studies, prediction model studies (development, validation, and extending or updating), and prediction model impact studies (1).
Predictor finding studies (also known as risk factor or prognostic factor studies) aim to identify which predictors (such as age, disease stage, or biomarkers) independently contribute to the prediction of a diagnostic or prognostic outcome (1, 5).
Prediction model studies aim to develop, validate, or update (for example, extend) a multivariable prediction model. A prediction model uses multiple predictors in combination to estimate probabilities to inform and often guide individual care (2, 6, 7). These models can predict an individual's probability of either currently having a particular outcome or disease (diagnostic prediction model) or having a particular outcome in the future (prognostic prediction model). Both types of model are widely used in various medical domains and settings (8–10), as evidenced by the large number of models developed in cancer (11, 12), neurology (13, 14), and cardiovascular disease (15). Prediction models are sometimes described as risk prediction models, predictive models, prediction indices or rules, or risk scores (2, 7). An example is QRISK2 for predicting cardiovascular risk (16).
Prediction model impact studies evaluate the effect of using a model to guide patient care compared with not using such a model. They use a comparative design, such as a randomized trial, to study the model's effect on clinical decision making, patient outcomes, or costs of care (1).
Systematic reviews have a key role in evidence-based medicine and the development of clinical guidelines (17–19). They are considered to provide the most reliable form of evidence for the effects of an intervention or diagnostic test (20, 21). Systematic reviews of prediction models are a relatively new and evolving area but are increasingly undertaken to systematically identify, appraise, and summarize evidence on the performance of prediction models (1, 6, 22).
Assessing the quality of included studies is a crucial step in any systematic review (20, 21). The QUIPS (Quality In Prognosis Studies) tool has been developed to assess risk of bias (ROB) in predictor finding (prognostic factor) studies (23). Researchers can use the revised Cochrane ROB Tool (ROB 2.0) (24) to investigate the methodological quality of prediction model impact studies that use a randomized comparative design, or ROBINS-I (Risk Of Bias In Nonrandomized Studies of Interventions) for those that use a nonrandomized comparative design (25). As more prediction model studies and systematic reviews of such studies are used as evidence for clinical guidance, a tool facilitating quality assessment for individual prediction model studies is urgently needed.
We present PROBAST (Prediction model Risk Of Bias ASsessment Tool), a tool to assess the ROB and concerns regarding the applicability of diagnostic and prognostic prediction model studies. PROBAST can be used to assess studies of model development and model validation, including those updating a prediction model (Box 1 [26]). We refer to the accompanying explanation and elaboration document (27), for detailed explanations of how to use PROBAST and how to judge ROB and applicability.
Box 1. Types of diagnostic and prognostic modeling studies or reports addressed by PROBAST. Adopted from the TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) and CHARMS (CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies) guidance (7, 26). PROBAST = Prediction model Risk Of Bias ASsessment Tool.
Box 1. Types of diagnostic and prognostic modeling studies or reports addressed by PROBAST.
Adopted from the TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) and CHARMS (CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies) guidance (7, 26). PROBAST = Prediction model Risk Of Bias ASsessment Tool.

Methods: Development of PROBAST

Development of PROBAST was based on a 4-stage approach for developing health research reporting guidelines: define the scope, review the evidence base, use a Web-based Delphi procedure, and refine the tool through piloting (28). Guidelines explicitly aimed at the development of quality assessment tools were not available at the time (29).

Development Stage 1: Scope and Definitions

A steering group of 9 experts in prediction model studies and development of quality assessment tools agreed on key features of the desired scope of PROBAST. A panel of 38 experts with different backgrounds further refined the scope during the Web-based Delphi procedure.
PROBAST was designed mainly to assess primary studies included in a systematic review. The group agreed that PROBAST would assess both risk of bias and concerns regarding applicability of a study evaluating a multivariable prediction model to be used for individualized diagnosis or prognosis. A domain-based structure was adopted, similar to that used in other ROB tools, such as ROB 2.0 (24), ROBINS-I (25), QUADAS-2 (Quality Assessment of Diagnostic Accuracy Studies 2) (30), and ROBIS (31).
We agreed that PROBAST should cover primary studies that develop, validate, or update multivariable prediction models aiming to make individualized predictions of a diagnostic or prognostic outcome (Box 1). Studies that use multivariable modeling techniques to identify predictors (such as risk or prognostic factors) associated with an outcome but do not attempt to develop, validate, or update a model for making individualized predictions are not covered by PROBAST (5). Therefore, PROBAST is not intended for predictor finding studies or prediction model impact studies.
Studies of diagnostic and prognostic models often use different terms for predictors and outcomes (Box 2). A multivariable prediction model is defined as any combination or equation of 2 or more predictors for estimating probability or risk for an individual (6, 7, 32–34).
Box 2. Differences between diagnostic and prognostic prediction model studies. PROBAST = Prediction model Risk Of Bias ASsessment Tool.
Box 2. Differences between diagnostic and prognostic prediction model studies.
PROBAST = Prediction model Risk Of Bias ASsessment Tool.

Development Stage 2: Review of Evidence

We used the following 3 approaches to build an evidence base to inform the development of PROBAST: identifying relevant methodological reviews in the area of prediction model research (November 2012 to January 2013), asking members of the steering group to identify relevant methodological studies (January 2013 to March 2013), and using the Delphi procedure to ask members of the wider group to identify additional evidence (February 2012 to July 2014).
Identified literature was used to guide the scope and produce an initial list of signaling questions to consider for inclusion in PROBAST (1, 2, 5–7, 26, 33–40). We grouped signaling questions into common themes to identify possible domains. Additional literature provided as part of the Web-based surveys informed development of the explanation and elaboration document.

Development Stage 3: Web-Based Delphi Procedure

We used a modified Delphi process to gain feedback and agreement on the scope, structure, and content of PROBAST. Web-based surveys were developed to gather structured feedback for each round. The 38-member Delphi group comprised methodological experts in prediction model research and development of quality assessment tools, experienced systematic reviewers, commissioners, and representatives of reimbursement agencies. We included various stakeholders to ensure that the views of end users, methodological experts, and decision makers were represented.
The Delphi process consisted of 7 rounds. Round 1 asked about the scope of the tool, and participants agreed to focus on prediction model studies and follow a domain-based structure. Round 2 aimed to identify relevant domains and agree on which to include. The signaling questions for domains were refined in rounds 3 to 5. Respondents used a 1-to-5 Likert scale to rate each proposed signaling question for inclusion. They could also suggest rephrasing, provide supporting evidence (such as references to relevant studies), and suggest missing signaling questions. Round 6 refined the domains and introduced further optional guidance for using PROBAST. In the last round, participants received the agreed draft version of PROBAST and had the opportunity to provide any final feedback.

Development Stage 4: Piloting and Refining the Tool

We held 6 workshops on PROBAST at consecutive annual Cochrane Colloquia (Quebec, Canada, 2013; Hyderabad, India, 2014; Vienna, Austria, 2015; Seoul, South Korea, 2016; Cape Town, South Africa, 2017; and Edinburgh, United Kingdom, 2018). We also held numerous consecutive workshops with MSc and PhD students (for example, the master's program in epidemiology at Utrecht University [Utrecht, the Netherlands] and the Evidence-Based Health Care program at Oxford University [Oxford, United Kingdom]). In these workshops, we piloted the then-current version of PROBAST to gather feedback on practical issues associated with using the tool so that we could further refine and subsequently validate it. Finally, more than 50 review groups have already piloted PROBAST versions, including the final version, in their reviews. Topics included cancer, cardiology, endocrinology, pulmonology, and orthopedics.
All feedback received from these initiatives was used to further inform the content and structure of PROBAST, wording of the signaling questions, and content of the guidance documents (27).

Results: The PROBAST Tool

What Does PROBAST Assess?

PROBAST assesses both risk of bias and concerns regarding applicability of primary studies that developed or validated multivariable prediction models for diagnosis or prognosis (Boxes 1 and 2).
Development of a prediction model can include adding new predictors to an existing prediction model. Similarly, validation of an existing model can be accompanied by updating and extending the model—that is, development of a new model. PROBAST applies to both situations (Box 1).

Target Users

Although PROBAST was designed for use in systematic reviews, it can be used more generally in critical appraisal of prediction model studies. Potential users of PROBAST include organizations supporting decision making (such as the National Institute for Health and Care Excellence and the Institute for Quality and Efficiency in Health Care); researchers and clinicians who are interested in evidence-based medicine or involved in guideline development; and journal editors, manuscript reviewers, and readers who want to critically appraise prediction model studies.

Definition of ROB and Applicability

Bias is usually defined as the presence of systematic error in a study that leads to distorted or flawed results and hampers the study's internal validity. In prediction model development and validation, known features exist that make a study at ROB, although empirical evidence showing the most important sources of bias is limited. We define ROB to occur when shortcomings in the study design, conduct, or analysis lead to systematically distorted estimates of model predictive performance. Model predictive performance is typically evaluated using measures of calibration and discrimination, and sometimes (notably in diagnostic model studies) classification (7). Thinking about how a hypothetical prediction model study that is methodologically robust would have been designed, conducted, and analyzed helps to understand bias in study estimates of model predictive performance. Many sources of bias identified in other medical research areas are also relevant to prediction model studies, such as blinding of outcome assessors to other study features and use of consistent definitions and measurements for predictors and outcomes within the study.
Concerns regarding the applicability of a primary study to the review question can arise when the population, predictors, or outcomes of the study differ from those specified in the review question. Such concerns may arise when participants in the prediction model study are from a different medical setting from the population defined in the review question—for example, a study that enrolled patients from a hospital setting while the review question specifically relates to patients in primary care. The reported prediction model discrimination and calibration may not be applicable because patients in hospital settings typically have more severe disease than those in primary care (41, 42).
When eligibility criteria, predictors, and outcomes of the primary studies directly match a systematic review question, no concerns regarding applicability will arise. However, the inclusion criteria of a systematic review are typically broader than the focus of the review question. Broader inclusion criteria allow for variation in the searching of the primary studies and thus require careful assessment of each primary study's applicability to the actual review question (7, 27).

Types of Prediction Model Study

A primary study identified as relevant for the review may include the development, validation, or update of 1 or more prediction models. For each study, a PROBAST assessment should be completed for each distinct model that is developed, validated, or updated for making individualized predictions relevant to the systematic review question.
PROBAST includes 4 steps (Table 1). The tool is in the Supplement. We stress the importance of the accompanying paper (27), which provides detailed explanations and guidance for completing each step.
Table 1. Four Steps in PROBAST
Table 1. Four Steps in PROBAST

Step 1: Specify Your Systematic Review Question

Assessors are first asked to report their systematic review question in terms of intended use of the model, targeted participants, predictors used in the modeling, and predicted outcome. Existing guidance (CHARMS [CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies]) can help reviewers define a clear and focused review question (22, 26).

Step 2: Classify the Type of Prediction Model Evaluation

Different signaling questions apply to different types of prediction model evaluation. For each model assessment, reviewers classify a model as “development only,” “development and validation in the same publication,” or “validation only.” When a publication focuses on creating a model by adding 1 or more new predictors to established predictors (or an established model), “development only” should be used. When a publication focuses on validating an existing model in other data and then updating (adjusting or extending) the model such that a new model is actually being developed, “development and validation in the same publication” should be used. Note again that a single publication may address more than 1 model of interest.

Step 3: Assess ROB and Applicability

Step 3 aims to identify areas where bias may be introduced into the prediction model study or where concerns regarding applicability may exist. It involves assessment of the following 4 domains to cover key aspects of prediction model studies: participants, predictors, outcome, and analysis. The ROB component of each domain comprises 4 sections: information used to support the judgment, 2 to 9 signaling questions (20 total across domains), judgment of ROB, and rationale for the judgment (Table 2).
Table 2. PROBAST: Summary of Step 3—Assessment of Risk of Bias and Concerns Regarding Applicability*
Table 2. PROBAST: Summary of Step 3—Assessment of Risk of Bias and Concerns Regarding Applicability*
In the support for judgment box, assessors can record the information used to answer the signaling questions. Signaling questions are answered as “yes,” “probably yes,” “probably no,” “no,” or “no information.” Risk of bias is judged as low, high, or unclear. All signaling questions are phrased so that “yes” indicates absence of bias. Any signaling question answered as “no” or “probably no” flags the potential for bias; assessors will need to use their own judgment to determine whether the domain should be rated as high, low, or unclear ROB. A “no” answer does not automatically result in a high ROB rating. The “no information” category should be used only when reported information is insufficient to permit a judgment. When the rationale is recorded, the ROB rating will be transparent and, where necessary, will facilitate discussion among review authors completing assessments independently.
The first 3 domains are also rated for concern regarding applicability (low, high, or unclear) to the review question defined in step 1. Concerns regarding applicability are rated similarly to ROB, but without signaling questions.
All domains should be completed separately for each evaluation of a distinct model in each study. A team completing a PROBAST assessment likely needs both subject and methodological expertise. The explanation and elaboration document (27) and www.probast.org provide further details on how to score ROB and applicability concerns. Domain 1 (Participants) covers potential sources of bias and applicability concerns related to participant selection methods and data sources (for example, study designs); 2 signaling questions support ROB assessment. Domain 2 (Predictors) covers potential sources of bias and applicability concerns related to the definition and measurement of predictors evaluated for inclusion in the model; 3 signaling questions support ROB assessment. Domain 3 (Outcome) covers potential sources of bias and applicability concerns related to the definition and measurement of the outcome predicted by the model; 6 signaling questions support ROB assessment. Domain 4 (Analysis) covers potential sources of bias in the statistical analysis methods. It assesses aspects related to the choice of analysis method and whether key statistical considerations (for example, missing data) were correctly addressed, and 9 signaling questions support ROB assessment.
Table 2 presents an overview of step 3. Detailed examples of how to answer signaling questions and judge domains can be found in the explanation and elaboration document (27) and on www.probast.org.

Step 4: Overall Judgment

On the basis of the ROB classifications for each domain in step 3, assessors should judge the overall ROB of the prediction model as low, high, or unclear. We recommend rating the prediction model as having low ROB if no relevant shortcomings were identified in the ROB assessment—that is, all domains had low ROB. If at least 1 domain had high ROB, an overall judgment of high ROB should be used. Similarly, unclear ROB should be assigned if unclear ROB was noted in at least 1 domain and all other domains had low ROB.
However, if a prediction model was developed without any external validation on different participants, downgrading to high ROB should still be considered even if all 4 domains had low ROB, unless the model development was based on a very large data set or included some form of internal validation. The explanation and elaboration document (27) provides further details.
Based on the applicability classifications for each domain in step 3, an overall judgment about concerns regarding applicability of the prediction model is needed. A decision of “low concern” should be reached only if all domains showed low concern regarding applicability. Similarly, if 1 or more domains were judged to have high concern, the overall judgment should be “high concern.” “Unclear concern regarding applicability” should be reached only if 1 or more domains were judged as “unclear” in applicability and all other domains were rated to have “low concern.”
The accompanying explanation and elaboration document (27) and www.probast.org give detailed explanation and examples of how to judge the overall ROB and concerns regarding applicability. Table 3 suggests a way to present the results of the PROBAST assessments.
Table 3. Suggested Tabular Presentation for PROBAST Results*
Table 3. Suggested Tabular Presentation for PROBAST Results*

Discussion

Assessment of the quality of included studies is an essential component of all systematic reviews and evidence syntheses. Systematic reviews of prediction model studies are a rapidly evolving area (22). As more prediction model studies and systematic reviews of such studies enter the evidence base, a tool facilitating quality assessment for individual prediction model studies is urgently needed. To our knowledge, PROBAST is the first rigorously developed tool designed specifically to assess the quality of prediction model studies for development, validation, or updating of both diagnostic and prognostic models, regardless of the medical domain, type of outcome, predictors, or statistical technique used.
We adopted a domain-based structure similar to that used in other recently developed tools, such as ROB 2.0 (24), QUADAS-2 for diagnostic accuracy studies (30), ROBINS-I for nonrandomized studies (25), and ROBIS for systematic reviews (31). All stages of PROBAST development included a wide range of stakeholders, and we started piloting the tool in early versions to allow incorporation of feedback from direct reviewer experience into the final tool. We feel that these 2 features have resulted in a tool that is both methodologically sound and user-friendly.
Potential users of PROBAST include systematic review authors, health care decision makers, and researchers and clinicians who are interested in evidence-based medicine or involved in guideline development, as well as journal editors and manuscript reviewers.
The explanation and elaboration document (27) provides explicit guidance and an explanation of how to use PROBAST. Researchers seeking to understand and use PROBAST should always read the accompanying document in conjunction with the current article. A multidisciplinary team with both subject and methodological expertise should assess prediction model studies.
As with other ROB and reporting guidelines in medical research, PROBAST and its guidance will require updating as methods for prediction model studies develop. We recommend downloading the latest version of PROBAST and accompanying guidance, including detailed examples, from the Web site (www.probast.org).

Appendix: Members of the PROBAST Group

PROBAST Steering Group

Members of the PROBAST Group who authored this work: Robert F. Wolff, MD (Kleijnen Systematic Reviews, York, United Kingdom); Prof. Karel G.M. Moons, PhD (Julius Center for Health Sciences and Primary Care and Cochrane Netherlands, University Medical Center (UMC) Utrecht, Utrecht University, Utrecht, the Netherlands); Prof. Richard D. Riley, PhD (Keele University, Keele, United Kingdom); Penny F. Whiting, PhD (University Hospitals Bristol NHS Foundation Trust and University of Bristol, Bristol, United Kingdom); Marie Westwood, PhD (Kleijnen Systematic Reviews, York, United Kingdom); Prof. Gary S. Collins, PhD (Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, United Kingdom); Johannes B. Reitsma, MD, PhD (Julius Center for Health Sciences and Primary Care and Cochrane Netherlands, UMC Utrecht, Utrecht University, Utrecht, the Netherlands); Prof. Jos Kleijnen, MD, PhD (Kleijnen Systematic Reviews, York, United Kingdom, and School for Public Health and Primary Care, Maastricht University, Maastricht, the Netherlands); and Sue Mallett, DPhil (Institute of Applied Health Research, University of Birmingham, Birmingham, United Kingdom).

PROBAST Delphi Group

Members of the PROBAST group who were nonauthor contributors: Prof. Doug Altman, PhD (Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, United Kingdom); Prof. Patrick Bossuyt, PhD (Division of Clinical Methods & Public Health, University of Amsterdam, Amsterdam, the Netherlands); Prof. Nancy R. Cook, ScD (Brigham and Women's Hospital, Boston, Massachusetts); Gennaro D'Amico, MD (Ospedale Vincenzo Cervello, Palermo, Italy); Thomas P.A. Debray, PhD, MSc (Julius Center for Health Sciences and Primary Care and Cochrane Netherlands, UMC Utrecht, Utrecht University, Utrecht, the Netherlands); Prof. Jon Deeks, PhD (Institute of Applied Health Research, University of Birmingham, Birmingham, United Kingdom); Joris de Groot, PhD (Philips Image Guided Therapy Systems, Best, the Netherlands); Emanuele di Angelantonio, PhD, MSc (Department of Public Health and Primary Care, University of Cambridge, Cambridge, United Kingdom); Prof. Tom Fahey, MD, MSc (Royal College of Surgeons in Ireland, Dublin, Ireland); Prof. Frank Harrell, PhD (Department of Biostatistics, Vanderbilt University, Nashville, Tennessee); Prof. Jill A. Hayden, PhD (Department of Community Health and Epidemiology, Dalhousie University, Halifax, Nova Scotia, Canada); Martijn W. Heymans, PhD (Department of Epidemiology and Biostatistics, Amsterdam Public Health Research Institute, Vrije Universiteit UMC, Amsterdam, the Netherlands); Lotty Hooft, PhD (Julius Center for Health Sciences and Primary Care and Cochrane Netherlands, UMC Utrecht, Utrecht University, Utrecht, the Netherlands); Prof. Chris Hyde, PhD (Institute of Health Research, University of Exeter Medical School, Exeter, United Kingdom); Prof. John Ioannidis, MD, DSc (Meta-Research Innovation Center at Stanford, Stanford University, Palo Alto, California); Prof. Alfonso Iorio, MD, PhD (Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada); Stephen Kaptoge, PhD (Department of Public Health and Primary Care, University of Cambridge, Cambridge, United Kingdom); Prof. André Knottnerus, MD, PhD (Department of Family Medicine, Maastricht University, Maastricht, the Netherlands); Mariska Leeflang, PhD, DVM (Department of Clinical Epidemiology, Biostatistics and Bioinformatics, University of Amsterdam, Amsterdam, the Netherlands); Frances Nixon, BSc (National Institute for Health and Care Excellence, Manchester, United Kingdom); Prof. Pablo Perel, MD, PhD, MSc (Centre for Global Chronic Conditions, London School of Hygiene and Tropical Medicine, London, United Kingdom); Bob Phillips, PhD, MMedSci (Centre for Reviews and Dissemination, York, United Kingdom); Heike Raatz, MD, MSc (Kleijnen Systematic Reviews, York, United Kingdom); Rob Riemsma, PhD (Kleijnen Systematic Reviews, York, United Kingdom); Prof. Maroeska Rovers, PhD (Departments of Operating Rooms and Health Evidence, Radboud UMC, Nijmegen, the Netherlands); Anne W.S. Rutjes, PhD, MHSc (Institute of Social and Preventive Medicine and Institute of Primary Health Care, University of Bern, Bern, Switzerland); Prof. Willi Sauerbrei, PhD (Institute of Medical Biometry and Statistics, Faculty of Medicine and Medical Center, University of Freiburg, Freiburg, Germany); Stefan Sauerland, MD, MPH (Institute for Quality and Efficiency in Healthcare, Cologne, Germany); Fülöp Scheibler, PhD, MA (UMC Schleswig-Holstein, Kiel, Germany); Prof. Rob Scholten, MD, PhD (Julius Center for Health Sciences and Primary Care and Cochrane Netherlands, UMC Utrecht, Utrecht University, Utrecht, the Netherlands); Ewoud Schuit, PhD, MSc (Julius Center for Health Sciences and Primary Care and Cochrane Netherlands, UMC Utrecht, Utrecht University, Utrecht, the Netherlands); Prof. Ewout Steyerberg, PhD (Department of Public Health, Erasmus UMC, Rotterdam, and Department of Biomedical Data Sciences, Leiden UMC, Leiden, the Netherlands); Toni Tan, MSc (National Institute for Health and Care Excellence, Manchester, United Kingdom); Gerben ter Riet, MD, PhD (Department of General Practice, University of Amsterdam, Amsterdam, the Netherlands); Prof. Danielle van der Windt, PhD (Centre for Prognosis Research, Keele University, Keele, United Kingdom); Yvonne Vergouwe, PhD (Department of Public Health, Erasmus UMC, Rotterdam, the Netherlands); Andrew Vickers, PhD (Memorial Sloan-Kettering Cancer Center, New York, New York); and Angela M. Wood, PhD (Department of Public Health and Primary Care, University of Cambridge, Cambridge, United Kingdom).
The Delphi group members made substantial contributions to the conception and design, acquisition of data, or analysis and interpretation of the data; they drafted the article or revised it critically for important intellectual content; and they approved the final version to be published.

Supplemental Material

Supplement. PROBAST Form

References

1.
Bouwmeester WZuithoff NPMallett SGeerlings MIVergouwe YSteyerberg EWet al. Reporting and methods in clinical prediction research: a systematic review. PLoS Med. 2012;9:1-12. [PMID: 22629234]  doi: 10.1371/journal.pmed.1001221
2.
Steyerberg EWMoons KGvan der Windt DAHayden JAPerel PSchroter Set alPROGRESS Group. Prognosis Research Strategy (PROGRESS) 3: prognostic model research. PLoS Med. 2013;10:e1001381. [PMID: 23393430]  doi: 10.1371/journal.pmed.1001381
3.
Knottnerus JA. Diagnostic prediction rules: principles, requirements and pitfalls. Prim Care. 1995;22:341-63. [PMID: 7617791]
4.
Lamain-de Ruiter MKwee ANaaktgeboren CAde Groot IEvers IMGroenendaal Fet al. External validation of prognostic models to predict risk of gestational diabetes mellitus in one Dutch cohort: prospective multicentre cohort study. BMJ. 2016;354:i4338. [PMID: 27576867]  doi: 10.1136/bmj.i4338
5.
Riley RDHayden JASteyerberg EWMoons KGAbrams KKyzas PAet alPROGRESS Group. Prognosis Research Strategy (PROGRESS) 2: prognostic factor research. PLoS Med. 2013;10:e1001380. [PMID: 23393429]  doi: 10.1371/journal.pmed.1001380
6.
Collins GSReitsma JBAltman DGMoons KG. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement. Ann Intern Med. 2015;162:55-63. [PMID: 25560714].  doi: 10.7326/M14-0697
7.
Moons KGAltman DGReitsma JBIoannidis JPMacaskill PSteyerberg EWet al. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med. 2015;162:W1-73. [PMID: 25560730].  doi: 10.7326/M14-0698
8.
Collins GSMallett SOmar OYu LM. Developing risk prediction models for type 2 diabetes: a systematic review of methodology and reporting. BMC Med. 2011;9:103. [PMID: 21902820]  doi: 10.1186/1741-7015-9-103
9.
Kansagara DEnglander HSalanitro AKagen DTheobald CFreeman Met al. Risk prediction models for hospital readmission: a systematic review. JAMA. 2011;306:1688-98. [PMID: 22009101]  doi: 10.1001/jama.2011.1515
10.
Steurer JHaller CHäuselmann HBrunner FBachmann LM. Clinical value of prognostic instruments to identify patients with an increased risk for osteoporotic fractures: systematic review. PLoS One. 2011;6:e19994. [PMID: 21625596]  doi: 10.1371/journal.pone.0019994
11.
Altman DG. Prognostic models: a methodological framework and review of models for breast cancer. Cancer Invest. 2009;27:235-43. [PMID: 19291527]  doi: 10.1080/07357900802572110
12.
Shariat SFKarakiewicz PISuardi NKattan MW. Comparison of nomograms with other methods for predicting outcomes in prostate cancer: a critical analysis of the literature. Clin Cancer Res. 2008;14:4400-7. [PMID: 18628454]  doi: 10.1158/1078-0432.CCR-07-4713
13.
Counsell CDennis M. Systematic review of prognostic models in patients with acute stroke. Cerebrovasc Dis. 2001;12:159-70. [PMID: 11641579]
14.
Perel PPrieto-Merino DShakur HClayton TLecky FBouamra Oet al. Predicting early death in patients with traumatic bleeding: development and validation of prognostic model. BMJ. 2012;345:e5166. [PMID: 22896030]  doi: 10.1136/bmj.e5166
15.
Damen JAHooft LSchuit EDebray TPCollins GSTzoulaki Iet al. Prediction models for cardiovascular disease risk in the general population: systematic review. BMJ. 2016;353:i2416. [PMID: 27184143]  doi: 10.1136/bmj.i2416
16.
Hippisley-Cox JCoupland CVinogradova YRobson JMinhas RSheikh Aet al. Predicting cardiovascular risk in England and Wales: prospective derivation and validation of QRISK2. BMJ. 2008;336:1475-82. [PMID: 18573856]  doi: 10.1136/bmj.39609.449676.25
17.
Graham RMancher MMiller Wolman DGreenfield SSteinberg E eds. Clinical Practice Guidelines We Can Trust. Washington, DC: National Academies Pr; 2011.
18.
Goff DC JrLloyd-Jones DMBennett GCoady SD'Agostino RBGibbons Ret alAmerican College of Cardiology/American Heart Association Task Force on Practice Guidelines. 2013 ACC/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. Circulation. 2014;129:S49-73. [PMID: 24222018]  doi: 10.1161/01.cir.0000437741.48606.98
19.
Rabar SLau RO'Flynn NLi LBarry PGuideline Development Group. Risk assessment of fragility fractures: summary of NICE guidance. BMJ. 2012;345:e3698. [PMID: 22875946]  doi: 10.1136/bmj.e3698
20.
Centre for Reviews and Dissemination. Systematic Reviews: CRD's Guidance for Undertaking Reviews in Health Care. York, United Kingdom: University of York; 2009.
21.
Higgins JPTGreen S eds. Cochrane Handbook for Systematic Reviews of Interventions. Chichester, United Kingdom: Wiley-Blackwell; 2011.
22.
Debray TPDamen JASnell KIEnsor JHooft LReitsma JBet al. A guide to systematic review and meta-analysis of prediction model performance. BMJ. 2017;356:i6460. [PMID: 28057641]  doi: 10.1136/bmj.i6460
23.
Hayden JAvan der Windt DACartwright JLCôté PBombardier C. Assessing bias in studies of prognostic factors. Ann Intern Med. 2013;158:280-6. [PMID: 23420236].  doi: 10.7326/0003-4819-158-4-201302190-00009
24.
Higgins JPTSavović JPage MJSterne JACROB2 Development Group. A revised tool for assessing risk of bias in randomized trials. In: Chandler J, McKenzie J, Boutron I, Welch V, eds. Cochrane Methods. London: Cochrane; 2018:1-69.
25.
Sterne JAHernán MAReeves BCSavovic JBerkman NDViswanathan Met al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355:i4919. [PMID: 27733354]  doi: 10.1136/bmj.i4919
26.
Moons KGde Groot JABouwmeester WVergouwe YMallett SAltman DGet al. Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist. PLoS Med. 2014;11:e1001744. [PMID: 25314315]  doi: 10.1371/journal.pmed.1001744
27.
Moons KGMWolff RFRiley RDWhiting PFWestwood MCollins GSet al. PROBAST: a tool to assess risk of bias and applicability of prediction model studies: explanation and elaboration. Ann Intern Med. 2019;170:W1-W33.  doi: 10.7326/M18-1377
28.
Moher DSchulz KFSimera IAltman DG. Guidance for developers of health research reporting guidelines. PLoS Med. 2010;7:e1000217. [PMID: 20169112]  doi: 10.1371/journal.pmed.1000217
29.
Whiting PWolff RMallett SSimera ISavovic J. A proposed framework for developing quality assessment tools. Syst Rev. 2017;6:204. [PMID: 29041953]  doi: 10.1186/s13643-017-0604-6
30.
Whiting PFRutjes AWWestwood MEMallett SDeeks JJReitsma JBet alQUADAS-2 Group. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011;155:529-36. [PMID: 22007046].  doi: 10.7326/0003-4819-155-8-201110180-00009
31.
Whiting PSavovic JHiggins JPCaldwell DMReeves BCShea Bet alROBIS group. ROBIS: a new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016;69:225-34. [PMID: 26092286]  doi: 10.1016/j.jclinepi.2015.06.005
32.
Canet JGallart LGomar CPaluzie GVallès JCastillo Jet alARISCAT Group. Prediction of postoperative pulmonary complications in a population-based surgical cohort. Anesthesiology. 2010;113:1338-50. [PMID: 21045639]  doi: 10.1097/ALN.0b013e3181fc6e0a
33.
Collins GSOmar OShanyinde MYu LM. A systematic review finds prediction models for chronic kidney disease were poorly reported and often developed using inappropriate methods. J Clin Epidemiol. 2013;66:268-77. [PMID: 23116690]  doi: 10.1016/j.jclinepi.2012.06.020
34.
Moons KGRoyston PVergouwe YGrobbee DEAltman DG. Prognosis and prognostic research: what, why, and how? BMJ. 2009;338:b375. [PMID: 19237405]  doi: 10.1136/bmj.b375
35.
Harrell FE. Regression Modeling Strategies, With Applications to Linear Models, Logistic Regression, and Survival Analysis. New York: Springer; 2001.
36.
Hemingway HCroft PPerel PHayden JAAbrams KTimmis Aet alPROGRESS Group. Prognosis Research Strategy (PROGRESS) 1: a framework for researching clinical outcomes. BMJ. 2013;346:e5595. [PMID: 23386360]  doi: 10.1136/bmj.e5595
37.
Mallett SRoyston PDutton SWaters RAltman DG. Reporting methods in studies developing prognostic models in cancer: a review. BMC Med. 2010;8:20. [PMID: 20353578]  doi: 10.1186/1741-7015-8-20
38.
Altman DGVergouwe YRoyston PMoons KG. Prognosis and prognostic research: validating a prognostic model. BMJ. 2009;338:b605. [PMID: 19477892]  doi: 10.1136/bmj.b605
39.
Moons KGAltman DGVergouwe YRoyston P. Prognosis and prognostic research: application and impact of prognostic models in clinical practice. BMJ. 2009;338:b606. [PMID: 19502216]  doi: 10.1136/bmj.b606
40.
Royston PMoons KGAltman DGVergouwe Y. Prognosis and prognostic research: developing a prognostic model. BMJ. 2009;338:b604. [PMID: 19336487]  doi: 10.1136/bmj.b604
41.
Knottnerus JA. Between iatrotropic stimulus and interiatric referral: the domain of primary care research. J Clin Epidemiol. 2002;55:1201-6. [PMID: 12547450]
42.
Oudega RHoes AWMoons KG. The Wells rule does not adequately rule out deep venous thrombosis in primary care patients. Ann Intern Med. 2005;143:100-7. [PMID: 16027451]

Comments

0 Comments
Sign In to Submit A Comment
Robert F. Wolff, Karel G.M. Moons, Sue Mallett1 February 2019
Response to: Feasibility of using PROBAST to assess bias and applicability of dementia prediction models
We would like to congratulate Silvan Licher and his colleagues on their work on dementia prediction models(1,2) and would like to thank them for their positive feedback on the use of PROBAST, a tool to assess the risk of bias and applicability of prediction model studies.

In fact, we are pleasantly surprised to see that PROBAST, which has only been published recently, is already been used.(3,4) Furthermore, we are pleased to see that the comment highlighted the usefulness of two key elements of PROBAST:
1. Four domains (participants, predictors, outcome, and analysis) are rated as part of a PROBAST assessment, allowing users to pinpoint shortcomings in the methodology of the underlying study reporting the development and/or validation of a prediction model
2. PROBAST allows the assessment of the risk of bias and applicability of a study. The comment illustrated that PROBAST appears to have been a useful tool in assessing the five models investigated by Licher et al., i.e. identifying a prediction model study which was rated as low risk of bias and low concerns regarding applicability.

On www.probast.org, we will list this study as an example of the use of PROBAST. The website also presents other details relevant to prediction research, e.g. information on workshops, details on relevant research initiatives as well as the current version of the PROBAST tool.

References:
1. Licher S, Yilmaz P, Leening MJG, Wolters FJ, Vernooij MW, Stephan BCM, et al. External validation of four dementia prediction models for use in the general community-dwelling population: a comparative analysis from the Rotterdam Study. European Journal of Epidemiology. 2018;33(7):645-55.
2. Licher S, Leening MJG, Yilmaz P, Wolters FJ, Heeringa J, Bindels PJE, et al. Development and validation of a dementia risk prediction model in the general population: an analysis of three longitudinal studies. Am J Psychiatry. 2018. Epub Ahead of Print.
3. Wolff RF, Moons KGM, Riley RD, Whiting PF, Westwood M, Collins GS, Reitsma JB, Kleijnen J, Mallett S, PROBAST Group. PROBAST: a tool to assess the risk of bias and applicability of prediction model studies. Ann Intern Med 2019;170(1):51-58. Freely available from: http://annals.org/aim/fullarticle/2719961/probast-tool-assess-risk-bias-applicability-prediction-model-studies
4. Moons KGM, Wolff RF, Riley RD, Whiting PF, Westwood M, Collins GS, Reitsma JB, Kleijnen J, Mallett S. PROBAST: a tool to assess the risk of bias and applicability of prediction model studies: explanation and elaboration. Ann Intern Med 2019;170(1):W1-W33. Freely available from: http://annals.org/aim/fullarticle/2719962/probast-tool-assess-risk-bias-applicability-prediction-model-studies-explanation

Disclosures: See disclosure information in the PROBAST article

Silvan Licher; MD – [email protected], Pinar Yilmaz; MD – [email protected], M. Kamran Ikram; MD;PhD – [email protected], M. Arfan Ikram;MD;PhD – [email protected], Maarten J.G. Leening;MD;PhD – [email protected]30 January 2019
Feasibility of using PROBAST to assess bias and applicability of dementia prediction models
With the recent publication of the Prediction model Risk Of Bias ASsessment Tool (PROBAST) in this journal, a framework for critical and systematic evaluation of prediction models has been established (1). We aimed to assess the feasibility of PROBAST and whether its results can facilitate model selection for clinical practice. We used dementia prediction models as an illustration. Numerous prediction models for dementia have been developed (2), yet none of these has been recognized as an established model to facilitate targeted preventive efforts or select high risk individuals for inclusion in clinical trials. Several systematic reviews on dementia risk prediction models have been published, but the internal validity (i.e. bias), and applicability of these models have not been systematically evaluated (2-4).

Following PROBAST, we selected dementia prediction models that have been developed and externally validated to identify individuals at high risk in the general population in order to facilitate targeted preventive efforts (Steps 1 and 2). We systematically examined the risk of bias and concerns regarding the applicability for each of these models (Step 3). Based on a previously published literature search on dementia models (updated until Jan 1st, 2019) (4, 5), we identified five validated models (Cardiovascular Risk Factors, Aging, and Incidence of Dementia; ANU-Alzheimer's Disease Risk Index; Brief Dementia Screening Indicator; Dementia Risk Score; and The Rotterdam Study model). PROBAST identified high risk of bias in three of these models that could compromise their internal validity but raised low concern regarding applicability (Step 4). For one model, we identified both high risk of bias and high concerns regarding applicability. A single model met all criteria, indicating that there were low concerns regarding risk of bias and applicability. Of the four domains in PROBAST (i.e. participants, predictors, outcome, and analysis), a high risk of bias in the analysis domain was common, being present in four of the five models. Detailed PROBAST assessments for each of these models are available upon request.

We conclude that PROBAST facilitates systematic examination and summarizing of the quality and applicability of prediction models, and thereby facilitates selection of prediction models for clinical use. Importantly, our application of PROBAST revealed methodological shortcomings for the majority of dementia prediction models, which may compromise reliable risk estimation in clinical practice. These findings highlight that rigorous external validation of prediction models is not sufficient to guarantee internal validity.
References
1. Wolff RF, Moons KGM, Riley RD, Whiting PF, Westwood M, Collins GS, et al. PROBAST: a tool to assess the risk of bias and applicability of prediction model studies. Ann Intern Med. 2019;170(1):51-8.
2. Tang EY, Harrison SL, Errington L, Gordon MF, Visser PJ, Novak G, et al. Current developments in dementia risk prediction modelling: an updated systematic review. Plos One. 2015;10(9):e0136181.
3. Hou XH, Feng L, Zhang C, Cao XP, Tan L, Yu JT. Models for predicting risk of dementia: a systematic review. J Neurol Neurosurg Psychiatry. 2018. Epub Ahead of Print.
4. Licher S, Yilmaz P, Leening MJG, Wolters FJ, Vernooij MW, Stephan BCM, et al. External validation of four dementia prediction models for use in the general community-dwelling population: a comparative analysis from the Rotterdam Study. European Journal of Epidemiology. 2018;33(7):645-55.
5. Licher S, Leening MJG, Yilmaz P, Wolters FJ, Heeringa J, Bindels PJE, et al. Development and validation of a dementia risk prediction model in the general population: an analysis of three longitudinal studies. Am J Psychiatry. 2018. Epub Ahead of Print.

Disclosures: We declare no potential financial conflicts of interest. The literature search that was done for this work also identified one model that that has been developed and validated by the authors themselves.

Information & Authors

Information

Published In

cover image Annals of Internal Medicine
Annals of Internal Medicine
Volume 170Number 11 January 2019
Pages: 51 - 58

History

Published online: 1 January 2019
Published in issue: 1 January 2019

Keywords

Authors

Affiliations

Robert F. Wolff, MD
Kleijnen Systematic Reviews, York, United Kingdom (R.F.W., M.W.)
Karel G.M. Moons, PhD
Julius Center for Health Sciences and Primary Care and Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, the Netherlands (K.G.M., J.B.R.)
Richard D. Riley, PhD
Centre for Prognosis Research, Research Institute for Primary Care and Health Sciences, Keele University, Keele, United Kingdom (R.D.R.)
Penny F. Whiting, PhD
Medical School of the University of Bristol and National Institute for Health Research Collaboration for Leadership in Applied Health Research and Care West, University Hospitals Bristol National Health Service Foundation Trust, Bristol, United Kingdom (P.F.W.)
Marie Westwood, PhD
Kleijnen Systematic Reviews, York, United Kingdom (R.F.W., M.W.)
Gary S. Collins, PhD
Centre for Statistics in Medicine, University of Oxford, Oxford, United Kingdom (G.S.C.)
Johannes B. Reitsma, MD, PhD
Julius Center for Health Sciences and Primary Care and Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, the Netherlands (K.G.M., J.B.R.)
Jos Kleijnen, MD, PhD
Kleijnen Systematic Reviews, York, United Kingdom, and School for Public Health and Primary Care, Maastricht University, Maastricht, the Netherlands (J.K.)
Sue Mallett, DPhil
Institute of Applied Health Research, National Institute for Health Research Birmingham Biomedical Research Centre, College of Medical and Dental Sciences, University of Birmingham, Birmingham, United Kingdom (S.M.)
for the PROBAST Group†
Disclaimer: This report presents independent research supported by the National Institute for Health Research (NIHR). The views and opinions expressed in this publication are those of the authors and do not necessarily reflect those of the National Health Service (NHS), the NIHR, or the Department of Health and Social Care.
Acknowledgment: The authors thank the members of the Delphi panel (Appendix) for their valuable input and all testers, especially Cordula Braun, Johanna A.A.G. Damen, Paul Glasziou, Pauline Heus, Lotty Hooft, and Romin Pajouheshnia, for providing feedback on PROBAST. They also thank Janine Ross and Steven Duffy for support in managing the references.
Financial Support: Drs. Moons and Reitsma received financial support from the Netherlands Organisation for Scientific Research (ZONMW 918.10.615 and 91208004). Dr. Riley is a member of the Evidence Synthesis Working Group funded by the NIHR School for Primary Care Research (project 390). Dr. Whiting (time) was supported by the NIHR Collaboration for Leadership in Applied Health Research and Care West at University Hospitals Bristol NHS Foundation Trust. Dr. Collins was supported by the NIHR Biomedical Research Centre, Oxford. Dr. Mallett is supported by NIHR Birmingham Biomedical Research Centre at the University Hospitals Birmingham NHS Foundation Trust and the University of Birmingham. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Disclosures: Dr. Wolff reports grants from Bayer, Biogen, Pfizer, UCB, Amgen, BioMarin, Grünenthal, Chiesi, and TESARO outside the submitted work. Dr. Westwood reports grants from Bayer, Biogen, Pfizer, UCB, Amgen, BioMarin, Grünenthal, Chiesi, and TESARO outside the submitted work. Dr. Kleijnen reports grants from Bayer, Biogen, Pfizer, UCB, Amgen, BioMarin, Grünenthal, Chiesi, and TESARO outside the submitted work. Authors not named here have disclosed no conflicts of interest. Disclosures can also be viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum=M18-1376.
Corresponding Author: Robert F. Wolff, MD, Kleijnen Systematic Reviews Ltd, Unit 6, Escrick Business Park, Riccall Road, Escrick, York YO19 6FD, United Kingdom; e-mail, [email protected].
Current Author Addresses: Drs. Wolff, Westwood, and Kleijnen: Kleijnen Systematic Reviews Ltd, Unit 6, Escrick Business Park, Riccall Road, Escrick, York YO19 6FD, United Kingdom.
Drs. Moons and Reitsma: Julius Centre for Health Sciences and Primary Care, UMC Utrecht, Utrecht University, PO Box 85500, 3508 GA Utrecht, the Netherlands.
Dr. Riley: Centre for Prognosis Research, Research Institute for Primary Care and Health Sciences, Keele University, Staffordshire ST5 5BG, United Kingdom.
Dr. Whiting: NIHR CLAHRC West, University Hospitals Bristol NHS Foundation Trust and School of Social and Community Medicine, University of Bristol, Bristol BS1 2NT, United Kingdom.
Dr. Collins: Centre for Statistics in Medicine, NDORMS, University of Oxford, Botnar Research Centre, Windmill Road, Oxford OX3 7LD, United Kingdom.
Dr. Mallett: Institute of Applied Health Sciences, University of Birmingham, Edgbaston, Birmingham B15 2TT, United Kingdom.
Author Contributions: Conception and design: R.F. Wolff, K.G.M. Moons, R.D. Riley, P.F. Whiting, M. Westwood, G.S. Collins, J.B. Reitsma, J. Kleijnen, S. Mallett.
Analysis and interpretation of the data: R.F. Wolff, K.G.M. Moons, R.D. Riley, P.F. Whiting, M. Westwood, G.S. Collins, J.B. Reitsma, J. Kleijnen, S. Mallett.
Drafting of the article: R.F. Wolff, K.G.M. Moons, P.F. Whiting, M. Westwood, S. Mallett.
Critical revision of the article for important intellectual content: R.F. Wolff, K.G.M. Moons, R.D. Riley, P.F. Whiting, M. Westwood, G.S. Collins, J.B. Reitsma, J. Kleijnen, S. Mallett.
Final approval of the article: R.F. Wolff, K.G.M. Moons, R.D. Riley, P.F. Whiting, M. Westwood, G.S. Collins, J.B. Reitsma, J. Kleijnen, S. Mallett.
Statistical expertise: K.G.M. Moons, R.D. Riley, G.S. Collins, J.B. Reitsma, S. Mallett.
Obtaining of funding: K.G.M. Moons, R.D. Riley, P.F. Whiting, G.S. Collins, J.B. Reitsma, J. Kleijnen, S. Mallett.
Administrative, technical, or logistic support: R.F. Wolff, K.G.M. Moons, J. Kleijnen, S. Mallett.
Collection and assembly of data: R.F. Wolff, K.G.M. Moons, R.D. Riley, P.F. Whiting, M. Westwood, G.S. Collins, J.B. Reitsma, J. Kleijnen, S. Mallett.
* Drs. Wolff and Moons contributed equally to this work.
† For members of the PROBAST Group, see the Appendix.

Metrics & Citations

Metrics

Citations

If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. For an editable text file, please select Medlars format which will download as a .txt file. Simply select your manager software from the list below and click Download.

For more information or tips please see 'Downloading to a citation manager' in the Help menu.

Format





Download article citation data for:
Robert F. Wolff, Karel G.M. Moons, Richard D. Riley, et al; for the PROBAST Group†. PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies. Ann Intern Med.2019;170:51-58. [Epub 1 January 2019]. doi:10.7326/M18-1376

View More

Get Access

Login Options:
Purchase

You will be redirected to acponline.org to sign-in to Annals to complete your purchase.

Access to EPUBs and PDFs for FREE Annals content requires users to be registered and logged in. A subscription is not required. You can create a free account below or from the following link. You will be redirected to acponline.org to create an account that will provide access to Annals. If you are accessing the Free Annals content via your institution's access, registration is not required.

Create your Free Account

You will be redirected to acponline.org to create an account that will provide access to Annals.

View options

PDF/ePub

View PDF/ePub

Media

Figures

Other

Tables

Share

Share

Copy the content Link

Share on social media