Research and Reporting Methods
14 June 2022

QUAPAS: An Adaptation of the QUADAS-2 Tool to Assess Prognostic Accuracy Studies

Publication: Annals of Internal Medicine
Volume 175, Number 7

Abstract

Whereas diagnostic tests help detect the cause of signs and symptoms, prognostic tests assist in evaluating the probable course of the disease and future outcome. Studies to evaluate prognostic tests are longitudinal, which introduces sources of bias different from those for diagnostic accuracy studies. At present, systematic reviews of prognostic tests often use the QUADAS-2 (Quality Assessment of Diagnostic Accuracy Studies 2) tool to assess risk of bias and applicability of included studies because no equivalent instrument exists for prognostic accuracy studies.
QUAPAS (Quality Assessment of Prognostic Accuracy Studies) is an adaptation of QUADAS-2 for prognostic accuracy studies. Questions likely to identify bias were evaluated in parallel and collated from QUIPS (Quality in Prognosis Studies) and PROBAST (Prediction Model Risk of Bias Assessment Tool) and paired to the corresponding question (or domain) in QUADAS-2. A steering group conducted and reviewed 3 rounds of modifications before arriving at the final set of domains and signaling questions.
QUAPAS follows the same steps as QUADAS-2: Specify the review question, tailor the tool, draw a flow diagram, judge risk of bias, and identify applicability concerns. Risk of bias is judged across the following 5 domains: participants, index test, outcome, flow and timing, and analysis. Signaling questions assist the final judgment for each domain. Applicability concerns are assessed for the first 4 domains.
The authors used QUAPAS in parallel with QUADAS-2 and QUIPS in a systematic review of prognostic accuracy studies. QUAPAS improved the assessment of the flow and timing domain and flagged a study at risk of bias in the new analysis domain. Judgment of risk of bias in the analysis domain was challenging because of sparse reporting of statistical methods.

Get full access to this article

View all available purchase options and get full access to this article.

Supplemental Material

Supplement. Supplementary Material

References

1.
FDA-NIH Biomarker Working Group. BEST (Biomarkers, EndpointS, and other Tools) Resource. Food and Drug Administration, National Institutes of Health; 2016.
2.
Guyatt GH, Tugwell PX, Feeny DH, et al. A framework for clinical evaluation of diagnostic technologies. CMAJ. 1986;134:587-94. [PMID: 3512062]
3.
Raja AS, Greenberg JO, Qaseem A, et al; Clinical Guidelines Committee of the American College of Physicians. Evaluation of patients with suspected acute pulmonary embolism: best practice advice from the clinical guidelines committee of the American College of Physicians. Ann Intern Med. 2015;163:701-11. [PMID: 26414967] doi: 10.7326/M14-1772
4.
Hemingway H. Prognosis research: why is Dr. Lydgate still waiting. J Clin Epidemiol. 2006;59:1229-38. [PMID: 17098565]
5.
Freund Y, Lemachatti N, Krastinova E, et al; French Society of Emergency Medicine Collaborators Group. Prognostic accuracy of Sepsis-3 criteria for in-hospital mortality among patients with suspected infection presenting to the emergency department. JAMA. 2017;317:301-308. [PMID: 28114554] doi: 10.1001/jama.2016.20329
6.
Cardoso FS, Ricardo LB, Oliveira AM, et al. C-reactive protein prognostic accuracy in acute pancreatitis: timing of measurement and cutoff points. Eur J Gastroenterol Hepatol. 2013;25:784-9. [PMID: 23492986] doi: 10.1097/MEG.0b013e32835fd3f0
7.
Knottnerus JA, Muris JW. Assessment of the accuracy of diagnostic tests: the cross-sectional study. J Clin Epidemiol. 2003;56:1118-28. [PMID: 14615003]
8.
Pepe MS, Feng Z, Janes H, et al. Pivotal evaluation of the accuracy of a biomarker used for classification or prediction: standards for study design. J Natl Cancer Inst. 2008;100:1432-8. [PMID: 18840817] doi: 10.1093/jnci/djn326
9.
Whiting PF, Rutjes AW, Westwood ME, et al; QUADAS-2 Group. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011;155:529-36. [PMID: 22007046] doi: 10.7326/0003-4819-155-8-201110180-00009
10.
Hayden JA, van der Windt DA, Cartwright JL, et al. Assessing bias in studies of prognostic factors. Ann Intern Med. 2013;158:280-6. [PMID: 23420236] doi: 10.7326/0003-4819-158-4-201302190-00009
11.
Wolff RF, Moons KGM, Riley RD, et al; PROBAST Group. PROBAST: a tool to assess the risk of bias and applicability of prediction model studies. Ann Intern Med. 2019;170:51-58. [PMID: 30596875] doi: 10.7326/M18-1376
12.
Moons KGM, Wolff RF, Riley RD, et al. PROBAST: a tool to assess risk of bias and applicability of prediction model studies: explanation and elaboration. Ann Intern Med. 2019;170:W1-W33. [PMID: 30596876] doi: 10.7326/M18-1377
13.
Halligan S, Boone D, Bhatnagar G, et al. Prognostic biomarkers to identify patients destined to develop severe Crohn's disease who may benefit from early biological therapy: protocol for a systematic review, meta-analysis and external validation. Syst Rev. 2016;5:206. [PMID: 27903285]
14.
Reynolds JC, Issa MS, Nicholson TC, et al; Advanced Life Support Task Force of the International Liaison Committee on Resuscitation. Prognostication with point-of-care echocardiography during cardiac arrest: a systematic review. Resuscitation. 2020;152:56-68. [PMID: 32437781] doi: 10.1016/j.resuscitation.2020.05.004
15.
Riley RD, Ridley G, Williams K, et al. Prognosis research: toward evidence-based results and a Cochrane methods group [Letter]. J Clin Epidemiol. 2007;60:863-5; author reply 865-6. [PMID: 17606185]
16.
Moons KG, Altman DG, Vergouwe Y, et al. Prognosis and prognostic research: application and impact of prognostic models in clinical practice. BMJ. 2009;338:b606. [PMID: 19502216] doi: 10.1136/bmj.b606
17.
McShane LM, Altman DG, Sauerbrei W, et al; Statistics Subcommittee of the NCI-EORTC Working Group on Cancer Diagnostics. Reporting Recommendations for Tumor Marker Prognostic Studies (REMARK). J Natl Cancer Inst. 2005;97:1180-4. [PMID: 16106022]
18.
Tin Tin S, Woodward A, Ameratunga S. Estimating bias from loss to follow-up in a prospective cohort study of bicycle crash injuries. Inj Prev. 2014;20:322-9. [PMID: 24336816] doi: 10.1136/injuryprev-2013-040997
19.
Stukel TA, Fisher ES, Wennberg DE, et al. Analysis of observational studies in the presence of treatment selection bias: effects of invasive cardiac management on AMI survival using propensity score and instrumental variable methods. JAMA. 2007;297:278-85. [PMID: 17227979]
20.
Altman DG, Bland JM. Missing data. BMJ. 2007;334:424. [PMID: 17322261]
21.
Mack C, Su Z, Westreich D. Managing missing data in patient registries: addendum to Registries for Evaluating Patient Outcomes: A User's Guide, Third Edition. Agency for Healthcare Research and Quality; 2018. AHRQ publication no. 17(18)-EHC015-EF.
22.
Azur MJ, Stuart EA, Frangakis C, et al. Multiple imputation by chained equations: what is it and how does it work. Int J Methods Psychiatr Res. 2011;20:40-9. [PMID: 21499542] doi: 10.1002/mpr.329
23.
Shinkins B, Thompson M, Mallett S, et al. Diagnostic accuracy studies: how to report and analyse inconclusive test results. BMJ. 2013;346:f2778. [PMID: 23682043] doi: 10.1136/bmj.f2778
24.
Leung KM, Elashoff RM, Afifi AA. Censoring issues in survival analysis. Annu Rev Public Health. 1997;18:83-104. [PMID: 9143713]
25.
Altman DG, Bland JM. Time to event (survival) data. BMJ. 1998;317:468-9. [PMID: 9703534]
26.
Kamarudin AN, Cox T, Kolamunnage-Dona R. Time-dependent ROC curve analysis in medical research: current methods and applications. BMC Med Res Methodol. 2017;17:53. [PMID: 28388943] doi: 10.1186/s12874-017-0332-6
27.
Bansal A, Heagerty PJ. A comparison of landmark methods and time-dependent ROC methods to evaluate the time-varying performance of prognostic markers for survival outcomes. Diagn Progn Res. 2019;3:14. [PMID: 31367681] doi: 10.1186/s41512-019-0057-6
28.
Austin PC, Lee DS, Fine JP. Introduction to the analysis of survival data in the presence of competing risks. Circulation. 2016;133:601-9. [PMID: 26858290] doi: 10.1161/CIRCULATIONAHA.115.017719
29.
Wolbers M, Koller MT, Witteman JC, et al. Prognostic models with competing risks: methods and application to coronary risk prediction. Epidemiology. 2009;20:555-61. [PMID: 19367167] doi: 10.1097/EDE.0b013e3181a39056
30.
Andersen PK, Geskus RB, de Witte T, et al. Competing risks in epidemiology: possibilities and pitfalls. Int J Epidemiol. 2012;41:861-70. [PMID: 22253319] doi: 10.1093/ije/dyr213
31.
Verduijn M, Grootendorst DC, Dekker FW, et al. The analysis of competing events like cause-specific mortality—beware of the Kaplan-Meier method. Nephrol Dial Transplant. 2011;26:56-61. [PMID: 21059831] doi: 10.1093/ndt/gfq661
32.
Fine JP, Gray RJ. A proportional hazards model for the subdistribution of a competing risk. J Am Stat Assoc. 1999;94:496-509. doi: 10.1080/01621459.1999.10474144
33.
Noordzij M, Leffondré K, van Stralen KJ, et al. When do we need competing risks methods for survival analysis in nephrology. Nephrol Dial Transplant. 2013;28:2670-7. [PMID: 23975843] doi: 10.1093/ndt/gft355
34.
Vilar-Gomez E, Calzadilla-Bertot L, Friedman SL, et al. Serum biomarkers can predict a change in liver fibrosis 1 year after lifestyle intervention for biopsy-proven NASH. Liver Int. 2017;37:1887-1896. [PMID: 28544769] doi: 10.1111/liv.13480
35.
Siddiqui MS, Yamada G, Vuppalanchi R, et al; NASH Clinical Research Network. Diagnostic accuracy of noninvasive fibrosis models to detect change in fibrosis stage. Clin Gastroenterol Hepatol. 2019;17:1877-1885.e5. [PMID: 30616027] doi: 10.1016/j.cgh.2018.12.031
36.
McPherson S, Hardy T, Henderson E, et al. Evidence of NAFLD progression from steatosis to fibrosing-steatohepatitis using paired biopsies: implications for prognosis and clinical management. J Hepatol. 2015;62:1148-55. [PMID: 25477264] doi: 10.1016/j.jhep.2014.11.034
37.
Chalasani N, Abdelmalek MF, Loomba R, et al. Relationship between three commonly used non-invasive fibrosis biomarkers and improvement in fibrosis stage in patients with non-alcoholic steatohepatitis. Liver Int. 2019;39:924-932. [PMID: 30253043] doi: 10.1111/liv.13974
38.
Angulo P, Bugianesi E, Bjornsson ES, et al. Simple noninvasive systems predict long-term outcomes of patients with nonalcoholic fatty liver disease. Gastroenterology. 2013;145:782-9.e4. [PMID: 23860502] doi: 10.1053/j.gastro.2013.06.057
39.
Boursier J, Vergniol J, Guillet A, et al. Diagnostic accuracy and prognostic significance of blood fibrosis tests and liver stiffness measurement by FibroScan in non-alcoholic fatty liver disease. J Hepatol. 2016;65:570-8. [PMID: 27151181] doi: 10.1016/j.jhep.2016.04.023
40.
Hagström H, Nasr P, Ekstedt M, et al. Accuracy of noninvasive scoring systems in assessing risk of death and liver-related endpoints in patients with nonalcoholic fatty liver disease. Clin Gastroenterol Hepatol. 2019;17:1148-1156.e4. [PMID: 30471458] doi: 10.1016/j.cgh.2018.11.030
41.
Önnerhag K, Hartman H, Nilsson PM, et al. Non-invasive fibrosis scoring systems can predict future metabolic complications and overall mortality in non-alcoholic fatty liver disease (NAFLD). Scand J Gastroenterol. 2019;54:328-334. [PMID: 30907181] doi: 10.1080/00365521.2019.1583366
42.
Peleg N, Sneh Arbib O, Issachar A, et al. Noninvasive scoring systems predict hepatic and extra-hepatic cancers in patients with nonalcoholic fatty liver disease. PLoS One. 2018;13:e0202393. [PMID: 30106985] doi: 10.1371/journal.pone.0202393
43.
Xun YH, Guo JC, Lou GQ, et al. Non-alcoholic fatty liver disease (NAFLD) fibrosis score predicts 6.6-year overall mortality of Chinese patients with NAFLD. Clin Exp Pharmacol Physiol. 2014;41:643-9. [PMID: 24837195] doi: 10.1111/1440-1681.12260
44.
Lee J, Vali Y, Boursier J, et al. Prognostic accuracy of FIB-4, NAFLD fibrosis score and APRI for NAFLD-related events: a systematic review. Liver Int. 2021;41:261-270. [PMID: 32946642] doi: 10.1111/liv.14669
45.
Altman DG, McShane LM, Sauerbrei W, et al. Reporting Recommendations for Tumor Marker Prognostic Studies (REMARK): explanation and elaboration. BMC Med. 2012;10:51. [PMID: 22642691] doi: 10.1186/1741-7015-10-51
46.
Moons KG, Altman DG, Reitsma JB, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med. 2015;162:W1-73. [PMID: 25560730] doi: 10.7326/M14-0698
47.
Bossuyt PM, Reitsma JB, Bruns DE, et al; STARD Group. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. Clin Chem. 2015;61:1446-52. [PMID: 26510957] doi: 10.1373/clinchem.2015.246280
48.
Yang B, Mallett S, Takwoingi Y, et al; QUADAS-C Group. QUADAS-C: a tool for assessing risk of bias in comparative diagnostic accuracy studies. Ann Intern Med. 2021;174:1592-1599. [PMID: 34698503] doi: 10.7326/M21-2234

Information & Authors

Information

Published In

cover image Annals of Internal Medicine
Annals of Internal Medicine
Volume 175Number 7July 2022
Pages: 1010 - 1018

History

Published online: 14 June 2022
Published in issue: July 2022

Keywords

Authors

Affiliations

Department of Epidemiology and Data Science, Amsterdam University Medical Center, University of Amsterdam, Amsterdam, the Netherlands (J.L., M.L., P.M.B.)
Frits Mulder, MD
Department of Vascular Medicine, Cardiovascular Sciences, Amsterdam University Medical Center, University of Amsterdam, Amsterdam, the Netherlands (F.M.)
Mariska Leeflang, DVM, PhD https://orcid.org/0000-0001-5960-0471
Department of Epidemiology and Data Science, Amsterdam University Medical Center, University of Amsterdam, Amsterdam, the Netherlands (J.L., M.L., P.M.B.)
Kleijnen Systematic Reviews, Escrick, United Kingdom (R.W.)
Bristol Medical School, University of Bristol, Bristol, United Kingdom (P.W.).
Department of Epidemiology and Data Science, Amsterdam University Medical Center, University of Amsterdam, Amsterdam, the Netherlands (J.L., M.L., P.M.B.)
Acknowledgment: The authors thank Jill Hayden, PhD (Dalhousie University, Faculty of Medicine), for her valuable advice in the earlier stages of planning the development of QUAPAS. They further thank the volunteers for reviewing the tool.
Corresponding Author: Jenny Lee, MSc, Epidemiology and Data Science, Amsterdam UMC, Location AMC, Meibergdreef 9, 1105AZ Amsterdam, the Netherlands; e-mail, [email protected].
Author Contributions: Conception and design: P.M. Bossuyt, J. Lee, M. Leeflang.
Analysis and interpretation of the data: P.M. Bossuyt, J. Lee, M. Leeflang, F. Mulder, R. Wolff.
Drafting of the article: J. Lee.
Critical revision for important intellectual content: P.M. Bossuyt, J. Lee, M. Leeflang, F. Mulder, P. Whiting, R. Wolff.
Final approval of the article: P.M. Bossuyt, J. Lee, M. Leeflang, F. Mulder, P. Whiting, R. Wolff.
Statistical expertise: P.M. Bossuyt, J. Lee, F. Mulder.
Obtaining of funding: P.M. Bossuyt.
Administrative, technical, or logistic support: J. Lee.
Collection and assembly of data: J. Lee, F. Mulder.
This article was published at Annals.org on 14 June 2022.

Metrics & Citations

Metrics

Citations

If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. For an editable text file, please select Medlars format which will download as a .txt file. Simply select your manager software from the list below and click Download.

For more information or tips please see 'Downloading to a citation manager' in the Help menu.

Format





Download article citation data for:
Jenny Lee, Frits Mulder, Mariska Leeflang, et al. QUAPAS: An Adaptation of the QUADAS-2 Tool to Assess Prognostic Accuracy Studies. Ann Intern Med.2022;175:1010-1018. [Epub 14 June 2022]. doi:10.7326/M22-0276

View More

Get Access

Login Options:
Purchase

You will be redirected to acponline.org to sign-in to Annals to complete your purchase.

Access to EPUBs and PDFs for FREE Annals content requires users to be registered and logged in. A subscription is not required. You can create a free account below or from the following link. You will be redirected to acponline.org to create an account that will provide access to Annals. If you are accessing the Free Annals content via your institution's access, registration is not required.

Create your Free Account

You will be redirected to acponline.org to create an account that will provide access to Annals.

View options

PDF/ePub

View PDF/ePub

Related in ACP Journals

Full Text

View Full Text

Media

Figures

Other

Tables

Share

Share

Copy the content Link

Share on social media