Advanced searchCitation search
Skip main navigation
Close Drawer MenuOpen Drawer Menu
Research and Reporting Methods6 August 2013

Evaluating Diagnostic Accuracy in the Face of Multiple Reference Standards

    Author, Article, and Disclosure Information
    https://doi.org/10.7326/0003-4819-159-3-201308060-00009

    Abstract

    A universal challenge in studies that quantify the accuracy of diagnostic tests is establishing whether each participant has the disease of interest. Ideally, the same preferred reference standard would be used for all participants; however, for practical or ethical reasons, alternative reference standards that are often less accurate are frequently used instead. The use of different reference standards across participants in a single study is known as differential verification.

    Differential verification can cause severely biased accuracy estimates of the test or model being studied. Many variations of differential verification exist, but not all introduce the same risk of bias. A risk-of-bias assessment requires detailed information about which participants receive which reference standards and an estimate of the accuracy of the alternative reference standard. This article classifies types of differential verification and explores how they can lead to bias. It also provides guidance on how to report results and assess the risk of bias when differential verification occurs and highlights potential ways to correct for the bias.

    References

    • 1. Rutjes AW, Reitsma JB, Coomarasamy A, Khan KS, and Bossuyt PM. Evaluation of diagnostic tests when there is no gold standard. A review of methods. Health Technol Assess. 2007;11:iii, ix-51. [PMID: 18021577] CrossrefMedlineGoogle Scholar
    • 2. Knottnerus JA. The Evidence Base of Clinical Diagnosis. London BMJ 2003. Google Scholar
    • 3. de Groot JA, Bossuyt PM, Reitsma JB, Rutjes AW, Dendukuri N, Janssen KJ, et al. Verification problems in diagnostic accuracy studies: consequences and solutions. BMJ. 2011;343:d4770. [PMID: 21810869] CrossrefMedlineGoogle Scholar
    • 4. Lijmer JG, Mol BW, Heisterkamp S, Bonsel GJ, Prins MH, van der Meulen JH, et al. Empirical evidence of design-related bias in studies of diagnostic tests. JAMA. 1999;282:1061-6. [PMID: 10493205] CrossrefMedlineGoogle Scholar
    • 5. Rutjes AW, Reitsma JB, Di Nisio M, Smidt N, van Rijn JC, and Bossuyt PM. Evidence of bias and variation in diagnostic accuracy studies. CMAJ. 2006;174:469-76. [PMID: 16477057] CrossrefMedlineGoogle Scholar
    • 6. Lehman CD, Lee CI, Loving VA, Portillo MS, Peacock S, and DeMartini WB. Accuracy and value of breast ultrasound for primary imaging evaluation of symptomatic women 30-39 years of age. AJR Am J Roentgenol. 2012;199:1169-77. [PMID: 23096195] CrossrefMedlineGoogle Scholar
    • 7. Büller HR, Ten Cate-Hoek AJ, Hoes AW, Joore MA, Moons KG, Oudega R, et al. AMUSE (Amsterdam Maastricht Utrecht Study on thromboEmbolism) Investigators. Safely ruling out deep venous thrombosis in primary care. Ann Intern Med. 2009;150:229-35 LinkGoogle Scholar
    • 8. Thangaratinam S, Brown K, Zamora J, Khan KS, and Ewer AK. Pulse oximetry screening for critical congenital heart defects in asymptomatic newborn babies: a systematic review and meta-analysis. Lancet. 2012;379:2459-64. [PMID: 22554860] CrossrefMedlineGoogle Scholar
    • 9. Zuithoff NP, Vergouwe Y, King M, Nazareth I, Hak E, Moons KG, et al. A clinical prediction rule for detecting major depressive disorder in primary care: the PREDICT-NL study. Fam Pract. 2009;26:241-50. [PMID: 19546117] CrossrefMedlineGoogle Scholar
    • 10. Gupta A, Chandrasekhar A, Gupte N, Patil S, Bhosale R, Sambarey P, et al. Byramjee Jeejeebhoy Medical College-Johns Hopkins University Study Group. Symptom screening among HIV-infected pregnant women is acceptable and has high negative predictive value for active tuberculosis. Clin Infect Dis. 2011;53:1015-8. [PMID: 21940417] CrossrefMedlineGoogle Scholar
    • 11. Alonzo TA, Brinton JT, Ringham BM, and Glueck DH. Bias in estimating accuracy of a binary screening test with differential disease verification. Stat Med. 2011;30:1852-64. [PMID: 21495059] CrossrefMedlineGoogle Scholar
    • 12. Walter SD, Macaskill P, Lord SJ, and Irwig L. Effect of dependent errors in the assessment of diagnostic or screening test accuracy when the reference standard is imperfect. Stat Med. 2012;31:1129-38. [PMID: 22351623] CrossrefMedlineGoogle Scholar
    • 13. Whiting PF, Rutjes AW, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, et al. QUADAS-2 Group. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011;155:529-36 LinkGoogle Scholar
    • 14. de Groot JA, Dendukuri N, Janssen KJ, Reitsma JB, Bossuyt PM, and Moons KG. Adjusting for differential-verification bias in diagnostic-accuracy studies: a Bayesian approach. Epidemiology. 2011;22:234-41. [PMID: 21228702] CrossrefMedlineGoogle Scholar
    • 15. Gaffikin L, McGrath JA, Arbyn M, and Blumenthal PD. Visual inspection with acetic acid as a cervical cancer test: accuracy validated using latent class analysis. BMC Med Res Methodol. 2007;7:36. [PMID: 17663796] CrossrefMedlineGoogle Scholar
    • 16. Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, et al. STARD Group. Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative. Fam Pract. 2004;21:4-10. [PMID: 14760036] CrossrefMedlineGoogle Scholar
    • 17. Little RJA and Rubin DB. Statistical Analysis with Missing Data. New York Wiley-Interscience 2002. Google Scholar
    • 18. Begg CB and Greenes RA. Assessment of diagnostic tests when disease verification is subject to selection bias. Biometrics. 1983;39:207-15. [PMID: 6871349] CrossrefMedlineGoogle Scholar
    • 19. de Groot JA, Janssen KJ, Zwinderman AH, Moons KG, and Reitsma JB. Multiple imputation to correct for partial verification bias revisited. Stat Med. 2008;27:5880-9. [PMID: 18752256] CrossrefMedlineGoogle Scholar
    • 20. Differential Oxford Dictionaries Web site. Accessed at http://oxforddictionaries.com on 4 January 2013. Google Scholar
    • 21. Harel O and Zhou XH. Multiple imputation for correcting verification bias. Stat Med. 2006;25:3769-86. [PMID: 16435337] CrossrefMedlineGoogle Scholar
    • 22. Donders AR, van der Heijden GJ, Stijnen T, and Moons KG. Review: a gentle introduction to imputation of missing values. J Clin Epidemiol. 2006;59:1087-91. [PMID: 16980149] CrossrefMedlineGoogle Scholar
    • 23. Ewer AK, Furmston AT, Middleton LJ, Deeks JJ, Daniels JP, Pattison HM, et al. Pulse oximetry as a screening test for congenital heart defects in newborn infants: a test accuracy study with evaluation of acceptability and cost-effectiveness. Health Technol Assess. 2012;16:v-xiii, 1-184. [PMID: 22284744] CrossrefMedlineGoogle Scholar
    • 24. Lord SJ, Staub LP, Bossuyt PM, and Irwig LM. Target practice: choosing target conditions for test accuracy studies that are relevant to clinical practice. BMJ. 2011;343:d4684. [PMID: 21903693] CrossrefMedlineGoogle Scholar
    • 25. Whiting P, Rutjes AW, Reitsma JB, Bossuyt PM, and Kleijnen J. The development of QUADAS: a tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews. BMC Med Res Methodol. 2003;3:25. [PMID: 14606960] CrossrefMedlineGoogle Scholar
    • 26. Whiting PF, Weswood ME, Rutjes AW, Reitsma JB, Bossuyt PN, and Kleijnen J. Evaluation of QUADAS, a tool for the quality assessment of diagnostic accuracy studies. BMC Med Res Methodol. 2006;6:9. [PMID: 16519814] CrossrefMedlineGoogle Scholar
    • 27. Hayen A, Macaskill P, Irwig L, and Bossuyt P. Appropriate statistical methods are required to assess diagnostic tests for replacement, add-on, and triage. J Clin Epidemiol. 2010;63:883-91. [PMID: 20079607] CrossrefMedlineGoogle Scholar
    • 28. Sonke GS, Verbeek AL, and Kiemeney LA. A philosophical perspective supports the need for patient-outcome studies in diagnostic test evaluation. J Clin Epidemiol. 2009;62:58-61. [PMID: 18619792] CrossrefMedlineGoogle Scholar
    Previous
    Next

    Submit a Comment

    Contributors must reveal any conflict of interest. Comments are moderated. Please see our information for authorsregarding comments on an Annals publication.
    *All comments submitted after October 1, 2021 and selected for publication will be published online only.
    Cancel