Advanced searchCitation search
Skip main navigation
Close Drawer MenuOpen Drawer Menu
Research and Reporting Methods2 April 2013

Empirical Evidence of the Importance of Comparative Studies of Diagnostic Test Accuracy

    Author, Article, and Disclosure Information
    https://doi.org/10.7326/0003-4819-158-7-201304020-00006
    Background:

    Systematic reviews that “compare” the accuracy of 2 or more tests often include different sets of studies for each test.

    Purpose:

    To investigate the availability of direct comparative studies of test accuracy and to assess whether summary estimates of accuracy differ between meta-analyses of noncomparative and comparative studies.

    Data Sources:

    Systematic reviews in any language from the Database of Abstracts of Reviews of Effects and the Cochrane Database of Systematic Reviews from 1994 to October 2012.

    Study Selection:

    1 of 2 assessors selected reviews that evaluated at least 2 tests and identified meta-analyses that included both noncomparative studies and comparative studies.

    Data Extraction:

    1 of 3 assessors extracted data about review and study characteristics and test performance.

    Data Synthesis:

    248 reviews compared test accuracy; of the 6915 studies, 2113 (31%) were comparative. Thirty-six reviews (with 52 meta-analyses) had adequate studies to compare results of noncomparative and comparative studies by using a hierarchical summary receiver-operating characteristic meta-regression model for each test comparison. In 10 meta-analyses, noncomparative studies ranked tests in the opposite order of comparative studies. A total of 25 meta-analyses showed more than a 2-fold discrepancy in the relative diagnostic odds ratio between noncomparative and comparative studies. Differences in accuracy estimates between noncomparative and comparative studies were greater than expected by chance (P < 0.001).

    Limitation:

    A paucity of comparative studies limited exploration of direction in bias.

    Conclusion:

    Evidence derived from noncomparative studies often differs from that derived from comparative studies. Robustly designed studies in which all patients receive all tests or are randomly assigned to receive one or other of the tests should be more routinely undertaken and are preferred for evidence to guide test selection.

    Primary Funding Source:

    National Institute for Health Research (United Kingdom).

    References

    • 1. Concato J, Shah N, Horwitz RI. Randomized, controlled trials, observational studies, and the hierarchy of research designs. N Engl J Med. 2000;342:1887-92. [PMID: 10861325] CrossrefMedlineGoogle Scholar
    • 2. Ioannidis JP, Haidich AB, Pappa M, Pantazis N, Kokori SI, Tektonidou MG, et al. Comparison of evidence of treatment effects in randomized and nonrandomized studies. JAMA. 2001;286:821-30. [PMID: 11497536] CrossrefMedlineGoogle Scholar
    • 3. Deeks JJ, Dinnes J, D'Amico R, Sowden AJ, Sakarovitch C, Song F, et al. Evaluating non-randomised intervention studies. Health Technol Assess. 2003;7:1-173. [PMID: 14499048] CrossrefMedlineGoogle Scholar
    • 4. Reeves BC, Deeks JJ, Higgins JPT, Wells GA. Including nonrandomized studies. In: Higgins JPT, Green S, eds. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (Updated March 2011). The Cochrane Collaboration, 2011. Accessed at www.cochrane-handbook.org on 1 December 2011. Google Scholar
    • 5. Leeflang MM, Deeks JJ, Gatsonis C, Bossuyt PM; Cochrane Diagnostic Test Accuracy Working Group. Systematic reviews of diagnostic test accuracy. Ann Intern Med. 2008;149:889-97. [PMID: 19075208] LinkGoogle Scholar
    • 6. Bossuyt PM, Irwig L, Craig J, Glasziou P. Comparative accuracy: assessing new tests against existing diagnostic pathways. BMJ. 2006;332:1089-92. [PMID: 16675820] CrossrefMedlineGoogle Scholar
    • 7. Lumley T. Network meta-analysis for indirect treatment comparisons. Stat Med. 2002;21:2313-24. [PMID: 12210616] CrossrefMedlineGoogle Scholar
    • 8. Sutton A, Ades AE, Cooper N, Abrams K. Use of indirect and mixed treatment comparisons for technology assessment. Pharmacoeconomics. 2008;26:753-67. [PMID: 18767896] CrossrefMedlineGoogle Scholar
    • 9. Glenny AM, Altman DG, Song F, Sakarovitch C, Deeks JJ, D'Amico R, et al; International Stroke Trial Collaborative Group. Indirect comparisons of competing interventions. Health Technol Assess. 2005;9:1-134. [PMID: 16014203] CrossrefMedlineGoogle Scholar
    • 10. Dinnes J, Deeks J, Kirby J, Roderick P. A methodological review of how heterogeneity has been examined in systematic reviews of diagnostic test accuracy. Health Technol Assess. 2005;9:1-113. [PMID: 15774235] CrossrefMedlineGoogle Scholar
    • 11. Whiting P, Rutjes AW, Dinnes J, Reitsma J, Bossuyt PM, Kleijnen J. Development and validation of methods for assessing the quality of diagnostic accuracy studies. Health Technol Assess. 2004;8:1-234. [PMID: 15193208] CrossrefMedlineGoogle Scholar
    • 12. Rutter CM, Gatsonis CA. A hierarchical regression approach to meta-analysis of diagnostic test accuracy evaluations. Stat Med. 2001;20:2865-84. [PMID: 11568945] CrossrefMedlineGoogle Scholar
    • 13. Macaskill P. Empirical Bayes estimates generated in a hierarchical summary ROC analysis agreed closely with those of a full Bayesian analysis. J Clin Epidemiol. 2004;57:925-32. [PMID: 15504635] CrossrefMedlineGoogle Scholar
    • 14. Macaskill P, Gatsonis C, Deeks JJ, Harbord RM, Takwoingi Y. Chapter 10: Analysing and presenting results. In: Deeks JJ, Bossuyt PM, Gatsonis C, eds. Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy Version 1.0. The Cochrane Collaboration; 2010. Accessed at http://srdta.cochrane.org/ on 1 December 2011. Google Scholar
    • 15. Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L. Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry. J Clin Epidemiol. 2008;61:991-6. [PMID: 18538991] CrossrefMedlineGoogle Scholar
    • 16. Abba K, Deeks JJ, Olliaro P, Naing CM, Jackson SM, Takwoingi Y, et al. Rapid diagnostic tests for diagnosing uncomplicated P. falciparum malaria in endemic countries. Cochrane Database Syst Rev. 2011;:CD008122. [PMID: 21735422] MedlineGoogle Scholar
    • 17. Al-Khayal KA, Al-Omran MA. Computed tomography and ultrasonography in the diagnosis of equivocal acute appendicitis. A meta-analysis. Saudi Med J. 2007;28:173-80. [PMID: 17268692] MedlineGoogle Scholar
    • 18. Alldred SK, Deeks JJ, Guo B, Neilson JP, Alfirevic Z. Second trimester serum tests for Down's Syndrome screening. Cochrane Database Syst Rev. 2012;6:CD009925. [PMID: 22696388] MedlineGoogle Scholar
    • 19. Balk EM, Ioannidis JP, Salem D, Chew PW, Lau J. Accuracy of biomarkers to diagnose acute cardiac ischemia in the emergency department: a meta-analysis. Ann Emerg Med. 2001;37:478-94. [PMID: 11326184] CrossrefMedlineGoogle Scholar
    • 20. Choi HJ, Ju W, Myung SK, Kim Y. Diagnostic performance of computer tomography, magnetic resonance imaging, and positron emission tomography or positron emission tomography/computer tomography for detection of metastatic lymph nodes in patients with cervical cancer: meta-analysis. Cancer Sci. 2010;101:1471-9. [PMID: 20298252] CrossrefMedlineGoogle Scholar
    • 21. de Bondt RB, Nelemans PJ, Hofman PA, Casselman JW, Kremer B, van Engelshoven JM, et al. Detection of lymph node metastases in head and neck cancer: a meta-analysis comparing US, USgFNAC, CT and MR imaging. Eur J Radiol. 2007;64:266-72. [PMID: 17391885] CrossrefMedlineGoogle Scholar
    • 22. Elamin MB, Murad MH, Mullan R, Erickson D, Harris K, Nadeem S, et al. Accuracy of diagnostic tests for Cushing's syndrome: a systematic review and metaanalyses. J Clin Endocrinol Metab. 2008;93:1553-62. [PMID: 18334594] CrossrefMedlineGoogle Scholar
    • 23. Fleischmann KE, Hunink MG, Kuntz KM, Douglas PS. Exercise echocardiography or exercise SPECT imaging? A meta-analysis of diagnostic test performance. JAMA. 1998;280:913-20. [PMID: 9739977] CrossrefMedlineGoogle Scholar
    • 24. Gisbert JP, Abraira V. Accuracy of Helicobacter pylori diagnostic tests in patients with bleeding peptic ulcer: a systematic review and meta-analysis. Am J Gastroenterol. 2006;101:848-63. [PMID: 16494583] CrossrefMedlineGoogle Scholar
    • 25. Glas AS, Roos D, Deutekom M, Zwinderman AH, Bossuyt PM, Kurth KH. Tumor markers in the diagnosis of primary bladder cancer. A systematic review. J Urol. 2003;169:1975-82. [PMID: 12771702] CrossrefMedlineGoogle Scholar
    • 26. Gu P, Huang G, Chen Y, Zhu C, Yuan J, Sheng S. Diagnostic utility of pleural fluid carcinoembryonic antigen and CYFRA 21-1 in patients with pleural effusion: a systematic review and meta-analysis. J Clin Lab Anal. 2007;21:398-405. [PMID: 18022924] CrossrefMedlineGoogle Scholar
    • 27. Gu P, Pan LL, Wu SQ, Sun L, Huang G. CA 125, PET alone, PET-CT, CT and MRI in diagnosing recurrent ovarian carcinoma: a systematic review and meta-analysis. Eur J Radiol. 2009;71:164-74. [PMID: 18378417] CrossrefMedlineGoogle Scholar
    • 28. Heim SW, Schectman JM, Siadaty MS, Philbrick JT. D-dimer testing for deep venous thrombosis: a meta-analysis. Clin Chem. 2004;50:1136-47. [PMID: 15142977] CrossrefMedlineGoogle Scholar
    • 29. Mahajan N, Polavaram L, Vankayala H, Ference B, Wang Y, Ager J, et al. Diagnostic accuracy of myocardial perfusion imaging and stress echocardiography for the diagnosis of left main and triple vessel coronary artery disease: a comparative meta-analysis. Heart. 2010;96:956-66. [PMID: 20538671] CrossrefMedlineGoogle Scholar
    • 30. Mant J, Doust J, Roalfe A, Barton P, Cowie MR, Glasziou P, et al. Systematic review and individual patient data meta-analysis of diagnosis of heart failure, with modelling of implications of different diagnostic strategies in primary care. Health Technol Assess. 2009;13:1-207. [PMID: 19586584] CrossrefMedlineGoogle Scholar
    • 31. Mirza TA, Karthikesalingam A, Jackson D, Walsh SR, Holt PJ, Hayes PD, et al. Duplex ultrasound and contrast-enhanced ultrasound versus computed tomography for the detection of endoleak after EVAR: systematic review and bivariate meta-analysis. Eur J Vasc Endovasc Surg. 2010;39:418-28. [PMID: 20122853] CrossrefMedlineGoogle Scholar
    • 32. Mitchell AJ, Bird V, Rizzo M, Meader N. Diagnostic validity and added value of the Geriatric Depression Scale for depression in primary care: a meta-analysis of GDS30 and GDS15. J Affect Disord. 2010;125:10-7. [PMID: 19800132] CrossrefMedlineGoogle Scholar
    • 33. Mowatt G, Zhu S, Kilonzo M, Boachie C, Fraser C, Griffiths TR, et al. Systematic review of the clinical effectiveness and cost-effectiveness of photodynamic diagnosis and urine biomarkers (FISH, ImmunoCyt, NMP22) and cytology for the detection and follow-up of bladder cancer. Health Technol Assess. 2010;14:1-331. [PMID: 20082749] CrossrefMedlineGoogle Scholar
    • 34. Ngamruengphong S, Sharma VK, Nguyen B, Das A. Assessment of response to neoadjuvant therapy in esophageal cancer: an updated systematic review of diagnostic accuracy of endoscopic ultrasonography and fluorodeoxyglucose positron emission tomography. Dis Esophagus. 2010;23:216-31. [PMID: 19515185] CrossrefMedlineGoogle Scholar
    • 35. Niekel MC, Bipat S, Stoker J. Diagnostic imaging of colorectal liver metastases with CT, MR imaging, FDG PET, and/or FDG PET/CT: a meta-analysis of prospective studies including patients who have not previously undergone treatment. Radiology. 2010;257:674-84. [PMID: 20829538] CrossrefMedlineGoogle Scholar
    • 36. Nishimura K, Sugiyama D, Kogata Y, Tsuji G, Nakazawa T, Kawano S, et al. Meta-analysis: diagnostic accuracy of anti-cyclic citrullinated peptide antibody and rheumatoid factor for rheumatoid arthritis. Ann Intern Med. 2007;146:797-808. [PMID: 17548411] LinkGoogle Scholar
    • 37. Noguchi Y, Nagata-Kobayashi S, Stahl JE, Wong JB. A meta-analytic comparison of echocardiographic stressors. Int J Cardiovasc Imaging. 2005;21:189-207. [PMID: 16015428] CrossrefMedlineGoogle Scholar
    • 38. Safdar N, Fine JP, Maki DG. Meta-analysis: methods for diagnosing intravascular device-related bloodstream infection. Ann Intern Med. 2005;142:451-66. [PMID: 15767623] LinkGoogle Scholar
    • 39. Scheidler J, Hricak H, Yu KK, Subak L, Segal MR. Radiological evaluation of lymph node metastases in patients with cervical cancer. A meta-analysis. JAMA. 1997;278:1096-101. [PMID: 9315770] CrossrefMedlineGoogle Scholar
    • 40. Scherer K, Bedlack RS, Simel DL. Does this patient have myasthenia gravis? JAMA. 2005;293:1906-14. [PMID: 15840866] CrossrefMedlineGoogle Scholar
    • 41. Schuetz GM, Zacharopoulou NM, Schlattmann P, Dewey M. Meta-analysis: noninvasive coronary angiography using computed tomography versus magnetic resonance imaging. Ann Intern Med. 2010;152:167-77. [PMID: 20124233] LinkGoogle Scholar
    • 42. Selman TJ, Luesley DM, Acheson N, Khan KS, Mann CH. A systematic review of the accuracy of diagnostic tests for inguinal lymph node status in vulvar cancer. Gynecol Oncol. 2005;99:206-14. [PMID: 16081147] CrossrefMedlineGoogle Scholar
    • 43. Shiga T, Wajima Z, Apfel CC, Inoue T, Ohe Y. Diagnostic accuracy of transesophageal echocardiography, helical computed tomography, and magnetic resonance imaging for suspected thoracic aortic dissection: systematic review and meta-analysis. Arch Intern Med. 2006;166:1350-6. [PMID: 16831999] CrossrefMedlineGoogle Scholar
    • 44. Smith TO, Hilton G, Toms AP, Donell ST, Hing CB. The diagnostic accuracy of acetabular labral tears using magnetic resonance imaging and magnetic resonance arthrography: a meta-analysis. Eur Radiol. 2011;21:863-74. [PMID: 20859632] CrossrefMedlineGoogle Scholar
    • 45. Smith-Bindman R, Hosmer W, Feldstein VA, Deeks JJ, Goldberg JD. Second-trimester ultrasound to detect fetuses with Down syndrome: a meta-analysis. JAMA. 2001;285:1044-55. [PMID: 11209176] CrossrefMedlineGoogle Scholar
    • 46. St. John A, Boyd JC, Lowes AJ, Price CP. The use of urinary dipstick tests to exclude urinary tract infection: a systematic review of the literature. Am J Clin Pathol. 2006;126:428-36. [PMID: 16880133] CrossrefMedlineGoogle Scholar
    • 47. Terasawa T, Blackmore CC, Bent S, Kohlwes RJ. Systematic review: computed tomography and ultrasonography to detect acute appendicitis in adults and adolescents. Ann Intern Med. 2004;141:537-46. [PMID: 15466771] LinkGoogle Scholar
    • 48. Tian XY, Zhu H, Zhao J, She Q, Zhang GX. Diagnostic performance of urea breath test, rapid urea test, and histology for Helicobacter pylori infection in patients with partial gastrectomy: a meta-analysis. J Clin Gastroenterol. 2012;46:285-92. [PMID: 22392025] CrossrefMedlineGoogle Scholar
    • 49. Toloza EM, Harpole L, McCrory DC. Noninvasive staging of non-small cell lung cancer: a review of the current evidence. Chest. 2003;123:137S-146S. [PMID: 12527573] CrossrefMedlineGoogle Scholar
    • 50. van der Windt DA, Jellema P, Mulder CJ, Kneepkens CM, van der Horst HE. Diagnostic testing for celiac disease among patients with abdominal symptoms: a systematic review. JAMA. 2010;303:1738-46. [PMID: 20442390] CrossrefMedlineGoogle Scholar
    • 51. van Vliet EP, Heijenbrok-Kal MH, Hunink MG, Kuipers EJ, Siersema PD. Staging investigations for oesophageal cancer: a meta-analysis. Br J Cancer. 2008;98:547-57. [PMID: 18212745] CrossrefMedlineGoogle Scholar
    • 52. Wardlaw JM, Chappell FM, Best JJ, Wartolowska K, Berry E; NHS Research and Development Health Technology Assessment Carotid Stenosis Imaging Group. Non-invasive imaging compared with intra-arterial angiography in the diagnosis of symptomatic carotid stenosis: a meta-analysis. Lancet. 2006;367:1503-12. [PMID: 16679163] CrossrefMedlineGoogle Scholar
    • 53. Xu GZ, Zhu XD, Li MY. Accuracy of whole-body PET and PET-CT in initial M staging of head and neck cancer: a meta-analysis. Head Neck. 2011;33:87-94. [PMID: 20848421] CrossrefMedlineGoogle Scholar
    • 54. Yin ZG, Zhang JB, Kan SL, Wang XG. Diagnosing suspected scaphoid fractures: a systematic review and meta-analysis. Clin Orthop Relat Res. 2010;468:723-34. [PMID: 19756904] CrossrefMedlineGoogle Scholar
    • 55. Sheps SB, Schechter MT. The assessment of diagnostic tests. A survey of current medical research. JAMA. 1984;252:2418-22. [PMID: 6481928] CrossrefMedlineGoogle Scholar
    • 56. Song F, Altman DG, Glenny AM, Deeks JJ. Validity of indirect comparison for estimating efficacy of competing interventions: empirical evidence from published meta-analyses. BMJ. 2003;326:472. [PMID: 12609941] CrossrefMedlineGoogle Scholar
    • 57. Lijmer JG, Mol BW, Heisterkamp S, Bonsel GJ, Prins MH, van der Meulen JH, et al. Empirical evidence of design-related bias in studies of diagnostic tests. JAMA. 1999;282:1061-6. [PMID: 10493205] CrossrefMedlineGoogle Scholar
    • 58. Whiting P, Rutjes AW, Reitsma JB, Glas AS, Bossuyt PM, Kleijnen J. Sources of variation and bias in studies of diagnostic accuracy: a systematic review. Ann Intern Med. 2004;140:189-202. [PMID: 14757617] LinkGoogle Scholar
    • 59. Rutjes AW, Reitsma JB, Di Nisio M, Smidt N, van Rijn JC, Bossuyt PM. Evidence of bias and variation in diagnostic accuracy studies. CMAJ. 2006;174:469-76. [PMID: 16477057] CrossrefMedlineGoogle Scholar
    • 60. Leeflang M, Reitsma J, Scholten R, Rutjes A, Di Nisio M, Deeks J, et al. Impact of adjustment for quality on results of metaanalyses of diagnostic accuracy. Clin Chem. 2007;53:164-72. [PMID: 17185365] CrossrefMedlineGoogle Scholar
    • 61. Medical Services Advisory Committee. Guidelines for the assessment of diagnostic technologies. Australian Government Department of Health and Aging; 2005. Accessed at www.msac.gov.au/internet/msac/publishing.nsf/Content/guidelines-1 on 3 June 2012. Google Scholar
    • 62. Dahabreh IJ, Chung M, Kitsios GD, Terasawa T, Raman G, Tatsioni A, et al. Comprehensive overview of methods and reporting of meta-analyses of test accuracy. AHRQ publication no. 12-EHC044-EF. Rockville, MD: Agency for Healthcare Research and Quality; 2012. Google Scholar
    Previous
    Next

    Submit a Comment

    Contributors must reveal any conflict of interest. Comments are moderated. Please see our information for authorsregarding comments on an Annals publication.
    *All comments submitted after October 1, 2021 and selected for publication will be published online only.
    Cancel