Research and Reporting Methods

19 February 2013

Assessing Bias in Studies of Prognostic Factors

Authors: Jill A. Hayden, DC, PhD, Danielle A. van der Windt, PhD, Jennifer L. Cartwright, MSc, Pierre Côté, DC, PhD, and Claire Bombardier, MDAuthor, Article, & Disclosure Information

Publication: Annals of Internal Medicine

Volume 158, Number 4

Abstract

Previous work has identified 6 important areas to consider when evaluating validity and bias in studies of prognostic factors: participation, attrition, prognostic factor measurement, confounding measurement and account, outcome measurement, and analysis and reporting. This article describes the Quality In Prognosis Studies tool, which includes questions related to these areas that can inform judgments of risk of bias in prognostic research.

A working group comprising epidemiologists, statisticians, and clinicians developed the tool as they considered prognosis studies of low back pain. Forty-three groups reviewing studies addressing prognosis in other topic areas used the tool and provided feedback. Most reviewers (74%) reported that reaching consensus on judgments was easy. Median completion time per study was 20 minutes; interrater agreement (κ statistic) reported by 9 review teams varied from 0.56 to 0.82 (median, 0.75). Some reviewers reported challenges making judgments across prompting items, which were addressed by providing comprehensive guidance and examples. The refined Quality In Prognosis Studies tool may be useful to assess the risk of bias in studies of prognostic factors.

Get full access to this article

View all available purchase options and get full access to this article.

Supplemental Material

Supplement. QUIPS Risk of Bias Assessment Instrument for Prognostic Factor Studies

References

1.

Hayden JA, Côté P, Bombardier C. Evaluation of the quality of prognosis studies in systematic reviews. Ann Intern Med. 2006;144:427-37. [PMID: 16549855]

2.

Hemingway H, Riley RD, Altman DG. Ten steps towards improving prognosis research. BMJ. 2009;339:b4184. [PMID: 20042483]

3.

Hayden JA, Chou R, Hogg-Johnson S, Bombardier C. Systematic reviews of low back pain prognosis had variable methods and results: guidance for future prognosis reviews. J Clin Epidemiol. 2009;62:781-796.e1. [PMID: 19136234]

4.

Riley RD, Sauerbrei W, Altman DG. Prognostic markers in cancer: the evolution of evidence from single studies to meta-analysis, and beyond. Br J Cancer. 2009;100:1219-29. [PMID: 19367280]

5.

Kamper SJ, Hancock MJ, Maher CG. Optimal designs for prediction studies of whiplash. Spine (Phila Pa 1976). 2011;36:S268-74. [PMID: 22020594]

6.

Hemingway H, Philipson P, Chen R, Fitzpatrick NK, Damant J, Shipley M, et al. Evaluating the quality of research into a single prognostic biomarker: a systematic review and meta-analysis of 83 studies of C-reactive protein in stable coronary artery disease. PLoS Med. 2010;7:1000286. [PMID: 20532236]

7.

Hayden JA, Côté P, Steenstra IA, Bombardier C; QUIPS-LBP Working Group. Identifying phases of investigation helps planning, appraising, and applying the results of explanatory prognosis studies. J Clin Epidemiol. 2008;61:552-60. [PMID: 18471659]

8.

Jones J, Hunter D. Consensus methods for medical and health services research. BMJ. 1995;311:376-80. [PMID: 7640549]

9.

Hayden JA. Methodological Issues in Systematic Reviews of Prognosis and Prognostic Factors: Low Back Pain. Toronto: Univ Toronto; 2007.

10.

Chapple CM, Nicholson H, Baxter GD, Abbott JH. Patient characteristics that predict progression of knee osteoarthritis: a systematic review of prognostic studies. Arthritis Care Res (Hoboken). 2011;63:1115-25. [PMID: 21560257]

11.

Pickett CA, Jackson JL, Hemann BA, Atwood JE. Carotid bruits as a prognostic indicator of cardiovascular death and myocardial infarction: a meta-analysis. Lancet. 2008;371:1587-94. [PMID: 18468542]

12.

Pickett CA, Jackson JL, Hemann BA, Atwood JE. Carotid bruits and cerebrovascular disease risk: a meta-analysis. Stroke. 2010;41:2295-302. [PMID: 20724720]

13.

Palmer SC, Hayen A, Macaskill P, Pellegrini F, Craig JC, Elder GJ, et al. Serum levels of phosphorus, parathyroid hormone, and calcium and risks of death and cardiovascular disease in individuals with chronic kidney disease: a systematic review and meta-analysis. JAMA. 2011;305:1119-27. [PMID: 21406649]

14.

Mathew A, Devereaux PJ, O'Hare A, Tonelli M, Thiessen-Philbrook H, Nevis IF, et al. Chronic kidney disease and postoperative mortality: a systematic review and meta-analysis. Kidney Int. 2008;73:1069-81. [PMID: 18288098]

15.

Higgins JP, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, et al; Cochrane Bias Methods Group. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928. [PMID: 22008217]

16.

Whiting PF, Rutjes AW, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, et al; QUADAS-2 Group. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011;155:529-36. [PMID: 22007046]

17.

Singh AS, Mulder C, Twisk JW, van Mechelen W, Chinapaw MJ. Tracking of childhood overweight into adulthood: a systematic review of the literature. Obes Rev. 2008;9:474-88. [PMID: 18331423]

18.

Moulaert VR, Verbunt JA, van Heugten CM, Wade DT. Cognitive impairments in survivors of out-of-hospital cardiac arrest: a systematic review. Resuscitation. 2009;80:297-305. [PMID: 19117659]

19.

van Drongelen A, Boot CR, Merkus SL, Smid T, van der Beek AJ. The effects of shift work on body weight change—a systematic review of longitudinal studies. Scand J Work Environ Health. 2011;37:263-75. [PMID: 21243319]

20.

Clay FJ, Newstead SV, McClure RJ. A systematic review of early prognostic factors for return to work following acute orthopaedic trauma. Injury. 2010;41:787-803. [PMID: 20435304]

21.

Nijrolder I, van der Horst H, van der Windt D. Prognosis of fatigue. A systematic review. J Psychosom Res. 2008;64:335-49. [PMID: 18374732]

22.

Proper KI, Singh AS, van Mechelen W, Chinapaw MJ. Sedentary behaviors and health outcomes among adults: a systematic review of prospective studies. Am J Prev Med. 2011;40:174-82. [PMID: 21238866]

23.

Spee LA, Madderom MB, Pijpers M, van Leeuwen Y, Berger MY. Association between helicobacter pylori and gastrointestinal symptoms in children. Pediatrics. 2010;125:651-69. [PMID: 20156901]

24.

van Duijvenbode DC, Hoozemans MJ, van Poppel MN, Proper KI. The relationship between overweight and obesity, and sick leave: a systematic review. Int J Obes (Lond). 2009;33:807-16. [PMID: 19528969]

25.

Jiménez D, Uresandi F, Otero R, Lobo JL, Monreal M, Martí D, et al. Troponin-based risk stratification of patients with acute nonmassive pulmonary embolism: systematic review and metaanalysis. Chest. 2009;136:974-82. [PMID: 19465511]

26.

Wright AA, Cook C, Abbott JH. Variables associated with the progression of hip osteoarthritis: a systematic review. Arthritis Rheum. 2009;61:925-36. [PMID: 19565541]

27.

Elshout G, Monteny M, van der Wouden JC, Koes BW, Berger MY. Duration of fever and serious bacterial infections in children: a systematic review. BMC Fam Pract. 2011;12:33. [PMID: 21575193]

28.

Gieteling MJ, Bierma-Zeinstra SM, Lisman-van Leeuwen Y, Passchier J, Berger MY. Prognostic factors for persistence of chronic abdominal pain in children. J Pediatr Gastroenterol Nutr. 2011;52:154-61. [PMID: 21057328]

29.

Jeejeebhoy FM, Zelop CM, Windrim R, Carvalho JC, Dorian P, Morrison LJ. Management of cardiac arrest in pregnancy: a systematic review. Resuscitation. 2011;82:801-9. [PMID: 21549495]

30.

Johnson SR, Swiston JR, Swinton JR, Granton JT. Prognostic factors for survival in scleroderma associated pulmonary arterial hypertension. J Rheumatol. 2008;35:1584-90. [PMID: 18597400]

31.

McShane LM, Altman DG, Sauerbrei W, Taube SE, Gion M, Clark GM; Statistics Subcommittee of NCI-EORTC Working Group on Cancer Diagnostics. REporting recommendations for tumor MARKer prognostic studies (REMARK). Breast Cancer Res Treat. 2006;100:229-35. [PMID: 16932852]

32.

National Institute for Health and Clinical Excellence. Appendix J: methodology checklist: prognostic studies. The Guidelines Manual, London: National Institute for Health and Clinical Excellence; 2009:218-22.

33.

Shamliyan T, Kane RL, Jansen S. Systematic reviews synthesized evidence without consistent quality assessment of primary studies examining epidemiology of chronic diseases. J Clin Epidemiol. 2012;65:610-8. [PMID: 22424987]

34.

Sanderson S, Tatt ID, Higgins JP. Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int J Epidemiol. 2007;36:666-76. [PMID: 17470488]

35.

Hartling L, Hamm M, Milne A, Vandermeer B, Santaguida PL, Ansari M, Tsertsvadze A, et al. Validity and Inter-Rater Reliability Testing of Quality Assessment Instruments. Rockville, MD: Agency for Healthcare Research and Quality; 2012. Accessed at www.ncbi.nlm.nih.gov/books/NBK92293/ on 20 November 2012.

Comments

0 Comments

Sign In to Submit A Comment

Information & Authors

Information

Published In

cover image Annals of Internal Medicine

Annals of Internal Medicine

Volume 158 • Number 4 • 19 February 2013

Pages: 280 - 286

History

Published online: 19 February 2013

Published in issue: 19 February 2013

Keywords

Copyright

© 2013 American College of Physicians.

Authors

Affiliations

Jill A. Hayden, DC, PhD

From Dalhousie University, Halifax, Nova Scotia, Canada; Arthritis Research UK Primary Care Centre, Primary Care and Health Sciences, Keele University, Staffordshire, United Kingdom; University of Ontario Institute of Technology, Oshawa, Ontario, Canada; and University of Toronto and Institute for Work & Health, Toronto, Ontario, Canada.

View all articles by this author

Danielle A. van der Windt, PhD

From Dalhousie University, Halifax, Nova Scotia, Canada; Arthritis Research UK Primary Care Centre, Primary Care and Health Sciences, Keele University, Staffordshire, United Kingdom; University of Ontario Institute of Technology, Oshawa, Ontario, Canada; and University of Toronto and Institute for Work & Health, Toronto, Ontario, Canada.

View all articles by this author

Jennifer L. Cartwright, MSc

From Dalhousie University, Halifax, Nova Scotia, Canada; Arthritis Research UK Primary Care Centre, Primary Care and Health Sciences, Keele University, Staffordshire, United Kingdom; University of Ontario Institute of Technology, Oshawa, Ontario, Canada; and University of Toronto and Institute for Work & Health, Toronto, Ontario, Canada.

View all articles by this author

Pierre Côté, DC, PhD

From Dalhousie University, Halifax, Nova Scotia, Canada; Arthritis Research UK Primary Care Centre, Primary Care and Health Sciences, Keele University, Staffordshire, United Kingdom; University of Ontario Institute of Technology, Oshawa, Ontario, Canada; and University of Toronto and Institute for Work & Health, Toronto, Ontario, Canada.

View all articles by this author

Claire Bombardier, MD

From Dalhousie University, Halifax, Nova Scotia, Canada; Arthritis Research UK Primary Care Centre, Primary Care and Health Sciences, Keele University, Staffordshire, United Kingdom; University of Ontario Institute of Technology, Oshawa, Ontario, Canada; and University of Toronto and Institute for Work & Health, Toronto, Ontario, Canada.

View all articles by this author

Acknowledgment: The authors thank the QUIPS-Low Back Pain Working Group members (2006 and 2007) for their important contributions. They also thank the prognosis systematic review authors who completed their survey and review authors who responded to their additional questions and requests for data, including Amika Singh, James Chalmers, Roger Chou, Fiona Clay, Hanneke Creemers, Lotte Dyhrberg O'Neill, Jan Hartvigsen, Ross Iles, David Jimenez, Sindhu Johnson, Bindee Kuriya, Jolanda Luime, Veronique Moulaert, Tinca Polderman, Cara Wasywich, Stephen Wilton, Susan Woolfenden, Lexie Wright, and Christina Wyatt.

Financial Support: Dr. Hayden received infrastructure funding through the Nova Scotia Cochrane Resource Centre provided by the Nova Scotia Health Research Foundation and holds a Research Professorship in Epidemiology funded by the Canadian Chiropractic Research Foundation and Dalhousie University. Dr. van der Windt is a member of the Prognosis Research Strategy Initiative Medical Research Council, Prognosis Research Strategy Initiative Partnership (G0902393/99558).

Disclosures: Disclosures can be viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum=M12-1871.

Corresponding Author: Jill A. Hayden, DC, PhD, Department of Community Health & Epidemiology, Dalhousie University, 5790 University Avenue, Room 222, Halifax, Nova Scotia B3H 1V7, Canada; e-mail, [email protected].

Current Author Addresses: Dr. Hayden: Department of Community Health & Epidemiology, Dalhousie University, 5790 University Avenue, Room 222, Halifax, Nova Scotia B3H 1V7, Canada.

Dr. van der Windt: Arthritis Research UK Primary Care Centre, Primary Care Sciences, Keele University, Staffordshire ST5 5BG, United Kingdom.

Ms. Cartwright: Department of Community Health & Epidemiology, Dalhousie University, 5790 University Avenue, Room 228, Halifax, Nova Scotia B3H 1V7, Canada.

Dr. Côté: Faculty of Health Sciences, University of Ontario Institute of Technology, 2000 Simcoe Street North, Oshawa, Ontario L1H 7K4, Canada.

Dr. Bombardier: Toronto General Hospital, Eaton North Wing, 6th Floor, Room 231A, 200 Elizabeth Street, Toronto, Ontario M5G 2C4, Canada.

Author Contributions: Conception and design: J.A. Hayden, D.A. van der Windt, P. Côté.

Analysis and interpretation of the data: J.A. Hayden, D.A. van der Windt, J.L. Cartwright, P. Côté, C. Bombardier.

Drafting of the article: J.A. Hayden, J.L. Cartwright, P. Côté.

Critical revision of the article for important intellectual content: J.A. Hayden, D.A. van der Windt, J.L. Cartwright, P. Côté, C. Bombardier.

Final approval of the article: J.A. Hayden, D.A. van der Windt, J.L. Cartwright, P. Côté, C. Bombardier.

Collection and assembly of data: J.A. Hayden, D.A. van der Windt, J.L. Cartwright.

Metrics & Citations

Metrics

Citations

If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. For an editable text file, please select Medlars format which will download as a .txt file. Simply select your manager software from the list below and click Download.

For more information or tips please see 'Downloading to a citation manager' in the Help menu.

Format	RIS (ProCite, Reference Manager) EndNote BibTex Medlars RefWorks
Direct import
Tips for downloading citations

Download article citation data for:

Jill A. Hayden, Danielle A. van der Windt, Jennifer L. Cartwright, et al. Assessing Bias in Studies of Prognostic Factors. Ann Intern Med.2013;158:280-286. [Epub 19 February 2013]. doi:10.7326/0003-4819-158-4-201302190-00009

View More

Login Options:

Purchase

You will be redirected to acponline.org to sign-in to Annals to complete your purchase.

buy this article subscribe

Access to EPUBs and PDFs for FREE Annals content requires users to be registered and logged in. A subscription is not required. You can create a free account below or from the following link. You will be redirected to acponline.org to create an account that will provide access to Annals. If you are accessing the Free Annals content via your institution's access, registration is not required.

Create your Free Account

You will be redirected to acponline.org to create an account that will provide access to Annals.

create account learn more

View options

PDF/EPUB

Related in ACP Journals

Full Text

Figures

Tables

Media

View full text|Download PDF