The Development of Clinical Guidelines and Guidance Statements by the Clinical Guidelines Committee of the American College of Physicians: Update of Methods

The American College of Physicians (ACP) officially initiated its guideline development program in 1981 but has participated in the development of evidence-based clinical recommendations since the late 1970s, making it one of the oldest guideline programs in the United States (1). The founding principle of ACP's guideline development program is that its clinical guidelines should be based on scientific evidence. As the arena of clinical guidelines grows and matures, guideline developers face increasingly rigorous standards to ensure the production of trustworthy, high-quality, and useful products (2, 3). Red flags that are raised to question the trustworthiness of a clinical guideline include but are not limited to sponsoring by a professional society with substantial industry funding, conflicts of interest for chair and committee members, lack of or insufficient methodologist involvement, inadequate external peer review, and exclusion of public members in the development. Guideline developers are further challenged to keep up to date with the current state of the evidence because the volume of available scientific literature continues to rapidly expand.

ACP's guideline development program has evolved with advancements in the field. Enhancements to its policies and methods include more stringent disclosure of interests and management of conflicts; public involvement; use of the GRADE (Grading of Recommendations Assessment, Development and Evaluation) methods; and reporting formats that consider value of care, patient comorbid conditions, patient values and preferences, and costs. ACP develops clinical guidelines in accordance with standards set forth by the Guidelines International Network and National Academy of Medicine (previously called the Institute of Medicine), and it voluntarily completes and posts standard reporting forms with each guideline in the Guidelines International Network library and on ACP's Web site (2, 3). This article describes ACP's current methods for developing clinical guidelines and guidance statements and serves as an update to the 2010 methods paper (Table) (4). The updated methods have been included in Clinical Guidelines Committee (CGC) publications since 2018.

Table. Additions and Changes in the 2019 CGC Methods Paper Compared With the 2010 CGC Methods Paper

Guideline Development Group

ACP convenes a standing CGC that works with ACP Clinical Policy staff to develop clinical guidelines and guidance statements. ACP does not convene separate committees for each guideline topic; rather, the CGC is a standing committee that develops clinical guidelines and guidance statements on a broad range of topics relevant to internal medicine. All members of the CGC, including the chair and vice chair, are appointed by the Board of Regents (ACP's governing body) and serve a 1-year term that is renewable for a maximum of 4 years. To maintain continuity, an immediate past chair serves a 1-year term after the expiration of his or her term as chair, with no reappointment. The vice chair facilitates chairing CGC meetings and calls when the current chair cannot do so because of absence or an intellectual conflict of interest. Nomination and appointment to the CGC follow standard ACP procedures for selection of committee members by ACP, which oversees appointments to its committees. The CGC meets in person 3 times per year and works between meetings via e-mail and conference calls. Figure 1 summarizes the groups involved in the development and approval of CGC guidelines and guidance statements.

• Download figure
• Download PowerPoint

CGC Members

The CGC is a multidisciplinary group of 14 members. Twelve of these members are internal medicine physicians representing various clinical areas of expertise across hospital and ambulatory medicine, including internal medicine subspecialties (for example, geriatrics, nephrology, rheumatology, pulmonology, and hospital medicine). Several CGC members also have expertise in epidemiology, health policy, evidence synthesis, and guideline development. All physicians on the CGC must be current ACP members.

In addition, 2 nonphysician public members have equal standing to and the same terms as the physician members, including voting and authorship privileges (as of 2016). Public members of the CGC provide a layperson perspective on values and preferences. Prospective members are recruited as described for the Public Panel in the following paragraph; they are not selected on the basis of experience with a specific disease and thus do not represent specific patients for each guideline.

CGC Public Panel

The Clinical Policy staff convenes a CGC Public Panel to seek its input for all clinical guidelines and guidance statements (established in 2016). This panel consists of 7 nonphysician members with 1-year, renewable terms. Prospective participants are recruited via open calls for applications (distributed through various informal and formal patient and professional networks), and applicants are screened for conflicts of interest, access to the Internet, and ability to meet the minimum time commitment. Members of the CGC Public Panel are not selected on the basis of experience with a specific disease but provide input from a layperson, general perspective. In the panel's composition, ACP aims for diversity with regard to sociodemographic characteristics and experiences within the health care system (for example, patient vs. caregiver) and a balance of members who have previous experience or formal training as “patient representatives” with those who have little formal experience. Public Panel members cannot vote on or author CGC papers. All members attend 1 CGC meeting in person at the beginning of their term, although much of their work is done via conference calls, online surveys, and e-mail. Because incorporating the CGC Public Panel into our guideline development work is still in the early stages, our methods for soliciting patient preferences and values are continually evolving.

Clinical Policy Staff

The development of guidelines and guidance statements is a team effort between the Clinical Policy staff and the CGC, both of which contribute clinical, scientific, and methodological expertise to the process. In addition, the Clinical Policy staff provides administrative support and serves as a link among the CGC, the CGC Public Panel, evidence reviewers, and journals.

CGC Topic and Methods Subgroups

A topic subgroup of 2 to 4 CGC member volunteers, including the chair or vice chair, is assigned to each guideline or guidance statement. Topic subgroup members lead the development of the guideline or guidance statement and are the primary authors of the published manuscript. The topic subgroup works with Clinical Policy staff through conference calls and e-mails to draft the recommendations and final manuscript for review and approval by the whole CGC at the in-person meetings.

A methods subgroup consists of the CGC chair, 2 to 4 CGC members with expertise in the methodology of systematic reviews and guideline development, and Clinical Policy staff. The methods group meets regularly via conference calls and reviews and provides general oversight for CGC methods.

Training New CGC Members and CGC Public Panel Members

All new members of the CGC (both physicians and public members) and of the CGC Public Panel receive training in evidence-based medicine and CGC policies and methods via Webinars and an in-person educational session led by Clinical Policy staff and current members. The training covers the basics of evidence-based medicine, relevant statistics, member roles and responsibilities, the development processes for guidelines and guidance statements, confidentiality, media relations, and what to expect at committee meetings.

Disclosure of Interests and Management of Potential Conflicts

The CGC's policy on disclosure of interests and management of conflicts of interest is addressed in a separate article (5). In brief, the policy is stringent and thorough and emphasizes full disclosure of all health care–related interests for CGC members; CGC Public Panel members; relevant ACP staff; and any other persons involved in development of clinical guidelines, guidance statements, or evidence reviews. Participants disclose all financial and intellectual interests from the past 3 years, and an oversight panel consisting of ACP staff and the CGC chair and vice chair reviews the disclosures for conflicts of interest and determines management. The panel grades potential conflicts of interest as high, moderate, or low, and members are restricted from participation according to the conflict grade (restrictions may include recusal from authorship, voting, or discussion pertaining to recommendations). An author of a recent and relevant evidence review, regardless of whether it directly supports the guideline under consideration, cannot author the guideline or vote on recommendations. Participant disclosures and conflict management summaries for each CGC meeting are posted publicly online and linked to in CGC publications (www.acponline.org/about-acp/who-we-are/leadership/committees-boards-councils/clinical-guidelines-committee/disclosure-of-interests-for-clinical-guidelines-committee).

Selection and Scope of Topics and Target Audience

ACP clinical guidelines and guidance statements address prevention, screening, diagnosis, and treatment of various diseases relevant to internal medicine. Topic ideas come from ACP members, CGC members, and other ACP committees and governance. In selecting a topic, the CGC considers the following characteristics of a condition: its effect on morbidity and mortality, its prevalence and impact, whether effective health care alternatives are available, areas of clinical uncertainty, evidence that current performance does not meet best practices, cost and resource implications, available management options, the likelihood that evidence is available to develop recommendations, and relevance to internal medicine and its subspecialties. For guidance statements, it is also necessary to ensure the availability of guidelines produced by other organizations.

CGC Guidelines and Their Development Process

Key Questions and Scope

The CGC and Clinical Policy staff assess the nature, quality, and quantity of evidence before commissioning an evidence review to support a new guideline. When proceeding with a new guideline, the assigned CGC topic subgroup, with input from a technical expert panel, drafts or revises the initial key questions and determines the population, interventions, comparators, and outcomes of interest (PICO), which are presented to the full CGC for feedback and approval.

Evidence Reviews

All ACP clinical guidelines are based on systematic reviews of evidence, which are derived from several sources. The CGC may 1) commission an evidence review directly from an external evidence review group, such as an Agency for Healthcare Research and Quality Evidence-based Practice Center or a Cochrane Center; 2) nominate a topic to the Agency's Effective Health Care Program for an evidence review conducted through the Evidence-based Practice Center Program or use an existing such review; or 3) use an evidence review done by the Veterans Affairs Evidence Synthesis Program. A technical expert panel is convened to inform the evidence review and assist in refining the scope and PICO, as well as to provide clinical guidance for the review via e-mail and conference calls. The panel typically includes internal medicine subspecialists in the topic area, methodologists, and CGC topic subgroup members. In addition to serving on the technical expert panel, the topic subgroup members participate in conference calls with the evidence review team and provide a link between the evidence reviewers and the full CGC.

Rating Outcomes

The CGC and CGC Public Panel independently rate outcomes to consider in the evidence reviews and clinical guidelines via online surveys. Participants rate outcomes according to criteria from the GRADE working group (6). Outcomes are categorized as critical, important, or not important for decision making. Those rated as critical are considered the most crucial for making recommendations and carry more weight in decision making than those rated as important. Rating may vary by condition or intervention but must include both clinical benefits and harms. For example, mortality is weighted as a critical outcome in assessment of cancer screening but may be considered less important in evaluation of treatments for back pain.

Determining Certainty of Evidence (Quality of Evidence)

The evidence review team and the CGC use GRADE to rate the certainty of evidence for each critical and important outcome and for the overall body of evidence. The certainty—or quality—of evidence is graded as high, moderate, or low depending on the quality of included studies, consistency of findings across studies, number of patients included, precision of estimated treatment effects, directness of the evidence to the question of interest (relevance of the outcomes and populations assessed to the outcomes and populations of interest), and potential for publication bias (Figure 2) (6–8). The overall findings are summarized in standardized GRADE evidence summaries (evidence profiles or summary-of-findings tables), which report both relative and absolute effects. When possible, evidence reviews also address whether the effect met an established clinically meaningful threshold for the outcomes of interest. Figure 2 summarizes the grades for certainty of evidence. The higher the certainty, the more confidence we have that the effect estimate is aligned with the true effect. The CGC does not have a “very low” category for certainty of evidence and instead considers such evidence to be insufficient, a minor difference from the GRADE framework.

• Download figure
• Download PowerPoint

Developing and Finalizing Recommendations

Evidence-to-Decision Tables

Following the GRADE framework (6), the CGC uses evidence-to-decision (EtD) tables, which serve as the roadmap for documenting decisions and the evidence used while drafting, deliberating, and finalizing recommendations. The EtD tables summarize the PICO, assessment of desirable versus undesirable effects of the intervention, certainty of evidence, patient values and preferences, costs and resources, and judgments to support the recommendation. The EtD tables containing the final recommendations are published as an appendix to each guideline and ensure transparency about judgments in the development of recommendations from the available evidence (see the Supplement for a sample blank EtD table and https://gradepro.org/guidelines for examples of published EtD tables online).

Determining the Strength of Recommendations

The CGC rigorously considers an intervention's clinical benefits and harms, as well as variation in patient values and preferences, in deliberations about recommendations for each guideline. It also considers costs and burden of care when assessing health care value and developing recommendations. However, the CGC has not done cost-effectiveness analyses or incorporated information from such analyses into its recommendations. More rigorously assessing and incorporating cost into guideline recommendations is an ongoing initiative within the CGC. The Public Panel helps to inform patient values and preferences via outcome rating, participation in various stages of guideline development, and thorough review of the guideline. The CGC grades the strength of each recommendation as strong or conditional on the basis of the overall balance of benefits and harms, the certainty (or quality) of the evidence on treatment effects, patient preferences and values, and considerations about cost or resource allocation (Figure 2) (3). Strong recommendations usually support actions in which benefits clearly outweigh harms, or vice versa, and for which patient values and preferences would have little variation. They generally apply to most patients in most circumstances and may be suitable to inform relevant performance measures. Strong recommendations are generally based on high or moderate certainty in evidence. In rare and extenuating circumstances, a strong recommendation may be based on low-certainty evidence—for example, when the evidence indicates a substantial net benefit in a life-threatening situation (9). Conditional recommendations are often based on evidence that is of low certainty, shows benefits closely balanced with harms, or shows variability in patient preferences. They apply to many but not most patients and are not directly suited to translation into performance measures. Their implementation is often determined by variation in individual clinical situations, including disease factors, patient preferences and characteristics, and resource use, and usually involves a shared decision-making situation. Figure 2 provides examples of strong and conditional recommendations from ACP clinical guidelines.

CGC Guidance Statements and Their Development Process

Guidance statements differ from clinical guidelines in several respects. The CGC develops guidance statements on topics where several conflicting clinical guidelines are available. The aim of ACP guidance statements is to reconcile clinical guidelines to help clinicians provide evidence-based care for their patients. The CGC accomplishes this by rigorously reviewing the available guidelines and their evidence base and developing subsequent guidance statements based on an assessment of the reported benefits, harms, costs, and patient preferences and values from the assessed guidelines and their evidence. Unlike recommendations in CGC clinical guidelines, the guidance in guidance statements is not derived from a de novo systematic evidence review that was specifically done for or used by ACP (that is, ACP is not directly involved with the topic development, key question or PICO refinement, or outcome rating), and hence the CGC does not use GRADE to assess the certainty of evidence or strength of recommendations. As such, guidance statements are typically not suitable for the development of performance measures.

Selection of Clinical Guidelines

To select clinical guidelines for review and assessment, Clinical Policy staff search the Guidelines International Network library and PubMed for relevant guidelines developed by national-level organizations and published in English no earlier than 5 years before work on the topic. On the basis of the knowledge and expertise of the CGC members, the topic subgroup may select additional clinical guidelines that are commonly used in clinical practice or search Web sites of relevant medical societies or other national organizations to identify such guidelines. The topic subgroup reviews the titles and abstracts of each document and excludes those that directly adapted a guideline from another organization.

Evaluation of Clinical Guidelines

Each member of the CGC topic subgroup independently evaluates the selected clinical guidelines using the AGREE II (Appraisal of Guidelines for Research and Evaluation II) instrument. AGREE II asks 23 questions in 6 domains—scope and purpose, stakeholder involvement, rigor of development, clarity of presentation, applicability, and editorial independence—in addition to overall assessment questions (10). For the overall assessment questions, each appraiser is asked to give each guideline an overall score and to indicate whether they would recommend the guideline for use in clinical practice. The domains of the AGREE II instrument assess only methodology and rigor of development, whereas the overall score also incorporates CGC judgments about the transparency of the processes that resulted in the final recommendations. Thus, in some cases, a guideline may score high on some AGREE II domains but receive a lower overall rating. For example, beyond whether patient values and preferences were sought, the CGC looks into how they were taken into account when the recommendations were drafted. Also, in addition to the explicit link between recommendations and evidence, the CGC evaluates the appropriateness of the weighting of benefits and harms.

Guidance Statements

The guidance is based on an evaluation of the recommendations and evidence included in the selected and most highly rated clinical guidelines (according to the AGREE II instrument). In developing the guidance, the CGC does not simply adopt recommendations from the guidelines with the highest AGREE II scores. Rather, it assesses the evidence base informing these recommendations and independently assesses benefits, harms, costs, and patient values and preferences. The CGC public members help to inform patient values and preferences by participating in the discussion during meetings, and the Public Panel provides further input by reviewing and commenting on the guidance statement. The final result is adoption or adaptation of the existing high-quality clinical guidelines.

Addressing Insufficient Evidence in CGC Clinical Guidelines and Guidance Statements

When evidence is deemed inadequate to accurately assess the net benefit of an intervention overall or in particular patient or intervention subgroups, the CGC addresses this in a section of the guideline or guidance statement dedicated to inconclusive areas of evidence and does not issue recommendations. Areas often addressed include patients with multiple comorbid conditions, differences by sex or race, patients at higher or lower risk for the condition, variation in patient preferences or treatment burden, and the importance of cost and health care value in treatment considerations.

CGC Clinical Guideline and Guidance Statement Presentation Format

The CGC believes that implementation of scientifically sound clinical guidelines is enhanced by clear, consistent, and practical reporting presentation. All clinical guidelines and guidance statements from the CGC follow a standard format that includes an introduction briefly outlining the condition, its prevalence, interventions of interest, and the intended purpose and target population. The CGC outlines the methods and data sources for both clinical guidelines and guidance statements, including the literature search dates. Systematic evidence reviews always accompany CGC clinical guidelines. For clinical guidelines, the CGC states the strength of each recommendation and the certainty of its evidence. The text below the recommendations or guidance statements highlights the evidence of benefits, harms, and costs, as well as other relevant information. For guidance statements, the CGC describes, evaluates, summarizes, and attempts to reconcile and explain commonalities and differences in the guideline group recommendations. It also describes the methods and evidence contained in the selected guidelines to further inform readers about factors that may have led to specific recommendation statements. The CGC includes a rationale for the final guidance statements. Clinical guidelines and guidance statements typically have 5 or fewer concise recommendations, which are also listed in the abstract in italics to enhance visibility. When possible, the CGC also includes information on patients with comorbid conditions, performance measurement implications, and future research needs. All papers include a summary figure with key information; clinical considerations; and, in many cases, talking points for patients. Clinical guidelines and guidance statements submitted for publication in Annals of Internal Medicine are also accompanied by a summary for patients.

Review and Approval of Clinical Guidelines and Guidance Statements

CGC Review

The CGC reviews and discusses all clinical guidelines and guidance statements at in-person meetings. The topic subgroup introduces the clinical guideline or guidance statement with a brief presentation summarizing the evidence and proposed recommendations. For clinical guidelines, the CGC reviews and appraises the evidence reports, accompanying literature contained in those reports, and EtD tables to ensure an explicit link between evidence and recommendations. The CGC uses a similar process for guidance statements with regard to assessment of the existing guidelines and their accompanying evidence. Although no formal consensus method is used, members discuss recommendations and guidance statements and revise accordingly until they achieve a general consensus on the final version.

CGC Voting Policy

Only CGC members can participate in voting. Voter eligibility within the CGC is determined on the basis of the management of conflicts of interest for the topic (5). Votes are taken for each recommendation or guidance statement individually. A 75% agreement among eligible voters is required to approve a recommendation or guidance statement. This threshold is the same for both conditional and strong recommendations in clinical guidelines. If the threshold is not met, the recommendation or guidance statement can be discussed further, revised, and voted on again, or removed from the paper. Votes cast during CGC meetings are blinded during the meeting to avoid bias, and a record of voting results is kept and recorded in the meeting minutes (unblinded). The CGC does not publicly disclose the voting records of individual members.

CGC Public Panel Review

The CGC Public Panel reviews and provides feedback on CGC clinical guidelines and guidance statements at various stages of development, including key questions, outcome rating (guidelines only), and the CGC-approved guidelines or guidance statements. When papers include talking points with patients, the CGC Public Panel reviews this section carefully. The CGC reviews the comments and takes them into consideration in its decision making and in the final manuscript.

Peer Review Process

ACP clinical guidelines and guidance statements are posted for review and comments by the ACP Board of Governors, which represents ACP members from all 50 states and territories, other countries, and various subspecialties. The Board of Regents, ACP's highest governing body, provides comments and final approval of the guideline or guidance statement as ACP policy. The Board of Regents votes to approve CGC papers with a simple yes-or-no vote and cannot make changes to the recommendations or guidance statements. ACP may send out guidelines for external peer review and feedback by clinical experts before approval by the Board of Regents or for endorsement from other medical societies once the guideline is complete and approved. Clinical guidelines and guidance statements also undergo a thorough peer review on submission to a journal for publication consideration.

Publication and Dissemination

All ACP clinical guidelines, guidance statements, and evidence reviews are submitted for publication in a high-impact journal wherein each manuscript is independently peer reviewed. All ACP clinical recommendations and guidance statements are considered public documents and are available for free. Links to the papers can be found on ACP's Web site at www.acponline.org/clinical-information/guidelines.

In addition to journal publication and Web site posting, ACP clinical guidelines and guidance statements are presented at ACP's annual meeting, announced in ACP newsletters, published in the free ACP Clinical Guidelines app (available from Google Play or the Apple Store), and covered by national media outlets. Guidelines are submitted to the Guidelines International Network library, where they are accompanied by a checklist of guideline standards.

Expiration and Updating of ACP Clinical Recommendations

All ACP clinical guidelines and guidance statements are considered automatically withdrawn or invalid 5 years after publication or once an update has been issued. Expired documents are available in an inactive clinical guidance section on the ACP Web site, as well as in the app. The CGC is working toward creating living systematic reviews and clinical guidelines and a core set of topics to ensure that certain topics do not expire.

Financial Support

Financial support for the development of ACP clinical guidelines and guidance statements and for evidence reviews commissioned by ACP comes exclusively from the ACP operating budget. Financial support for external reviews not commissioned by ACP is described in the published reviews and frequently derives from public monies (for example, the Agency for Healthcare Research and Quality or Department of Veterans Affairs). ACP staff who author the clinical guidelines or guidance statements receive no additional compensation for the development of the papers beyond their salary, which comes out of the ACP operating budget. No industry funding is accepted for any stage of development of clinical guidelines or guidance statements.

CGC Members

Physician CGC members do not receive any honoraria except reimbursement for travel-related costs, which comes out of the ACP operating budget. Public members receive an honorarium in addition to travel reimbursement for attending CGC meetings. All financial support for CGC members comes exclusively from the ACP operating budget in accordance with ACP policy.

CGC Public Panel Members

All Public Panel members receive an honorarium for completing training on evidence-based medicine offered by ACP. All financial support for Public Panel members comes exclusively from the ACP operating budget in accordance with ACP policy.

Conclusion

This paper represents the latest update of the CGC development processes for guidelines and guidance statements. The CGC continues to work on advancing its methods to keep current with or surpass standards in the field. The focus for future methods updates includes developing living systematic reviews and guidelines and incorporating costs in a methodologically rigorous and consistent way.