The PRISMA Extension Statement for Reporting of Systematic Reviews Incorporating Network Meta-analyses of Health Care Interventions: Checklist and Explanations
FREEAbstract
The PRISMA statement is a reporting guideline designed to improve the completeness of reporting of systematic reviews and meta-analyses. Authors have used this guideline worldwide to prepare their reviews for publication. In the past, these reports typically compared 2 treatment alternatives. With the evolution of systematic reviews that compare multiple treatments, some of them only indirectly, authors face novel challenges for conducting and reporting their reviews. This extension of the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analyses) statement was developed specifically to improve the reporting of systematic reviews incorporating network meta-analyses.
A group of experts participated in a systematic review, Delphi survey, and face-to-face discussion and consensus meeting to establish new checklist items for this extension statement. Current PRISMA items were also clarified. A modified, 32-item PRISMA extension checklist was developed to address what the group considered to be immediately relevant to the reporting of network meta-analyses.
This document presents the extension and provides examples of good reporting, as well as elaborations regarding the rationale for new checklist items and the modification of previously existing items from the PRISMA statement. It also highlights educational information related to key considerations in the practice of network meta-analysis. The target audience includes authors and readers of network meta-analyses, as well as journal editors and peer reviewers.
Systematic reviews and meta-analyses are fundamental tools for the generation of reliable summaries of health care information for clinicians, decision makers, and patients. Systematic reviews provide information on clinical benefits and harms of interventions, inform the development of clinical recommendations, and help to identify future research needs. In 1999 and 2009, respectively, groups developed the Quality of Reporting of Meta-Analyses (QUOROM) statement (1) and the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) statement (2, 3) to improve the reporting of systematic reviews and meta-analyses. Both statements have been widely used, and coincident with their adoption, the quality of reporting of systematic reviews has improved (4, 5).
Systematic reviews and meta-analyses often address the comparative effectiveness of multiple treatment alternatives. Because randomized trials that evaluate the benefits and harms of multiple interventions simultaneously are difficult to perform, comparative effectiveness reviews typically involve many studies that have addressed only a subset of the possible treatment comparisons. Traditionally, meta-analyses have usually compared only 2 interventions at a time, but the need to summarize a comprehensive and coherent set of comparisons based on all of the available evidence has led more recently to synthesis methods that address multiple interventions. These methods are commonly referred to as network meta-analysis, mixed treatment comparisons meta-analysis, or multiple treatments meta-analysis (6–8). In recent years, there has been a notable increase in the publication of articles using these methods (9). On the basis of our recent overview (10) of reporting challenges in the field, as well as findings from our Delphi exercise involving researchers and journal editors, we believe that reporting guidance for such analyses is sorely needed.
In this article, we describe the process of developing specific advice for the reporting of systematic reviews that incorporate network meta-analyses, and we present the guidance generated from this process.
Development of the PRISMA Network Meta-analysis Extension Statement
We followed an established approach for this work (11). We formed a steering committee (consisting of Drs. Hutton, Salanti, Moher, Caldwell, Chaimani, Schmid, Thorlund, and Altman); garnered input from 17 journal editors, reporting guideline authors, and researchers with extensive experience in systematic reviews and network meta-analysis; and performed an overview of existing reviews of the reporting quality of network meta-analyses to identify candidate elements important to report in network meta-analyses (10). We also implemented an online Delphi survey of authors of network meta-analyses in mid-2013 (215 invited; response rate, 114 [53%]) by using Fluid Surveys online software (Fluidware, Ottawa, Ontario, Canada) to determine consensus items for which either a new checklist item or an elaboration statement would be required, and to identify specific topics requiring further discussion.
Next, we held a 1-day, face-to-face meeting to discuss the structure of the extension statement, topics requiring further consideration, and publication strategy. After this meeting, members of the steering committee and some of the meeting participants were invited to contribute specific components for this guidance. All participants reviewed drafts of the report.
Scope of This Extension Statement
This document provides reporting guidance primarily intended for authors, peer reviewers, and editors. It may also help clinicians, technology assessment practitioners, and patients understand and interpret network meta-analyses. We also aim to help readers develop a greater understanding of core concepts, terminology, and issues associated with network meta-analysis.
This document is not intended to be prescriptive about how network meta-analyses should be conducted or interpreted; considerable literature addressing such matters is available (6, 12–51). Instead, we seek to provide guidance on important information to be included in reports of systematic reviews that address networks of multiple treatment comparisons. For specific checklist items where we have suggested modification of instructions from the PRISMA statement, we have included examples of potential approaches for reporting different types of information. However, modified approaches to those presented here may also be feasible.
How to Use This Document
This document describes modifications of checklist items from the original PRISMA statement for systematic reviews incorporating network meta-analyses. It also describes new checklist items that are important for transparent reporting of such reviews. We present an integrated checklist of 32 items, along with elaborations that demonstrate good reporting practice. The elaboration (Appendix,) describes each item and presents examples for new or modified items. Although new items have been added in what was deemed the most logical place in the core PRISMA checklist, we do not prescribe an order in which these must be addressed. The elaboration also includes 5 boxes that highlight methodological considerations for network meta-analysis.
The Table presents the PRISMA network analysis checklist that authors may use for tracking inclusion of key elements in reports of network meta-analyses. The checklist has been structured to present core PRISMA items and modifications of these items where needed, as well as new checklist items specific to network meta-analysis. Checklist items are designated "New Item" in the main text if they address a particular aspect of reporting that is novel to network meta-analyses; these are labeled S1 through S5. The heading "Addition" indicates discussion of an issue that was covered by the original PRISMA statement but requires additional considerations for reviews incorporating network meta-analyses. Examples with elaborations have been provided for checklist items in these 2 categories.
 |
What Is a Treatment Network?
Systematic reviews comparing the benefits and harms of multiple treatments are more complex than those comparing only 2 treatments. To present their underlying evidence base, reviews involving a network meta-analysis commonly include a graph of the network to summarize the numbers of studies that compared the different treatments and the numbers of patients who have been studied for each treatment (Figure 1). This network graph consists of nodes (points representing the competing interventions) and edges (adjoining lines between the nodes that show which interventions have been compared among the included studies). The sizes of the nodes and the thicknesses of the edges in network graphs typically represent the amounts of respective evidence for specific nodes and comparisons. Sometimes, additional edges are added to distinguish comparisons that may be part of multigroup studies that compare more than 2 treatments.
A network graph presenting the evidence base for a hypothetical review of 4 interventions is shown. Treatments are represented by nodes and head-to-head studies between treatments are represented by edges. The sizes of edges and nodes are used to visually depict the available numbers of studies comparing interventions and the numbers of patients studied with each treatment.
The graphs also allow readers to note particular features of the shape of a treatment network. This includes the identification of closed loops in the network; a closed loop is present in a treatment network when 3 or more comparators are connected to each other through a polygon, as in Figure 1 for treatments A, B, and C. This shows that treatments A, B, and C have all been compared against each other in existing studies, and thus each comparison in the closed loop (AB, AC, BC) is informed by both direct and indirect evidence (see the Box for definitions of direct and indirect evidence and Figure 2 for a graphical representation of terms in the Box).
Terms are discussed further in the Box.Top. Adjusted indirect treatment comparison of treatments B and C based on studies that used a common comparator, treatment A. Middle. A network of 8 treatments and a common comparator, with a mix of comparisons against the control treatment and a subset of all possible comparisons between active treatments. Bottom. A treatment network similar to that shown in the middle panel, but with study data available for an additional 4 comparisons in the network which form closed loops.
Discussion
All phases of the clinical research cycle generate considerable waste, from posing irrelevant questions to inappropriate study methods, bad reporting, and inadequate dissemination of the completed report. Poor reporting is not an esoteric issue. It can introduce biased estimates of an intervention's effectiveness and thus affect patient care and decision making. Journals regularly publish new evidence regarding some aspect of inadequate reporting (52). Improving the completeness and transparency of reporting research is a low-hanging fruit to help reduce waste, and possibly explains the rise in developing reporting guidelines (53, 54) and such initiatives as the EQUATOR Network.
The PRISMA statement was aimed at improving the reporting of traditional pairwise systematic reviews and meta-analyses; it has been endorsed by hundreds of journals and editorial groups. Some extensions have been developed, including PRISMA for reporting abstracts (55) and equity (56). Other extensions are in various stages of development, including those for individual patient–data meta-analyses and harms.
Here, we describe a PRISMA extension for reporting network meta-analyses, which includes a 32-item checklist and flow diagram. This extension adds 5 new items that authors should consider when reporting a network meta-analysis, as well as 11 modifications to existing PRISMA items. Some of these are minor, whereas others are more complex, such as items 20 and 21, which ask authors to describe the results of individual studies and the corresponding syntheses thereof.
For network meta-analysis, in which it is likely that more studies and treatments will be included compared with traditional pairwise reviews, this added reporting might require authors to prepare several supplemental files as part of the manuscript submission process. Journal editors will need to make allowances for these additional materials.
Certain modifications included in some of the checklist items (for example, assessment of model fit, rationale for lumping of interventions, and presentation of tabulated study characteristics) involve considerations that are equally applicable to traditional meta-analyses of 2 treatments. Although it could be suggested that these do not warrant listing as modifications, we believe this is worthwhile; several of these items were not explicitly addressed in the PRISMA statement and could be more commonly encountered when dealing with networks of treatments. Several coauthors of this reporting guidance are also members of the authorship team of the PRISMA statement and will bring these items forward when the PRISMA statement is updated in the future.
Optimally, we would like journals to endorse this extension in much the same way they have done for the PRISMA statement. Endorsement is probably best achieved through unambiguous language in the journal's instructions to authors; example wording is provided in the Appendix.
Endorsement is important, but it is less potent without implementation. At the simplest level, implementation can involve asking authors to populate the PRISMA network meta-analysis checklist with appropriate text from their report, and not accepting a submission unless this is provided. Some editors—particularly of those smaller journals, where most systematic reviews are published (57)—may perceive any endorsement and implementation as a barrier to receiving network meta-analyses reports. There are few data to support this perception. Editors can promote reporting guideline endorsement and implementation as an important way to improve the completeness and transparency of what they publish (58, 59), thus upholding one of the central tenets of the Declaration of Helsinki (60). In addition, this will reduce waste in reporting research.
There has been a steep upward trajectory of published network meta-analysis (8, 9) and methods research as the field rapidly gains momentum and interest. To help keep this PRISMA extension as up-to-date and evidence-based as possible, we invite readers to let us know about emerging evidence to help inform future updates.
Appendix: The PRISMA Network Meta-analysis Extension Statement
Title and Abstract
Item 1: Title
Addition
Identify the report as including the evaluation of a network of multiple treatment comparisons (for example, "network meta-analysis").
Examples
Different combined oral contraceptives and the risk of venous thrombosis: systematic review and network meta-analysis. (61)
Network meta-analysis on randomized trials focusing on the preventive effect of statins on contrast-induced nephropathy. (62)
Elaboration
Recent literature has documented the rapid increase in the publication of reviews incorporating networks of treatments and highlights a need to develop appropriate identification of such publications in literature databases (8). Consistent inclusion of the appropriate term in journal article titles will increase the ability to identify network meta-analyses.
Item 2: Structured Summary
Addition
Guidance from the PRISMA statement is transferable to reviews incorporating network meta-analyses, although some additional considerations are worthy of inclusion. The abstract from a recent systematic review of treatments for prevention of asthma exacerbations by Loymans and colleagues (63) highlights these features.
Examples
Objective. To determine the comparative effectiveness and safety of current maintenance strategies in preventing exacerbations of asthma.
Design. Systematic reviewand network meta-analysisusing Bayesian statistics.
Data Sources. Cochrane systematic reviewson chronic asthma, complemented by an updated search when appropriate.
Eligibility Criteria. Trials of adults with asthma randomised to maintenance treatments of at least 24 weeks duration and that reported on asthma exacerbations in full text. Low dose inhaled corticosteroid treatment was the comparator strategy. The primary effectiveness outcome was the rate of severe exacerbations. The secondary outcome was the composite of moderate or severe exacerbations. The rate of withdrawal was analysed as a safety outcome.
Results. 64 trials with 59,622 patient years of follow-up comparing 15 strategies and placebo were included. For prevention of severe exacerbations, combined inhaled corticosteroids and long acting β-agonists as maintenance and reliever treatment and combined inhaled corticosteroids and long acting β-agonists in a fixed daily dose performed equally well and were ranked first for effectiveness. The rate ratios compared with low dose inhaled corticosteroids were 0.44 (95% CrI 0.29 to 0.66) and 0.51 (0.35 to 0.77), respectively. Other combined strategies were not superior to inhaled corticosteroids and all single drug treatments were inferior to single low dose inhaled corticosteroids. Safety was best for conventional best (guideline based) practice and combined maintenance and reliever therapy.
Conclusions. Strategies with combined inhaled corticosteroids and long acting β-agonists are most effective and safe in preventing severe exacerbations of asthma, although some heterogeneity was observed in this network meta-analysisof full text reports.
Elaboration
The inclusion of some additional information is worthwhile for systematic reviewsthat include network meta-analyses. The design or methods section of the structured abstract should mention that a network meta-analysiswas conducted. Given that in some reviews treatment networks may be large and involve many pairwise comparisons between treatments, authors may summarize findings using estimates versus a particular treatment of interest (for example, the apparent "best" treatment, placebo, and so forth). When treatments are ranked by efficacy or safety (Appendix Box 1), it is also recommended that authors describe the relative effects. Selective focus on particular comparisons alone—for example, only those meeting statistical significance—should be avoided. Authors are also encouraged to briefly note any concerns (for example, violations of analytical assumptions as described in Appendix Boxes 2 and 3) that may have an important effect on the interpretation of findings.
Introduction
Item 3: Rationale
Addition
Briefly state why consideration of a network of multiple treatments is essential to the review (63–66).
Example
Although progress has been achieved in the field and patients live longer, the relative merits of the many different chemotherapy and targeted treatment regimens are not well understood. Hundreds of trials have been conducted to compare treatments for advanced breast cancer, but because each has compared only two or a few treatments, it is difficult to integrate information on the relative efficacy of all tested regimens. This integration is important because different regimens vary both in cost and in toxicity. Therefore, we performed a comprehensive systematic review of chemotherapy and targeted treatment regimens in advanced breast cancer and evaluated through a multiple-treatments meta-analysis the relative merits of the many different regimens used to prolong survival in advanced breast cancer patients. (67)
Elaboration
Authors should briefly clarify to readers why a systematic reviewusing a network meta-analysisapproach was chosen to answer the research question. Possible rationales may include a lack of head-to-head randomized trialscomparing treatments of interest, or the need to assess several treatments in developing a clinically meaningful understanding of the relative effectiveness or harms of different treatment options.
Item 4: Objectives
Guidance from the original PRISMA statement applies. State the research question being addressed in the systematic reviewin terms of the PICOS criteria (population, intervention, comparators, outcome[s], study design).
Methods
Item 5: Protocol and Registration
Guidance from the original PRISMA statement applies. The protocol for the review should be registered.
Item 6: Eligibility Criteria
Addition
The PRISMA statement outlines that authors provide a description of essential study characteristics (for example, PICOS details and duration of follow-up) and report characteristics (such as eligible publication years and eligible publication languages) that were used as eligibility criteria for the review. In network meta-analyses, authors should also clearly describe inclusion and exclusion criteria for treatment regimens (that is, nodes) and should provide justification when treatment nodes are merged to form single comparators (a practice sometimes described as "lumping" of interventions; see example below). Authors should describe the included treatments and adherence to and assessment of the transitivity assumption (Appendix Box 2).
Example: Lumping of Interventions
Our analysis classified fluids as crystalloids (divided into balanced and unbalanced solutions) and colloids (divided into albumin, gelatin, and low- and high-molecular weight hydroxyethyl starch [HES] [threshold molecular weight, 150 000 kDa]). We considered fluid balanced if it contained an anion of a weak acid (buffer) and its chloride content was correspondingly less than in 0.9% sodium chloride. The relevant analyses were a 4-node NMA [network meta-analysis] (crystalloids vs. albumin vs. HES vs. gelatin), a 6-node NMA (crystalloids vs. albumin vs. HES vs. gelatin, with crystalloids divided into balanced or unbalanced and HES divided into low or high molecular weight), and a conventional direct frequentist fixed effects meta-analytic comparison of crystalloids versus colloids. (68)
Elaboration
Often, one has to decide whether to lump or split treatments—that is, whether to combine different doses of the same drug, alternative forms of administration of the same drug, or varying durations of administration, or different controls. Lumping requires treatments to have similar treatment effects, and although this technique is appropriate in some cases, it should be supported by a clear rationale when performed.
Specification of the patient and study characteristics of interest should also be clarified in this section. Although this remains similar to guidance from the PRISMA statement, it is important to provide additional detail with regard to the interventions and comparators included to define the network structure. For example, older "legacy" treatments may no longer be considered relevant if they have been abandoned in clinical practice; however, their inclusion in the treatment network may be useful if they introduce connections to other treatments that are of primary interest. The most common example would be the inclusion of placebo, an intervention that will increase the amount of information available for many networks (32).
Issues of transitivity (that is, the existence of comparable distributions of patient characteristics across studies in the treatment network [Appendix Box 2]) can be discussed when describing eligibility criteria. Ideally, all evidence comparing relevant interventions in the target population of interest should be included in order to provide clinically useful results. However, the larger the network, the more likely it becomes that some of its pieces may not be exchangeable, owing to important differences in effect-modifying factors (for example, specific patient population or study design features); that is, the assumption of transitivity may become more difficult to defend. Accordingly, authors are encouraged to report relevant information on potentially influential patient and study characteristics to inform readers' judgments about the assumption of transitivity. Arguments in favor of defining the evidence base in a way that maximizes the plausibility of transitivity have been outlined elsewhere (18, 69), however, these are not shared by all meta-analysts. Known and well-validated effect modifiers are sparse in the medical literature, and therefore many meta-analysts feel that it is important to be maximally inclusive and allow the meta-analysisto explore for the presence of differences in effect sizes due to differences in potential effect modifiers.
Item 7: Information Sources
Guidance from the PRISMA statement regarding description of the information sources for a systematic reviewremains relevant for the reporting of network meta-analyses.
Item 8: Search
Guidance from the original PRISMA statement applies.
Item 9: Study Selection
Guidance from the original PRISMA statement applies.
Item 10: Data Collection Process
Guidance from the original PRISMA statement applies.
Item 11: Data Items
The guidance provided in the PRISMA statement remains applicable for network meta-analyses. Authors may also report whether additional information regarding possible effect modifiers was collected. This may be especially important in network meta-analyses involving interventions whose corresponding evidence base spans a broad time frame where co-interventions (or other aspects of care), diagnostic criteria, or other aspects of the patient population may have changed over time. Providing clarity of such information to readers will enhance their ability to appraise the validity of the network meta-analysis.
Item S1 (New Item): Review of Network Geometry
Describe the methods used to evaluate the geometry of the network of evidence and potential biases related to it. This should include how the evidence base has been graphically summarized.
Example
We analyzed published and unpublished randomized trials performed in patients with pulmonary hypertension. At the level of drug classes, we examined whether head-to-head comparisons are between agents in the same class or between agents in different classes. At the level of companies, we examined whether trials involve only agents (as active comparators or backbones) owned by the same company, or include treatments by different companies. In the networks of drug comparisons, each drug is drawn by a node and randomized comparisons between drugs are shown by links between the nodes. When a drug is compared against the same agent in different dose or formulation, this is represented by an auto-loop. In the networks of companies, nodes stand for companies and auto-loops around these nodes represent trials involving agents of a single company. Links between different nodes characterize trials comparing agents that belong to different companies. ( 70)
Elaboration
This new checklist item recommends that authors reporting network meta-analyses should evaluate the geometry(71) of the network (Appendix Box 4). Generation of a network graph is important and is of considerable help in reviewing network geometry. The graphical representation of all comparisons can help to determine whether a network meta-analysisis feasible (for example, whether the network of interventions is connected), and whether the network contains closed loops of treatments such that inconsistency (the agreement between the effects estimated from direct and indirect sources) can be assessed (Appendix Box 3)(72–74). The assessment of geometry can be qualitative (that is, a narrative summary of these features) and can optionally be supplemented with quantitative measures described elsewhere (33, 75) (Appendix Box 4).
Considerations can be made to address networks according to classes based on mechanism of action, line of treatment, sponsorship (as described in the above example), or other sources that may reflect biases on the choice of treatment comparisons made. For example, drug sponsors have little incentive to compare agents other than those they manufacture (76–78); drug treatments may not be compared against surgical or invasive treatments because they are used by different specialists (79); and first-line treatments, such as neglected tropical diseases, may not be adequately compared against each other (73).
Item 12: Risk of Bias in Individual Studies
As in the original PRISMA statement, researchers are encouraged to describe the level of assessment for each included study (at the study level itself, or for each outcome within the study) and the assessment tool used (for example, the Cochrane Risk of Bias Scale [80]). They should also mention how findings from risk of bias assessments will be used to inform data analyses and interpretation.
Item 13: Summary Measures
Addition
As outlined in the PRISMA statement, the chosen summary measures of effect to express comparisons between interventions (for example, odds ratios or mean differences) should be specified (81). Because the number of included studies can be considerably larger in network meta-analysesthan in traditional meta-analyses, and because a single analysis can generate considerably more pairwise comparisons, modified approaches to summarize findings may be required and should be mentioned in the methods section of the review (see item 21, which includes examples of treatment-level forest plots, league tables, and others). Additional summary measures of interest, such as treatment rankings or surface under the cumulative ranking curve (Appendix Box 1), may be described in the main text or supplements as deemed appropriate. Guidance about how to draw interpretations for all summary measures should be provided.
Example
For each pairwise comparison and each outcome at each time point, we used odds ratios (OR) with 95% confidence intervals (95% CIs) as a measure of the association between the treatment used and efficacy. As the outcomes are negative, ORs >1 correspond to beneficial treatment effects of the first treatment compared with the second treatment.
As a measure that reflects ranking and the uncertainty, we used the Surface Under the Cumulative RAnking curve (SUCRA) as described in Salanti 2011. This measure, expressed as percentage, showed the relative probability of an intervention being among the best options. (82)
Elaboration
Conventional effect measures (such as mean differences and odds ratios) that are also used in pairwise meta-analysisare the primary measures of comparative efficacy between pairwise comparisons of interventions. These should be reported with an associated measure of uncertainty, typically 95% CIs for frequentist analyses and 95% credible intervals (CrIs) for Bayesian analyses. An additional output of network meta-analysis may be a relative ranking of the competing interventions included in the meta-analysis. If authors include rankings, they need to describe the approaches and measures used to rank the treatments and how findings based on these measures are interpreted (Appendix Box 1). Reviewers who evaluate more than 1 outcome are encouraged to report the relative ranking for every outcome.
Similar to guidance from the PRISMA statement, authors should keep in mind that differences in relative effects do not necessarily imply clinical or policy relevance. As such, reporting absolute differences alongside relative measures of effect may aid in interpretation of findings. Regarding probabilities associated with treatment rankings, authors are encouraged to report not only the probability of each intervention being best, but also a more complete presentation of rankings that includes the probability of being second best, third best, and so forth. This provides a picture of the uncertainty associated with the rankings.
Item 14: Planned Methods of Analysis
Addition
Although much of the guidance from the PRISMA statement applies, additional information is needed to enable complete understanding or replication of a network meta-analysis. In addition, the PRISMA statement did not discuss the reporting of considerations for Bayesian meta-analyses (Appendix Box 2).
Example
The network meta-analysis was based on a bayesian random effects Poisson regression model, which preserves randomised treatment comparisons within trials. The model uses numbers of patients experiencing an event and accumulated patient years to estimate rate ratios. The specification of nodes in the network was based on the randomised intervention or in case of strategy trials, such as COURAGE [Clinical Outcomes Utilizing Revascularization and Aggressive Drug Evaluation] or FAME-2 [Fractional flow reserve versus Angiography for Multi-Vessel Evaluation], on the intervention received by the majority of patients in a trial arm. Analyses were performed using Markov-Chain Monte-Carlo methods. The prior distribution for treatment effects was minimally informative: a normal distribution with a mean of 1 and a 95% reference range from 0.01 to 100 on a rate ratio scale. The prior for the between trial variance τ2, which we assumed to be equal across comparisons, was based on empirical evidence derived from semi-objective outcomes of head to head comparisons: a log normal distribution with a geometric mean of τ2 of 0.04 and a 95% reference range from 0.001 to 1.58. Rate ratios were estimated from the median and corresponding 95% credibility intervals from the 2.5th and 97.5th centiles of the posterior distribution. Convergence was deemed to be achieved if plots of the Gelman-Rubin statistics indicated that widths of pooled runs and individual runs stabilised around the same value and their ratio was around 1. (83)
Elaboration
Many network meta-analyses to date have used Bayesian methods for 2 reasons. First, much of the initial development of the technique (as well as related software) used a Bayesian approach. Second, Bayesian methods are often practical in complex or sparse data problems when non-Bayesian (frequentist) methods are not. Recently, statisticians have implemented non-Bayesian techniques in statistical software packages, such as Stata and R (84, 85). It is important to justify the assumptions made for the analyses for the inferential method used.
Regardless of the chosen approach, it is important to check that the model fits the data well. Bayesian models often make use of the deviance information criterion to compare models and assess overall goodness of fit (86). Non-Bayesian models often use hypothesis tests based on deviance statistics. Users of Bayesian models must describe and justify the prior distributions used and describe the method by which they checked for convergence of the Markov chain if using a Markov-chain Monte Carlo simulation of the posterior distribution (86, 87). Authors are also encouraged to report on additional considerations, including whether arm-based or contrast-based analyses are used, whether study effects are considered to be fixed or random, and so forth.
Item S2 (New Item): Assessment of Inconsistency
When performing a network meta-analysis, we rely on the assumption of consistency of treatment effects (that is, the equivalency of treatment effects from direct and indirect evidence [Appendix Box 3]) across the different comparisons in the network.
Example
Consistency was mainly assessed by the comparison of the conventional network meta-analysis model, for which consistency is assumed, with a model that does not assume consistency (a series of pairwise meta-analyses analysed jointly). If the trade-off between model fit and complexity favoured the model with assumed consistency, this model was preferred. Moreover, we calculated the difference between direct and indirect evidence in all closed loops in the network; inconsistent loops were identified with a significant (95% CrI that excludes 0) disagreement between direct and indirect evidence. A loop of evidence is a collection of studies that links treatments to allow for indirect comparisons; the simplest loop is a triangle formed by three direct comparison studies with shared comparators. (88)
Elaboration
It is generally recommended to evaluate the consistency assumption by using both global and local approaches (Appendix Box 3). At the network level, one can check this assumption statistically by fitting a pair of related network meta-analysis models and comparing how well they fit to the data: one analysis wherein the model assumes consistency of direct and indirect evidence, and a second where the model does not make this assumption. Deviance information criteria can be used as mentioned earlier to consider model fit (19). If the models have a similar fit to the data, one can argue that consistency seems to hold.
To judge local consistency for particular contrasts of interventions that are part of a closed loop, one can use the method of Bucher and colleagues (89), or the node (or edge) splitting models presented by Dias and associates (16). The method of Bucher and colleagues assesses inconsistency in every available closed loop in the network separately and tests whether differences in treatment effects from direct and indirect evidence are present. Providing readers with a description of findings from an investigation to explore for inconsistency in the treatment network is important in order to shed light on the appropriateness of the assumption of consistency of evidence, which has implications for determining strength of confidence in the overall findings.
Item 15: Risk of Bias Across Studies
Guidance from the PRISMA statement applies. Authors should describe efforts taken to assess the risk of bias of included studies that may affect the cumulative evidence under study. Classical methods used to assess the risk of bias of included studies, such as use of the Cochrane Risk of Bias Scale, remain relevant and should be considered for each pairwise comparison in the treatment network; traditional approaches to presenting this information, as well as emerging approaches, such as color representation of bias in network diagrams (90), are possible. Given the complex structure of a network, identification of publication bias is more complex in a network meta-analysisowing to limited numbers of studies for each pairwise comparison, heterogeneity, and other limitations. Methods have been proposed that extend tests: for example, asymmetry testing and excess significance from pairwise meta-analyses also in the network space. The applicability of tests that evaluate the entire network has to be carefully considered in each network (91–93).
Item 16: Additional Analyses
Addition
The PRISMA statement notes that authors should describe all additional analyses that are performed to elucidate the robustness of primary findings, including meta-regressions, subgroup analyses, and sensitivity analyses. These and other efforts undertaken to establish the robustness of findings of a network meta-analysisshould be described.
Examples
We considered how decisions to group glaucoma treatments could affect the transitivity assumption and interpretation of the analysis. (27) [See Appendix Figure 1.]
Example of alternative geometries of a treatment network for the treatment of glaucoma based on the splitting (A) versus lumping (B) of treatment regimens in the treatment network. A sensitivity analysis considering alternative geometries should be considered when lumping treatment nodes. Depending on quantity, results may be best in appendices. APRAC = apraclonidine; BETAX = betaxolol; BIMAT = bimatoprost;BRIM = brimonidine; BRIN = brinzolamide; CART = carteolol; DOR = dorzolamide; NO TRT = no treatment; NR = not reported; LATAN = latanoprost; LEVO = levobudolol; PL = placebo; TIMO = timolol; TRAV = travoprost.
We a priori had selected allocation concealment, assessor blinding, treatment fidelity and imputation of numbers of responders as potentially important effect modifiers to be examined in sensitivity analyses to limit the included studies to those at low risk of bias. We conducted additional meta-regression analyses using random effects network meta-regression models to examine potential effect moderators such as the mean age of participants, the type of rating scales (clinician-rated versus self-rated), publication status (published versus dissertation), and therapy format (individual vs group). (94)
Random effects network meta-analyses with informative priors for heterogeneity variances were conducted for the analyses. We also conducted fixed and random effects models with vague priors. (95)
Elaboration
Various types of sensitivity analyses may be conducted to study the robustness of findings from a network meta-analysis. For example, network meta-analysis may be conducted by using alternative formulations of the treatment network, as in the example above. These analyses may potentially change clinical interpretations. If analyzed with Bayesian models, results may be sensitive to the specification of prior distributions, particularly for variance parameters (39). Sensitivity and subgroup analyses, as well as meta-regression models adjusting for covariates (34), also can affect findings. These alternative models should be described and the sensitivity of results to them reported. Although these analyses should be noted in the main text, the results may, if extensive, need to be reported in supplements.
The treatments of interest in a network meta-analysisshould be specified a priori. However, peripheral treatments may be included if, for example, they are a standard reference treatment not of direct interest that can connect an otherwise sparse network. Empirical evidence suggests that inclusion or exclusion of treatment nodes can affect estimates and treatment rankings (32).
Unless there is a clinical or analytical requirement in reference to the PICOS summary of the research question, the primary analysis should be restricted to specific doses of treatments and cotreatments. This is because lumping of different doses or cotreatments can introduce heterogeneity and inconsistency (96). However, as described earlier where a class effect may be considered, or different doses are considered to have the same efficacy, sensitivity analyses should be reported that take into account the alternative geometries.
Meta-regression analyses and subgroup analyses represent commonly used approaches to evaluating the effect of potential effect modifiers in traditional meta-analyses and remain applicable for network meta-analyses. The existing literature surveys methods for performing meta-regression analyses by using study-level covariates in network meta-analysis(20, 44), whereas subgroup analyses addressing the effect of effect modifiers, such as study-specific risk of bias (for example, low versus moderate to high risk of bias) or date of publication (for example, publication before versus after a particular year of interest), can be performed by repeating the analysis after limiting the network to include only studies meeting the criteria of interest.
Bayesian analyses should address choice of the prior distribution by reporting sensitivity analyses, particularly for variance parameters, which often have a large effect on results (39, 97).
Results
Item 17: Study Selection
As noted in the original PRISMA statement, there should be clear specification of the number of studies screened from the literature search, screened for eligibility from full-text reports, and subsequently included in the systematic review, with a corresponding flow diagram to summarize the study selection process
Item S3 (New Item): Presentation of Network Geometry
A network meta-analysis comparing all interventions of interest forms a network of treatments that are connected to each other on the basis of the pattern of comparisons made among the trials included in the review. The treatment comparisons for which trial data exist for an outcome of interest should be presented and summarized in a graph that enables readers to easily appraise the structure of existing evidence.
Example
Appendix Figure 2 shows a network graph comparing antipsychotic agents for prevention of schizophrenia relapse (12).
The size of treatment nodes reflects the number of patients randomly assigned to each treatment. The thickness of edges represents the number of studies underlying each comparison.
Elaboration
Figure 1 shows a generic example of a network graph that introduces its use as a visualization tool. The network graph in Appendix Figure 2 shows the evidence base comparing 9 treatments for prevention of relapse of schizophrenia (12). As mentioned earlier, the size of the treatment nodes reflects the proportionate numbers of patients randomly assigned to each of the treatments, whereas edge thickness indicates the number of studies supporting each comparison. Such visualizations can be generated by using statistical software, such as Stata and R (90), and can provide readers with insights on the evidence base under study (that is, the network geometry); these insights are discussed in items S1 and S4 and Appendix Box 4. It is optimal to illustrate these figures with as few overlapping lines as possible in order to facilitate interpretations regarding the network geometry. Network graphs can provide insight into parts of the evidence base that are informed by small versus large amounts of data, and thus can inform the consideration of interventions that may benefit from further research in terms of accumulating additional evidence.
In cases where the network is small (for example, networks of 3 treatments for which data are present for all comparisons), provision of a table of the data and a short narrative description may be sufficient. Proportionate sizing of nodes and edges in a network diagram may not be desirable in cases where there are large divergences in the numbers of patients and studies across interventions, because they may produce network graphs that are difficult to interpret.
Item S4 (New Item): Summary of Network Geometry
Provide a summary of the structure of the evidence base constructed from study selection.
Example
A total of 2,545 pulmonary hypertension patients received active pulmonary hypertension medication. The studied agents were more commonly bosentan (n = 13 trials; patients receiving treatment = 633) and sildenafil (n = 13 trials; patients receiving treatment = 593). Placebo was used as the comparator arm in 38 studies (patients receiving placebo = 1,643). Of the patients that received placebo, 52 participants were part of crossover studies with sildenafil. The most frequently used comparisons were bosentan versus placebo (n = 11) and sildenafil versus placebo (n = 11). Studies that used placebo as the comparator arm (n = 38) were for the most part sponsored by the pharmaceutical company that owned the product (n = 28 studies [74%]). The only two published head-to-head comparisons of different medications (sildenafil against bosentan) were not sponsored by pharmaceutical companies, but by the British Heart Foundation and the Italian Health Authority. (70)
Elaboration
Such geometry features as identification of a lack of information in relation to specific treatments and comparisons in the network should be described. Evaluations of network geometry may suggest specific biases related to the choice of treatments to be tested, their preferred (or avoided) comparisons, the effect of sponsoring on the selection of treatments and comparisons, and other biases that might affect the geometry of the network. These biases may have important implications for the strength of interpretation of the evidence.
Authors may choose whether to report specific measures of geometry described in Appendix Box 4. The graphical presentation of a network (for example, Appendix Figure 3 ) can be supplemented with a table (or text) describing the number of patients, number of studies, and number of events for each comparison or node. In instances where there are low numbers of events or low power, results should be interpreted with caution (38). In these instances, alternative network configurations may be considered (for example, lumping of interventions). Additional empirical work to clarify the role of network structure for interpreting findings from network meta-analyses is likely to be helpful and may lead to more specific reporting guidance in the future.
Each intervention is shown by a circular node, with the same color used to group interventions which belong to the same drug class. An auto-loop represents studies where different doses of the same medication have been compared. IV = intravenous; SC = subcutaneous.
Item 18: Study Characteristics
As reflected in the PRISMA statement, authors should present the characteristics of all included trials (PICOS-related information, study time frame, sample size, patient demographics) in the systematic review. This still applies to reviews that evaluate a network of treatments. This is commonly accomplished through both a summary in the main text and tables that provide detailed information for all included studies. Authors may wish to structure information tables by using subheadings such that subgroups of trials included in the treatment network are presented together (for example, all A versus B trials, then all A versus C trials). Authors should especially try to report effect modifiers collected to monitor for variations in treatment effects that may have arisen owing to broad time frames of research, because these may be particularly important in judging the appropriateness of the transitivity assumption.
Because systematic reviews incorporating network meta-analyses will often include data from studies of many different comparisons and many studies, authors should plan to make use of supplemental appendices (as described in a section below) in order to provide readers with adequate information for review of study characteristics.
Item 19: Risk of Bias Within Studies
As outlined in the PRISMA statement, we recommend that findings from risk of bias assessment of the included studies be reported at the level of the individual study, and not only in terms of aggregate counts of studies at lower or higher risk of bias. A summary of bias assessments presented in a table or graph format remains most convenient, and because network meta-analyses commonly include a large number of studies, this may be most simply summarized in an online supplemental appendix to the main report. An additional consideration may be to also present a network graph incorporating risk of bias coloring, as is commonly used in Cochrane systematic reviews (for example, where green indicates low risk of bias, red high risk of bias, and yellow unclear risk of bias) to demonstrate the perceived level of risk within different parts of the treatment network (90).
Item 20: Results of Individual Studies
Addition
The PRISMA statement recommends that for each outcome studied, the summary outcome data for each study's intervention groups (such as number of events and sample size for binary outcomes, and mean, standard deviation, and sample size for continuous outcomes) be provided. Use of a forest plot is recommended as ideal for traditional meta-analyses. Some modifications are needed for systematic reviewsincorporating a network meta-analysis.
Example
The Appendix Table presents an example of one possible approach to provision of data on mortality observed with five different interventions for treatment of left ventricular dysfunction (medical resynchronisation, cardiac resynchronisation, implantable defibrillator, combined resynchronisation and defibrillator, and amiodarone) as described elsewhere (98).
 |
Elaboration
For network meta-analysesin which many studies and many treatments may be considered, provision of outcome data at the study level in a forest plot or table in the main text may be unwieldy. Authors may alternatively report this information in one of several possible formats by using an online supplemental Web appendix (see the section on this topic below). This could include a table of data by study, provision of the data sets used for network meta-analyses (as shown in the example), or provision of forest plots that may have been prepared to study information within each of the edges of the treatment network. The sample tabular approach presented in the Appendix Table is intuitive; however, it can be inconvenient when dealing with many treatments or when outcomes are not counts, and varied approaches may be required.
Item 21: Synthesis of Results
Addition
The PRISMA statement advocates reporting of the main results of the review, including findings from meta-analyses and the corresponding measures of heterogeneity. This guidance applies to reviews incorporating network meta-analyses, although some additions beyond conventional practice for pairwise meta-analysis are needed given the potentially sizable increase in the amount of data to present.
Example
Two examples of reporting of comparative treatment efficacy from a review comparing efficacy of treatments for multiple sclerosis with regard to progression of disability are presented in Appendix Figures 4 and 5(82).
(lower triangle) and direct estimates (upper triangle) of efficacy (disability progression over 36 months) of immunomodulators and immunosuppressants for multiple sclerosis.
Treatments are reported in order of relative ranking for efficacy. Comparisons between treatments should be read from left to right, and their odds ratio is in the cell in common between the column-defining treatment and the row-defining treatment. Odds ratios less than 1 favor the column-defining treatment for the network estimates and the row-defining treatment for the direct estimates. IFN = interferon.
Summary estimates are reported for only a subset of all possible pairwise comparisons, namely active interventions versus placebo. Treatments are ranked according to their surface under the cumulative ranking values. OR = odds ratio; CrI = credible interval; IFN =interferon.
Elaboration
Reviews comparing 2 interventions commonly contain forest plots which present 1) the summary measures of effect for each included trial, and 2) the summary measure of effect generated by meta-analyzing data from the included trials (supplemented with outcome data from each trial, I2 values quantifying statistical heterogeneity between study-level summary measures, and so forth). Network meta-analyses may include large numbers of treatments (and thus many pairwise comparisons to summarize) as well as studies (and thus a burdensome number of study-level summaries to present).
For these reasons, forest plots often summarize findings from a network meta-analysis inefficiently. Instead, a large number of treatment comparisons may require 1) an alternative visual, such as a league table (a tabular approach used to succinctly present all possible pairwise comparisons between treatments, as shown in Appendix Figure 4) or 2) emphasis on a subset of all possible treatment comparisons in forest plots or other graphs of summary estimates (Appendix Figure 5). A network meta-analysis may focus on reporting odds ratios of a specific new intervention of interest versus all older interventions, or on comparisons of each active intervention against placebo.
Finally, the challenge of summarizing comparative efficacy and safety succinctly between multiple interventions has popularized the use of supplementary measures in the form of treatment rankings and relative probabilities of superiority (36). Appendix Figure 6 presents examples of tabular and graphical approaches to summarizing such information. Simultaneous presentation of treatment hierarchy (as based on ranking measures) and summary effect measures may be considered as the most appropriate way of reporting the 2 outputs. League tables (Appendix Figure 4) containing the competing treatments in the diagonal cells can be used and treatments can be ordered according to their hierarchy for the respective outcome. These figures can be challenging to interpret, and it is recommended that authors provide a clear description when used to maximize transparency to readers.
top) and graphical ( bottom) reporting of treatment rankings regarding comparison of treatment-associated risks of grade 3 or 4 hematologic toxicities for resected pancreatic adenocarcinoma.
Rankings nearer 1 suggest greater risk. 5-FU = 5-fluorouracil.
In addition, forest plots showing the summary effects of all treatments versus a common reference intervention can be generated in a manner such that they provide information on the relative ranking of treatments—for example, by organizing the order in which comparisons are presented to correspond to the values of a measure, such as the surface under the cumulative ranking curve for each treatment (Appendix Figure 5). Full relative ranking results (such as the estimated ranking probabilities for all treatments) may be reported as supplementary material. Tan and colleagues (99) present additional considerations for the presentation of findings from network meta-analysis.
Item S5 (New Item): Exploration for Inconsistency
Provide a description of findings from investigations performed to assess for the presence of inconsistency in the evidence base analyzed.
Example
The assumption of consistency was generally supported by a better trade-off between model fit and complexity when consistency was assumed than when it was not. Significant disagreement between direct and indirect estimates (inconsistency) was identified in only very few cases: for efficacy seven of 80 loops; for all-cause discontinuation three of 80 loops; for weight gain one of 62 loops; for extrapyramidal side-effects one of 56 loops; for prolactin increase three of 44 loops; for QTc prolongation two of 35 loops; and for sedation none of 49 loops were inconsistent (appendix pp 105-14). Data were double-checked and we could not identify any important variable that differed across comparisons in these loops. The number of included studies in the inconsistent loops was typically small, so the extent of inconsistency was not substantial enough to change the results. (88)
Elaboration
The approach to presenting inconsistency results depends on the method used to evaluate inconsistency. Results of global approaches (Appendix Box 5) are usually summarized in a specific value, which can be the P value of a chi-square test from a chosen model (such as a design-by-treatment model [21], Q test for inconsistency [101], or composite test for inconsistency [102]), the value of the I2measure for inconsistency, the difference in a measure of model fit or parsimony between consistency and inconsistency model (19), or the magnitude of inconsistency variance (random inconsistency models [22, 28]). Such values can be reported in tables or graphs that are primarily used to present the summary effects from a network meta-analysis. Local approaches (including the loop-specific approach [103], node-splitting (16), or "net-heat" approach [104]) require the presentation of inconsistency estimates for each different evaluated part of the network, which can result in huge tables or graphs (such as forest or matrix plots), particularly in the case of large networks.
One option for a more concise presentation would be to show the inconsistency results only for loops or comparisons that might be possible sources of inconsistency on the basis of findings from statistical tests. Review authors are recommended to consider both global and local methods for the evaluation of inconsistency. More detailed reporting of findings from use of local approaches to explore for inconsistency should be included in the supplementary material.
Item 22: Risk of Bias Across Studies
Guidance from the PRISMA statement remains applicable. Authors are recommended to present the results of any assessments made to explore the potential for risk of bias across included studies (see item 15). The incorporation of risk of bias assessments (and their effect across a network of treatments) into judgment of the strength, credibility, and interpretability of findings from a network meta-analysisis an area of current research. Recent publications have included efforts to achieve this objective (56), and works describing approaches to integrate strength of evidence have appeared in the literature (51, 105).
Item 23: Results of Additional Analyses
Addition
The PRISMA statement suggests describing results obtained from additional subgroup analyses, sensitivity analyses, meta-regression analyses, or different models (for example, fixed versus random effects) that were performed as part of the systematic review. This remains applicable for network meta-analyses but may include additional considerations, such as alternative geometry.
Example: Alternative Network Geometry
Standard adjusted dose vitamin K agonist (VKA) (odds ratio 0.11 (95% credible interval 0.04 to 0.27)), dabigatran, apixaban 5 mg, apixaban 2.5 mg, and rivaroxaban decreased the risk of recurrent venous thromboembolism, compared with ASA [acetylsalicylic acid]. Compared with low dose VKA, standard adjusted dose VKA reduced the risk of recurrent venous thromboembolism (0.25 (0.10 to 0.58)).
An appendix presents a detailed explanation for the potential discrepancy between ASA and placebo results. Results for most class level analyses also aligned with those reported previously in the treatment level analysis. Subgroup analyses, performed to account for heterogeneity due to study duration, yielded results that were more favourable for ASA than those obtained from the primary analysis. However, results for ASA were still less pronounced than those reported for other treatments (standard adjusted dose VKA, low intensity VKA, and dabigatran) that remained in the evidence network. Sensitivity analysis excluding ximelagatran from the analysis did not change the results reported. (95 )
Example: Subgroup Analysis
Table 2 presents an investigation into potential sources of variation in people with diabetes in the network. Estimates of relative risk comparing sirolimus eluting stents with paclitaxel eluting stents depended to some extent on the quality of the trials, the length of followup, and the time of completion of patient recruitment (table 2), but 95% credibility intervals were wide and tests for interaction negative (P for interaction ≥0.16). The estimated relative risk of death when sirolimus eluting stents were compared with bare metal stents was greater when the specified duration of dual antiplatelet therapy was less than six months (2.37, 95% credibility interval 1.18 to 5.12) compared with six months or longer (0.89, 0.58 to 1.40, P for interaction 0.02), however. (106)
Example: Meta-regression Analysis
None of the regression coefficients of the meta-regression examining possible effect moderators turned out to be statistically significant [-0.024 (95% CI -0.056 to 0.006) for age, -0.899 (95% CI -1.843 to 0.024 for rating scale), -0.442 (95% CI -1.399 to 0.520) for publication status, and 0.004 (95% CI -0.798 to 0.762) for therapy format]. (94)
Elaboration
Performance of additional analyses retains an important role in establishing the robustness of findings from any meta-analysis. This includes consideration of various ways to structure the treatment network (such as lumping and splitting in relation to dose levels versus any exposure, method of administration, or exclusion of certain doses), accounting for the effect of covariates on summary effect measures (such as meta-regression or subgroup analysis), use of different statistical models (especially involving a Bayesian approach, where different prior distributions may be chosen), and so forth. Authors are encouraged to report findings from such analyses so that readers have all available information for judging robustness of primary findings. Use of supplemental appendices to the main text may be required to present this information.
Selection of a statistical model for network meta-analyses where comparisons between treatments are largely based on single studies can also represent a challenge. We refer readers to the appendices of a recent review of antithrombotic agents, which illustrate a possible approach to reporting results when dealing with such a challenge (107).
Discussion
Item 24: Summary of Evidence
The PRISMA statement recommends that authors provide a summary of the main findings obtained from the review with regard to each outcome assessed, and that this be done in a way that reflects consideration of the review's key audiences, including clinicians, researchers, and policymakers. This guidance remains entirely applicable to the reporting of network meta-analyses. As with traditional systematic reviews, mention of how findings are similar to or different from past network meta-analyses can be helpful for readers and is encouraged.
Item 25: Limitations
Addition
The PRISMA statement recommends referral to limitations at the level of individual studies and outcomes in the review (including risk of bias concerns), as well as the review level. This guidance remains applicable in the context of network meta-analysis, with some potential modifications to address the nuances associated with network meta-analysis. A recent example studying pharmacotherapies for schizophrenia addresses such a collection of items (88).
Example
Our study has several limitations. The network could be expanded to old drugs such as perphenazine and sulpiride, which have had good results in effectiveness studies, but only a few relevant perphenazine trials have been done.
Reporting of side-effects is unsatisfactory in randomised controlled trials in patients with psychiatric disorders, and some side-effects were not recorded at all for some drugs. The meta-regression with percentage of withdrawals as a moderator could not rule out all potential bias associated with high attrition in schizophrenia trials.
Our findings cannot be generalised to young people with schizophrenia, patients with predominant negative symptoms, refractory patients, or stable patients, all of whom were excluded to enhance homogeneity as required by multiple-treatments meta-analysis. A funnel plot asymmetry was seen, which is not necessarily the expression of publication bias, but rather of higher efficacy in small trials than in larger ones, for various reasons. For example, sample size estimates for drugs with low efficacy might have needed higher numbers of participants to attain statistical significance than in trials with more effective drugs. However, accounting for trial size did not substantially change the rankings. Finally, because multiple-treatments meta-analysis requires reasonably homogeneous studies, we had to restrict ourselves to short-term trials. Because schizophrenia is often a chronic disorder, future multiple-treatments meta-analyses could focus on long-term trials, but these remain scarce. In any case, for clinicians to know to which drugs patients are most likely to respond within a reasonable duration such as 6 weeks is important. (88)
Elaboration
The risk for violating the assumption of transitivity may be increased in network meta-analyses when dealing with larger treatment networks or broad variation in dates of study performance (which may reflect important changes in co-medication use, improved expertise in disease management, modifications of diagnostic criteria or disease severity, or other factors). It is helpful for readers when the study authors provide insight on such information. Important considerations resulting from quantitative explorations for inconsistency of direct and indirect information should be noted; identified sources of inconsistencies and efforts taken to resolve them should be noted. Authors should also mention important changes in findings that may be related to sensitivity analyses, such as meta-regressions or modifications of the network structure. Weaknesses of the evidence base that informed data analyses (for example, limited amounts of information from head-to-head trials, or high risk of bias for particular edges or comparisons within the network) are also worthy of mention. Subtle or moderate changes in characteristics of study populations that may have implications regarding to whom results may apply should also be noted.
Item 26: Conclusions
The PRISMA statement's guidance proscribes stating an overall interpretation of the review's results while considering other related evidence, as well as a brief mention of the review's implications for future research. This guidance remains applicable for reviews including network meta-analyses.
Item 27: Funding
Sources of funding and related conflicts of interest should be stated, along with information about the involvement of funders, if any, in the design, analysis, and publication of the network meta-analysis. Traditional meta-analyses have long been influential tools for decision making and policy. Therefore, it is not surprising that potentially conflicted stakeholders may fund meta-analyses, and this remains a consideration for network meta-analyses.
There is evidence that industry-sponsored meta-analyses tend to have more favorable conclusions than other meta-analyses (108, 109). Therefore, it is essential that reports summarizing reviews of networks of treatments describe in detail both their funding and any related potential conflicts of interest, and explain whether funders had any involvement in study design, analysis or interpretation of the results, drafting of the manuscript, or the decision to publish the results.
Use of Supplemental Appendices for Complete Reporting of Network Meta-analyses
Supplemental appendices are key tools for aiding reproducible research and ensuring transparent reporting of network meta-analyses. Given the nature of questions they address, reports of network meta-analyses often contain large amounts of information on methods used, evidence studied, and results produced. Transparent reporting of the data and the steps underpinning a network meta-analysis can thus be challenging. Journals often have limits on word counts for the text and on the number of tables and figures that may be included, and desire that information be distilled for their readership. They may require a "palatable" presentation focusing on main findings rather than on detailed reporting of data underpinning the review and explanation of the statistical modeling techniques used.
Throughout this guidance, we have noted areas where authors might present information in supplements (for example, partial versus full reporting of summary estimates, study characteristics, or explorations of heterogeneity). Although we have attempted to provide comprehensive guidance on reporting network meta-analyses and we feel that the highlighted elements are needed to maximize their transparency, there will probably be a need to distribute this information between the main text and supplements differently, depending on the target journal. We suggest that readers consult good examples of reviews balancing reporting between main text and data supplements when considering the reporting of their own network meta-analyses. Future updates are likely to include further discussion on this aspect of presenting reviews that incorporate networks of treatments.
Software for Implementing Network Meta-analysis
Several software packages are available for implementing network meta-analysis. The choice of software package will depend on the statistical method under consideration.
For Bayesian statistics, WinBUGS (Imperial College and Medical Research Council, London, United Kingdom) (110) is the most widely used software package, although JAGS (111) and OpenBUGS can also be used. The NICE Decision Support Unit (112) published a series of technical support documents with code for conducting network meta-analysis for various outcomes within a Bayesian framework.
The Web sites of the Multi-Parameter Evidence Synthesis Group (113) and the IMMA project (University of Ioannina, Ioaninna, Greece) (114) also provide code that can be used to perform network meta-analysis in WinBUGS, OpenBUGS, or JAGS. These packages can be used directly or indirectly via widely general purpose software, such as R (R Foundation for Statistical Computing, Vienna, Austria) (115–118), STATA (Stata Corp., College Station, Texas) (116), SAS (SAS Institute, Cary, North Carolina) (116), or Microsoft Excel (Microsoft Corp., Seattle, Washington) (116), or via specialized software packages, such as ADDIS (ADDIS, Groningen, the Netherlands) (117–119).
For frequentist statistics, network meta-analyses can be conducted by using R (83, 120), STATA (121, 122), or SAS (26), whereas simple indirect comparisons can be conducted using the CADTH Indirect Treatment Comparison calculator (Canadian Agency for Drugs and Technologies in Health, Ottawa, Ontario, Canada) (123, 124). In addition to the statistical packages mentioned above, complementary software packages for developing graphical tools for network meta-analysis(90) and evaluating inconsistency ( 104) are available.
Example Wording for Endorsing This PRISMA Extension
[Journal name] requires a completed PRISMA 2015 network meta-analysis checklist as a condition of submission when reporting the results of a network meta-analysis. Templates can be found at [give hyperlink to location if relevant] or on the PRISMA Web site www.prisma-statement.org, which also describes other PRISMA extensions. You should ensure that your article, at minimum, reports content addressed by each item of the checklist. Meeting these basic reporting requirements will greatly improve the value of your network meta-analysis report and may enhance its chances for publication.
References
- 1.
Moher D ,Cook DJ ,Eastwood S ,Olkin I ,Rennie D ,Stroup DF . Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet. 1999;354:1896-900. [PMID: 10584742] CrossrefMedlineGoogle Scholar - 2.
Moher D ,Liberati A ,Tetzlaff J ,Altman DG ;PRISMA Group . Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;151:264-9. [PMID: 19622511] LinkGoogle Scholar - 3.
Liberati A ,Altman DG ,Tetzlaff J ,Mulrow C ,Gøtzsche PC ,Ioannidis JP ,et al . The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Ann Intern Med. 2009;151:W65-94. [PMID: 19622512] LinkGoogle Scholar - 4.
Panic N ,Leoncini E ,deBelvis G ,Ricciardi W ,Boccia S . Evaluation of the endorsement of the preferred reporting items for systematic reviews and meta-analysis (PRISMA) statement on the quality of published systematic review and meta-analyses. PLoS One. 2013;8:e83138. [PMID: 24386151] doi:10.1371/journal.pone.0083138 CrossrefMedlineGoogle Scholar - 5.
Wen J ,Ren Y ,Wang L ,Li Y ,Liu Y ,Zhou M ,et al . The reporting quality of meta-analyses improves: a random sampling study. J Clin Epidemiol. 2008;61:770-5. [PMID: 18411041] doi:10.1016/j.jclinepi.2007.10.008 CrossrefMedlineGoogle Scholar - 6.
Lu G ,Ades AE . Combination of direct and indirect evidence in mixed treatment comparisons. Stat Med. 2004;23:3105-24. [PMID: 15449338] CrossrefMedlineGoogle Scholar - 7.
Ioannidis JP . Integration of evidence from multiple meta-analyses: a primer on umbrella reviews, treatment networks and multiple treatments meta-analyses. CMAJ. 2009;181:488-93. [PMID: 19654195] doi:10.1503/cmaj.081086 CrossrefMedlineGoogle Scholar - 8.
Lee AW . Review of mixed treatment comparisons in published systematic reviews shows marked increase since 2009. J Clin Epidemiol. 2014;67:138-43. [PMID: 24090930] doi:10.1016/j.jclinepi.2013.07.014 CrossrefMedlineGoogle Scholar - 9.
Nikolakopoulou A ,Chaimani A ,Veroniki AA ,Vasiliadis HS ,Schmid CH ,Salanti G . Characteristics of networks of interventions: a description of a database of 186 published networks. PLoS One. 2014;9:e86754. [PMID: 24466222] doi:10.1371/journal.pone.0086754 CrossrefMedlineGoogle Scholar - 10.
Hutton B ,Salanti G ,Chaimani A ,Caldwell DM ,Schmid C ,Thorlund K ,et al . The quality of reporting methods and results in network meta-analyses: an overview of reviews and suggestions for improvement. PLoS One. 2014;9:e92508. [PMID: 24671099] doi:10.1371/journal.pone.0092508 CrossrefMedlineGoogle Scholar - 11.
Moher D ,Schulz KF ,Simera I ,Altman DG . Guidance for developers of health research reporting guidelines. PLoS Med. 2010;7:e1000217. [PMID: 20169112] doi:10.1371/journal.pmed.1000217 CrossrefMedlineGoogle Scholar - 12.
Ades AE ,Mavranezouli I ,Dias S ,Welton NJ ,Whittington C ,Kendall T . Network meta-analysis with competing risk outcomes. Value Health. 2010;13:976-83. [PMID: 20825617] doi:10.1111/j.1524-4733.2010.00784.x CrossrefMedlineGoogle Scholar - 13. Ades AE, Caldwell D, Reken S, Welton N, Sutton A, Dias S. NICE DSU Technical Support Document 7: Evidence Synthesis of Treatment Efficacy in Decision Making: A Reviewer's Checklist. London: National Institute for Health and Care Excellence; 2012. Accessed at www.nicedsu.org.uk/TSD7%20reviewer%20checklist.final.08.05.12.pdf on 28 February 2015. Google Scholar
- 14.
Caldwell DM ,Ades AE ,Higgins JP . Simultaneous comparison of multiple treatments: combining direct and indirect evidence. BMJ. 2005;331:897-900. [PMID: 16223826] CrossrefMedlineGoogle Scholar - 15.
Del Giovane C ,Vacchi L ,Mavridis D ,Filippini G ,Salanti G . Network meta-analysis models to account for variability in treatment definitions: application to dose effects. Stat Med. 2013;32:25-39. [PMID: 22815277] doi:10.1002/sim.5512 CrossrefMedlineGoogle Scholar - 16.
Dias S ,Welton NJ ,Caldwell DM ,Ades AE . Checking consistency in mixed treatment comparison meta-analysis. Stat Med. 2010;29:932-44. [PMID: 20213715] doi:10.1002/sim.3767 CrossrefMedlineGoogle Scholar - 17.
Dias S ,Welton N ,Marinho V ,Salanti G ,Ades A . Estimation and adjustment of bias in randomised evidence using mixed treatment comparison meta-analysis. J R Stat Soc Ser A. 2010;173:613-29. CrossrefGoogle Scholar - 18.
Dias S ,Sutton AJ ,Ades AE ,Welton NJ . Evidence synthesis for decision making 2: a generalized linear modeling framework for pairwise and network meta-analysis of randomized controlled trials. Med Decis Making. 2013;33:607-17. [PMID: 23104435] doi:10.1177/0272989X12458724 CrossrefMedlineGoogle Scholar - 19.
Dias S ,Welton NJ ,Sutton AJ ,Caldwell DM ,Lu G ,Ades AE . Evidence synthesis for decision making 4: inconsistency in networks of evidence based on randomized controlled trials. Med Decis Making. 2013;33:641-56. [PMID: 23804508] doi:10.1177/0272989X12455847 CrossrefMedlineGoogle Scholar - 20.
Dias S ,Sutton AJ ,Welton NJ ,Ades AE . Evidence synthesis for decision making 3: heterogeneity—subgroups, meta-regression, bias, and bias-adjustment. Med Decis Making. 2013;33:618-40. [PMID: 23804507] doi:10.1177/0272989X13485157 CrossrefMedlineGoogle Scholar - 21.
Higgins JPT ,Jackson D ,Barrett J ,Lu G ,Ades A ,White I . Consistency and inconsistency in network meta-analysis: concepts and models for multi-arm studies. Res Synth Methods. 2012;3:98-110. CrossrefMedlineGoogle Scholar - 22.
Jackson D ,Barrett JK ,Rice S ,White IR ,Higgins JP . A design-by-treatment interaction model for network meta-analysis with random inconsistency effects. Stat Med. 2014;33:3639-54. [PMID: 24777711] doi:10.1002/sim.6188 CrossrefMedlineGoogle Scholar - 23.
Jansen JP ,Cope S . Meta-regression models to address heterogeneity and inconsistency in network meta-analysis of survival outcomes. BMC Med Res Methodol. 2012;12:152. [PMID: 23043545] doi:10.1186/1471-2288-12-152 CrossrefMedlineGoogle Scholar - 24.
Jansen J . Network meta-analysis of individual and aggregate level data. Res Synth Methods. 2012;3:177-90. CrossrefMedlineGoogle Scholar - 25.
Jansen JP ,Naci H . Is network meta-analysis as valid as standard pairwise meta-analysis? It all depends on the distribution of effect modifiers. BMC Med. 2013;11:159. [PMID: 23826681] doi:10.1186/1741-7015-11-159 CrossrefMedlineGoogle Scholar - 26.
Jones B ,Roger J ,Lane PW ,Lawton A ,Fletcher C ,Cappelleri JC ,et al ;PSI Health Technology Special Interest Group Evidence Synthesis sub-team . Statistical approaches for conducting network meta-analysis in drug development. Pharm Stat. 2011;10:523-31. [PMID: 22213533] doi:10.1002/pst.533 CrossrefMedlineGoogle Scholar - 27.
Lindsley K ,Cameron N ,Wormald R ,Li T ,Dickersin K . Evaluating the transitivity assumption when constructing network meta-analysis: lumping or splitting? Cochrane Library Supplement. Presented at the 21st Cochrane Colloquium, Quebec, Canada. 23 September 2013. Google Scholar - 28.
Lu G ,Ades A . Assessing evidence inconsistency in mixed treatment comparisons. J Am Stat Assoc. 2006;101:447-59. CrossrefGoogle Scholar - 29.
Lu G ,Ades A . Modeling between-trial variance structure in mixed treatment comparisons. Biostatistics. 2009;10:792-805. [PMID: 19687150] doi:10.1093/biostatistics/kxp032 CrossrefMedlineGoogle Scholar - 30.
Mills EJ ,Bansback N ,Ghement I ,Thorlund K ,Kelly S ,Puhan MA ,et al . Multiple treatment comparison meta-analyses: a step forward into complexity. Clin Epidemiol. 2011;3:193-202. [PMID: 21750628] doi:10.2147/CLEP.S16526 CrossrefMedlineGoogle Scholar - 31.
Thorlund K ,Mills E . Stability of additive treatment effects in multiple treatment comparison meta-analysis: a simulation study. Clin Epidemiol. 2012;4:75-85. [PMID: 22570567] doi:10.2147/CLEP.S29470 CrossrefMedlineGoogle Scholar - 32.
Mills EJ ,Kanters S ,Thorlund K ,Chaimani A ,Veroniki AA ,Ioannidis JP . The effects of excluding treatments from network meta-analyses: survey. BMJ. 2013;347:f5195. [PMID: 24009242] doi:10.1136/bmj.f5195 CrossrefMedlineGoogle Scholar - 33.
Salanti G ,Kavvoura FK ,Ioannidis JP . Exploring the geometry of treatment networks. Ann Intern Med. 2008;148:544-53. [PMID: 18378949] LinkGoogle Scholar - 34.
Salanti G ,Marinho V ,Higgins JP . A case study of multiple-treatments meta-analysis demonstrates that covariates should be considered. J Clin Epidemiol. 2009;62:857-64. [PMID: 19157778] doi:10.1016/j.jclinepi.2008.10.001 CrossrefMedlineGoogle Scholar - 35.
Salanti G ,Dias S ,Welton NJ ,Ades AE ,Golfinopoulos V ,Kyrgiou M ,et al . Evaluating novel agent effects in multiple-treatments meta-regression. Stat Med. 2010;29:2369-83. [PMID: 20687172] doi:10.1002/sim.4001 CrossrefMedlineGoogle Scholar - 36.
Salanti G ,Ades AE ,Ioannidis JP . Graphical methods and numerical summaries for presenting results from multiple-treatment meta-analysis: an overview and tutorial. J Clin Epidemiol. 2011;64:163-71. [PMID: 20688472] doi:10.1016/j.jclinepi.2010.03.016 CrossrefMedlineGoogle Scholar - 37.
Sutton A ,Ades AE ,Cooper N ,Abrams K . Use of indirect and mixed treatment comparisons for technology assessment. Pharmacoeconomics. 2008;26:753-67. [PMID: 18767896] CrossrefMedlineGoogle Scholar - 38.
Thorlund K ,Mills EJ . Sample size and power considerations in network meta-analysis. Syst Rev. 2012;1:41. [PMID: 22992327] doi:10.1186/2046-4053-1-41 CrossrefMedlineGoogle Scholar - 39.
Thorlund K ,Thabane L ,Mills EJ . Modelling heterogeneity variances in multiple treatment comparison meta-analysis—are informative priors the better solution? BMC Med Res Methodol. 2013;13:2. [PMID: 23311298] doi:10.1186/1471-2288-13-2 CrossrefMedlineGoogle Scholar - 40.
Veroniki AA ,Vasiliadis HS ,Higgins JP ,Salanti G . Evaluation of inconsistency in networks of interventions. Int J Epidemiol. 2013;42:332-45. [PMID: 23508418] doi:10.1093/ije/dys222 CrossrefMedlineGoogle Scholar - 41.
Mills EJ ,Ioannidis JP ,Thorlund K ,Schünemann HJ ,Puhan MA ,Guyatt GH . How to use an article reporting a multiple treatment comparison meta-analysis. JAMA. 2012;308:1246-53. [PMID: 23011714] CrossrefMedlineGoogle Scholar - 42.
Cipriani A ,Higgins JP ,Geddes JR ,Salanti G . Conceptual and technical challenges in network meta-analysis. Ann Intern Med. 2013;159:130-7. [PMID: 53236683] doi:10.7326/0003-4819-159-2-201307160-00008 LinkGoogle Scholar - 43.
Woods BS ,Hawkins N ,Scott DA . Network meta-analysis on the log-hazard scale, combining count and hazard ratio statistics accounting for multi-arm trials: a tutorial. BMC Med Res Methodol. 2010;10:54. [PMID: 20537177] doi:10.1186/1471-2288-10-54 CrossrefMedlineGoogle Scholar - 44.
Cooper NJ ,Sutton AJ ,Morris D ,Ades AE ,Welton NJ . Addressing between-study heterogeneity and inconsistency in mixed treatment comparisons: application to stroke prevention treatments in individuals with non-rheumatic atrial fibrillation. Stat Med. 2009;28:1861-81. [PMID: 19399825] doi:10.1002/sim.3594 CrossrefMedlineGoogle Scholar - 45.
Donegan S ,Williamson P ,D'Alessandro U ,TudurSmith C . Assessing the key assumptions of network meta-analysis: a review of methods. Res Synth Methods. 2013;4:291-323. CrossrefMedlineGoogle Scholar - 46.
Achana FA ,Cooper NJ ,Dias S ,Lu G ,Rice SJ ,Kendrick D ,et al . Extending methods for investigating the relationship between treatment effect and baseline risk from pairwise meta-analysis to network meta-analysis. Stat Med. 2013;32:752-71. [PMID: 22865748] doi:10.1002/sim.5539 CrossrefMedlineGoogle Scholar - 47.
Jansen JP ,Fleurence R ,Devine B ,Itzler R ,Barrett A ,Hawkins N ,et al . Interpreting indirect treatment comparisons and network meta-analysis for health-care decision making: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 1. Value Health. 2011;14:417-28. [PMID: 21669366] doi:10.1016/j.jval.2011.04.002 CrossrefMedlineGoogle Scholar - 48.
Hoaglin DC ,Hawkins N ,Jansen JP ,Scott DA ,Itzler R ,Cappelleri JC ,et al . Conducting indirect-treatment-comparison and network-meta-analysis studies: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 2. Value Health. 2011;14:429-37. [PMID: 21669367] doi:10.1016/j.jval.2011.01.011 CrossrefMedlineGoogle Scholar - 49.
Jansen JP ,Trikalinos T ,Cappelleri JC ,Daw J ,Andes S ,Eldessouki R ,et al . Indirect treatment comparison/network meta-analysis study questionnaire to assess relevance and credibility to inform health care decision making: an ISPOR-AMCP-NPC Good Practice Task Force report. Value Health. 2014;17:157-73. [PMID: 24636374] doi:10.1016/j.jval.2014.01.004 CrossrefMedlineGoogle Scholar - 50.
Chaimani A ,Salanti G . Using network meta-analysis to evaluate the existence of small-study effects in a network of interventions. Res Synth Methods. 2012;3:161-76. CrossrefMedlineGoogle Scholar - 51.
Salanti G ,DelGiovane C ,Chaimani A ,Caldwell DM ,Higgins JP . Evaluating the quality of evidence from a network meta-analysis. PLoS One. 2014;9:e99682. [PMID: 24992266] doi:10.1371/journal.pone.0099682 CrossrefMedlineGoogle Scholar - 52.
Glasziou P ,Meats E ,Heneghan C ,Shepperd S . What is missing from descriptions of treatment in trials and reviews? BMJ. 2008;336:1472-4. [PMID: 18583680] doi:10.1136/bmj.39590.732037.47 CrossrefMedlineGoogle Scholar - 53.
Simera I ,Moher D ,Hoey J ,Schulz KF ,Altman DG . A catalogue of reporting guidelines for health research. Eur J Clin Invest. 2010;40:35-53. [PMID: 20055895] doi:10.1111/j.1365-2362.2009.02234.x CrossrefMedlineGoogle Scholar - 54.
Moher D ,Weeks L ,Ocampo M ,Seely D ,Sampson M ,Altman DG ,et al . Describing reporting guidelines for health research: a systematic review. J Clin Epidemiol. 2011;64:718-42. [PMID: 21216130] doi:10.1016/j.jclinepi.2010.09.013 CrossrefMedlineGoogle Scholar - 55.
Hopewell S ,Clarke M ,Moher D ,Wager E ,Middleton P ,Altman DG ,et al ;CONSORT Group . CONSORT for reporting randomized controlled trials in journal and conference abstracts: explanation and elaboration. PLoS Med. 2008;5:e20. [PMID: 18215107] doi:10.1371/journal.pmed.0050020 CrossrefMedlineGoogle Scholar - 56.
Welch V ,Petticrew M ,Tugwell P ,Moher D ,O'Neill J ,Waters E ,et al ;PRISMA-Equity Bellagio group . PRISMA-Equity 2012 extension: reporting guidelines for systematic reviews with a focus on health equity. PLoS Med. 2012;9:e1001333. [PMID: 23222917] doi:10.1371/journal.pmed.1001333 CrossrefMedlineGoogle Scholar - 57.
Moher D ,Tetzlaff J ,Tricco AC ,Sampson M ,Altman DG . Epidemiology and reporting characteristics of systematic reviews. PLoS Med. 2007;4:e78. [PMID: 17388659] CrossrefMedlineGoogle Scholar - 58.
Turner L ,Shamseer L ,Altman DG ,Weeks L ,Peters J ,Kober T ,et al . Consolidated standards of reporting trials (CONSORT) and the completeness of reporting of randomised controlled trials (RCTs) published in medical journals. Cochrane Database Syst Rev. 2012;11:MR000030. [PMID: 23152285] doi:10.1002/14651858.MR000030.pub2 CrossrefMedlineGoogle Scholar - 59.
Turner EH ,Matthews AM ,Linardatos E ,Tell RA ,Rosenthal R . Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med. 2008;358:252-60. [PMID: 18199864] doi:10.1056/NEJMsa065779 CrossrefMedlineGoogle Scholar - 60. World Medical Association. Declaration of Helsinki—ethical principles for medical research involving human subjects. 2008. Accessed at www.wma.net/en/30publications/10policies/b3/index.html on 31 August 2014. Google Scholar
- 61.
Stegeman BH ,de Bastos M ,Rosendaal FR ,van Hylckama Vlieg A ,Helmerhorst FM ,Stijnen T ,et al . Different combined oral contraceptives and the risk of venous thrombosis: systematic review and network meta-analysis. BMJ. 2013;347:f5298. [PMID: 24030561] doi:10.1136/bmj.f5298 CrossrefMedlineGoogle Scholar - 62.
Peruzzi M ,De Luca L ,Thomsen H ,Romagnoli E ,D'Ascenzo F ,Mancone M ,et al . Network meta-analysis on randomized trials focusing on the preventive effect of statins on contrast-induced nephropathy. Biomed Res Int. 2014;2014:213239. CrossrefMedlineGoogle Scholar - 63.
Loymans RJ ,Gemperli A ,Cohen J ,Rubinstein SM ,Sterk PJ ,Reddel HK ,et al . Comparative effectiveness of long term drug treatment strategies to prevent asthma exacerbations: network meta-analysis. BMJ. 2014;348:g3009. [PMID: 24919052] doi:10.1136/bmj.g3009 CrossrefMedlineGoogle Scholar - 64.
Liao WC ,Chien KL ,Lin YL ,Wu MS ,Lin JT ,Wang HP ,et al . Adjuvant treatments for resected pancreatic adenocarcinoma: a systematic review and network meta-analysis. Lancet Oncol. 2013;14:1095-103. [PMID: 24035532] doi:10.1016/S1470-2045(13)70388-7 CrossrefMedlineGoogle Scholar - 65.
Salanti G . Indirect and mixed-treatment comparison, network, or multiple treatments meta-analysis: many names, many benefits, many concerns for the next generation evidence synthesis tool. Res Synth Methods. 2012;3:80-97. CrossrefMedlineGoogle Scholar - 66.
Donegan S ,Williamson P ,D'Alessandro U ,Tudur Smith C . Assessing the consistency assumption by exploring treatment by covariate interactions in mixed treatment comparison meta-analysis: individual patient-level covariates versus aggregate trial-level covariates. Stat Med. 2012;31:3840-57. [PMID: 22786621] doi:doi:10.1002/sim.5470 CrossrefMedlineGoogle Scholar - 67.
Mauri D ,Polyzos NP ,Salanti G ,Pavlidis N ,Ioannidis JP . Multiple-treatments meta-analysis of chemotherapy and targeted therapies in advanced breast cancer. J Natl Cancer Inst. 2008;100:1780-91. [PMID: 19066278] doi:10.1093/jnci/djn410.1093/jnci/djn414 CrossrefMedlineGoogle Scholar - 68.
Rochwerg B ,Alhazzani W ,Sindi A ,Heels-Ansdell D ,Thabane L ,Fox-Robichaud A ,et al ;Fluids in Sepsis and Septic Shock Group . Fluid resuscitation in sepsis: a systematic review and network meta-analysis. Ann Intern Med. 2014;161:347-55. [PMID: 25047428] doi:10.7326/M14-0178 LinkGoogle Scholar - 69.
Cooper NJ ,Peters J ,Lai MC ,Juni P ,Wandel S ,Palmer S ,et al . How valuable are multiple treatment comparison methods in evidence-based health-care evaluation? Value Health. 2011;14:371-80. [PMID: 21296599] doi:10.1016/j.jval.2010.09.001 CrossrefMedlineGoogle Scholar - 70.
Tonelli AR ,Zein J ,Ioannidis JP . Geometry of the randomized evidence for treatments of pulmonary hypertension. Cardiovasc Ther. 2013;31:e138-46. [PMID: 24112824] doi:10.1111/1755-5922.12050 CrossrefMedlineGoogle Scholar - 71.
Salanti G ,Kavvoura FK ,Ioannidis JP . Exploring the geometry of treatment networks. Ann Intern Med. 2008;148:544-53. [PMID: 18378949] LinkGoogle Scholar - 72.
Salanti G ,Higgins JP ,Ades AE ,Ioannidis JP . Evaluation of networks of randomized trials. Stat Methods Med Res. 2008;17:279-301. [PMID: 17925316] CrossrefMedlineGoogle Scholar - 73.
Kappagoda S ,Ioannidis JP . Neglected tropical diseases: survey and geometry of randomised evidence. BMJ. 2012;345:e6512. [PMID: 23089149] doi:10.1136/bmj.e6512 CrossrefMedlineGoogle Scholar - 74.
Kim DD ,Tang JY ,Ioannidis JP . Network geometry shows evidence sequestration for medical vs. surgical practices: treatments for basal cell carcinoma. J Clin Epidemiol. 2014;67:391-400. [PMID: 24491794] doi:10.1016/j.jclinepi.2013.10.015 CrossrefMedlineGoogle Scholar - 75.
Rizos EC ,Salanti G ,Kontoyiannis DP ,Ioannidis JP . Homophily and co-occurrence patterns shape randomized trials agendas: illustration in antifungal agents. J Clin Epidemiol. 2011;64:830-42. [PMID: 21411286] doi:10.1016/j.jclinepi.2010.11.017 CrossrefMedlineGoogle Scholar - 76.
Lathyris DN ,Patsopoulos NA ,Salanti G ,Ioannidis JP . Industry sponsorship and selection of comparators in randomized clinical trials. Eur J Clin Invest. 2010;40:172-82. [PMID: 20050879] doi:10.1111/j.1365-2362.2009.02240.x CrossrefMedlineGoogle Scholar - 77.
Ioannidis JP ,Karassa FB ,Druyts E ,Thorlund K ,Mills EJ . Biologic agents in rheumatology: unmet issues after 200 trials and $200 billion sales. Nat Rev Rheumatol. 2013;9:665-73. [PMID: 23999553] doi:10.1038/nrrheum.2013.134 CrossrefMedlineGoogle Scholar - 78.
Ioannidis JP ,Karassa FB . The need to consider the wider agenda in systematic reviews and meta-analyses: breadth, timing, and depth of the evidence. BMJ. 2010;341:c4875. [PMID: 20837576] doi:10.1136/bmj.c4875 CrossrefMedlineGoogle Scholar - 79.
Kim D ,Tang J ,Ioannidis JP . Comparative evidence is lacking on conservative versus invasive treatments for basal cell carcinoma. J Clin Epidemiol. 2014;67:391-400. doi:10.1016/j.jclinepi.2013.10.015 CrossrefMedlineGoogle Scholar - 80.
Higgins JP ,Altman DG ,Gøtzsche PC ,Jüni P ,Moher D ,Oxman AD ,et al . Cochrane Bias Methods Group. /e> The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928. [PMID: 22008217] doi:10.1136/bmj.d5928 CrossrefMedlineGoogle Scholar - 81.
Caldwell D ,Welton N ,Dias S ,Ades A . Selecting the best scale for measuring treatment effect in a network meta-analysis: a case study in childhood nocturnal enuresis. Res Synth Methods. 2012;3:126-41. CrossrefMedlineGoogle Scholar - 82.
Filippini G ,Del Giovane C ,Vacchi L ,D'Amico R ,Di Pietrantonj C ,Beecher D ,et al . Immunomodulators and immunosuppressants for multiple sclerosis: a network meta-analysis. Cochrane Database Syst Rev. 2013;6:CD008933. [PMID: 23744561] doi:10.1002/14651858.CD008933.pub2 CrossrefMedlineGoogle Scholar - 83.
Windecker S ,Stortecky S ,Stefanini GG ,da Costa BR ,daCosta BR ,Rutjes AW ,et al . Revascularisation versus medical treatment in patients with stable coronary artery disease: network meta-analysis. BMJ. 2014;348:g3859. [PMID: 24958153] doi:10.1136/bmj.g3859 CrossrefMedlineGoogle Scholar - 84.
White I . Multivariate random effects meta-analysis. Stata J. 2009;9:40-56. CrossrefGoogle Scholar - 85. Rucker G, Shwarzer G. Package 'netmeta': network meta-analysis with R. The R Project. 2013. Accessed at http://cran.r-project.org/web/packages/netmeta/netmeta.pdf on 28 February 2015. Google Scholar
- 86.
Spiegelhalter D ,Best N ,Carlin B ,Van Der Linde A . Bayesian measures of model complexity and fit. J R Stat Soc Series B Stat Methodol. 2002;64:583-639. CrossrefGoogle Scholar - 87.
Spiegelhalter D ,Abrams K ,Myles J . Bayesian Approaches to Clinical Trials and Health-care Evaluation. Chichester, United Kingdom: J Wiley; 2004. Google Scholar - 88.
Leucht S ,Cipriani A ,Spineli L ,Mavridis D ,Orey D ,Richter F ,et al . Comparative efficacy and tolerability of 15 antipsychotic drugs in schizophrenia: a multiple-treatments meta-analysis. Lancet. 2013;382:951-62. [PMID: 23810019] doi:10.1016/S0140-6736(13)60733-3 CrossrefMedlineGoogle Scholar - 89.
Bucher HC ,Guyatt GH ,Griffith LE ,Walter SD . The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials. J Clin Epidemiol. 1997;50:683-91. [PMID: 9250266] CrossrefMedlineGoogle Scholar - 90.
Chaimani A ,Higgins JP ,Mavridis D ,Spyridonos P ,Salanti G . Graphical tools for network meta-analysis in STATA. PLoS One. 2013;8:e76654. [PMID: 24098547] doi:10.1371/journal.pone.0076654 CrossrefMedlineGoogle Scholar - 91.
Trinquart L ,Ioannidis JP ,Chatellier G ,Ravaud P . A test for reporting bias in trial networks: simulation and case studies. BMC Med Res Methodol. 2014;14:112. [PMID: 25262204] doi:10.1186/1471-2288-14-112 CrossrefMedlineGoogle Scholar - 92.
Trinquart L ,Chatellier G ,Ravaud P . Adjustment for reporting bias in network meta-analysis of antidepressant trials. BMC Med Res Methodol. 2012;12:150. [PMID: 23016799] doi:10.1186/1471-2288-12-150 CrossrefMedlineGoogle Scholar - 93.
Mavridis D ,Welton NJ ,Sutton A ,Salanti G . A selection model for accounting for publication bias in a full network meta-analysis. Stat Med. 2014;33:5399-412. [PMID: 25316006] doi:10.1002/sim.6321 CrossrefMedlineGoogle Scholar - 94.
Furukawa TA ,Noma H ,Caldwell DM ,Honyashiki M ,Shinohara K ,Imai H ,et al . Waiting list may be a nocebo condition in psychotherapy trials: a contribution from network meta-analysis. Acta Psychiatr Scand. 2014;130:181-92. [PMID: 24697518] doi:10.1111/acps.12275 CrossrefMedlineGoogle Scholar - 95.
Castellucci LA ,Cameron C ,Le Gal G ,Rodger MA ,Coyle D ,Wells PS ,et al . Efficacy and safety outcomes of oral anticoagulants and antiplatelet drugs in the secondary prevention of venous thromboembolism: systematic review and network meta-analysis. BMJ. 2013;347:f5133. [PMID: 23996149] doi:10.1136/bmj.f5133 CrossrefMedlineGoogle Scholar - 96.
Caldwell DM ,Gibb DM ,Ades AE . Validity of indirect comparisons in meta-analysis [Letter]. Lancet. 2007;369:270. [PMID: 17258659] CrossrefMedlineGoogle Scholar - 97.
Turner RM ,Davey J ,Clarke MJ ,Thompson SG ,Higgins JP . Predicting the extent of heterogeneity in meta-analysis, using empirical data from the Cochrane Database of Systematic Reviews. Int J Epidemiol. 2012;41:818-27. [PMID: 22461129] doi:10.1093/ije/dys041 CrossrefMedlineGoogle Scholar - 98.
Lam SK ,Owen A . Combined resynchronisation and implantable defibrillator therapy in left ventricular dysfunction: Bayesian network meta-analysis of randomised controlled trials. BMJ. 2007;335:925. [PMID: 17932160] CrossrefMedlineGoogle Scholar - 99.
Tan SH ,Cooper NJ ,Bujkiewicz S ,Welton NJ ,Caldwell DM ,Sutton AJ . Novel presentational approaches were developed for reporting network meta-analysis. J Clin Epidemiol. 2014;67:672-80. [PMID: 24560089] doi:10.1016/j.jclinepi.2013.11.006 CrossrefMedlineGoogle Scholar - 100.
Gelman A ,Carlin J ,Stern H ,Dunson D ,Vehtari A ,Rubin D . Bayesian Data Analysis. 3rd ed. Boca Raton, FL: CRC Pr; 2013. Google Scholar - 101.
Lu G ,Welton N ,Higgins JP ,White I ,Ades A . Linear inference for mixed treatment comparison meta-analysis: a two-stage approach. Res Synth Methods. 2011;2:43-60. CrossrefMedlineGoogle Scholar - 102.
Caldwell DM ,Welton NJ ,Ades AE . Mixed treatment comparison analysis provides internally coherent treatment effect estimates based on overviews of reviews and can reveal inconsistency. J Clin Epidemiol. 2010;63:875-82. [PMID: 20080027] doi:10.1016/j.jclinepi.2009.08.025 CrossrefMedlineGoogle Scholar - 103.
Song F ,Altman DG ,Glenny AM ,Deeks JJ . Validity of indirect comparison for estimating efficacy of competing interventions: empirical evidence from published meta-analyses. BMJ. 2003;326:472. [PMID: 12609941] CrossrefMedlineGoogle Scholar - 104.
Krahn U ,Binder H ,König J . A graphical tool for locating inconsistency in network meta-analyses. BMC Med Res Methodol. 2013;13:35. [PMID: 23496991] doi:10.1186/1471-2288-13-35 CrossrefMedlineGoogle Scholar - 105.
Puhan MA ,Schünemann HJ ,Murad MH ,Li T ,Brignardello-Petersen R ,Singh JA ,et al ;GRADE Working Group . A GRADE Working Group approach for rating the quality of treatment effect estimates from network meta-analysis. BMJ. 2014;349:g5630. [PMID: 25252733] doi:10.1136/bmj.g5630 CrossrefMedlineGoogle Scholar - 106.
Stettler C ,Allemann S ,Wandel S ,Kastrati A ,Morice MC ,Schömig A ,et al . Drug eluting and bare metal stents in people with and without diabetes: collaborative network meta-analysis. BMJ. 2008;337:a1331. [PMID: 18757996] doi:10.1136/bmj.a1331 CrossrefMedlineGoogle Scholar - 107.
Cameron C ,Coyle D ,Richter T ,Kelly S ,Gauthier K ,Steiner S ,et al . Systematic review and network meta-analysis comparing antithrombotic agents for the prevention of stroke and major bleeding in patients with atrial fibrillation. BMJ Open. 2014;4:e004301. [PMID: 24889848] doi:10.1136/bmjopen-2013-004301 CrossrefMedlineGoogle Scholar - 108.
Jørgensen AW ,Hilden J ,Gøtzsche PC . Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review. BMJ. 2006;333:782. [PMID: 17028106] CrossrefMedlineGoogle Scholar - 109.
Stamatakis E ,Weiler R ,Ioannidis JP . Undue industry influences that distort healthcare research, strategy, expenditure and practice: a review. Eur J Clin Invest. 2013;43:469-75. [PMID: 23521369] doi:10.1111/eci.12074 CrossrefMedlineGoogle Scholar - 110. Medical Research Council.The BUGS project. 2013. Accessed at www.mrc-bsu.cam.ac.uk/software/bugs/ on 28 February 2015. Google Scholar
- 111. JAGS. 26 April 2013. Accessed at http://mcmc-jags.sourceforge.net on 28 February 2015. Google Scholar
- 112. National Institute for Health and Care Excellence. NICE decision support unit. 2013. Accessed at www.nicedsu.org.uk/Evidence-Synthesis-TSD-series(2391675).htm on 28 February 2015. Google Scholar
- 113. University of Bristol. Multi-parameter evidence synthesis research. 2013. Accessed at www.bris.ac.uk/social-community-medicine/projects/mpes on28 February 2015. Google Scholar
- 114. Multiple-treatments meta-analysis: a framework for evaluating and ranking multiple healthcare technologies. 2013. Accessed at www.mtm.uoi.gr on 28 February 2015. Google Scholar
- 115. van Valkenhoef G, Kuiper J. Package 'gemtc': GeMTC network meta-analysis. 2013. Accessed at http://cran.r-project.org/web/packages/gemtc/gemtc.pdf on 28 February 2015. Google Scholar
- 116. Medical Research Council. The BUGS project: calling Winbugs 1.4 remotely. 2013. Accessed at www.mrc-bsu.cam.ac.uk/software/bugs/calling-winbugs-1-4-from-other-programs/ on 16 April 2015. Google Scholar
- 117.
van Valkenhoef G ,Lu G ,de Brock B ,Hillege H ,Ades A ,Welton N . Automating network meta-analysis. Res Synth Methods. 2012;3:285-99. CrossrefMedlineGoogle Scholar - 118.
van Valkenhoef G ,Tervonen T ,de Brock B ,Hillege H . Algorithmic parameterization of mixed treatment comparisons. Stat Comput. 2012;22:1099-11. CrossrefGoogle Scholar - 119. drugis.org. ADDIS: Aggregate Data Drug Information System. 2013. Accessed at http://drugis.org/addis on 1 February 2015. Google Scholar
- 120.
Rucker G . Network meta-analysis, electrical networks and graph theory. Research Synth Meth. 2012;3:112-24. CrossrefGoogle Scholar - 121.
White IR . Multivariate random-effects meta-regression: updates to mvmeta. Stata J. 2011;11:255-70. CrossrefGoogle Scholar - 122. Medical Research Council. STATA software. 2013. Accessed at www.mrc-bsu.cam.ac.uk/software/stata-software/ on 16 April 2015. Google Scholar
- 123. Canadian Agency for Drugs and Technologies in Health. CADTH indirect treatment comparisons calculator. 2013. Accessed at https://www.cadth.ca/resources/itc-user-guide on 16 April 2015. Google Scholar
- 124.
Wells G ,Sultan S ,Chen L ,Khan M ,Coyle D . Indirect Evidence: Indirect Treatment Comparisons in Meta-Analysis. Ottawa, Ontario, Canada: Canadian Agency for Drugs and Technologies in Health; 2009. Google Scholar
Author, Article, and Disclosure Information
Brian Hutton,
From Ottawa Hospital Research Institute, Ottawa, Ontario, Canada; University of Ioannina, Ioannina, Greece; School of Social and Community Medicine, University of Bristol, Bristol, United Kingdom; Center for Evidence-Based Medicine, Brown University School of Public Health, Providence, Rhode Island; Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, California; Li Ka Shing Knowledge Institute of St. Michaels Hospital and University of Toronto, Toronto, Ontario, Canada; McMaster University, Hamilton, Ontario, Canada; Tufts University School of Medicine, Boston, Massachusetts; American College of Physicians, Philadelphia, Pennsylvania; Spanish Medicines and Healthcare Products Agency, Madrid, Spain, Nordic Cochrane Centre, Copenhagen, Denmark; Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland; INSERM, L'Université Paris Descartes, Paris, France; and Centre for Statistics in Medicine and University of Oxford, Oxford, United Kingdom.
Financial Support: By the Canadian Agency for Drugs and Technologies in Health and Pfizer Canada for the development of this work. Dr. Hutton is supported by a New Investigator Award from the Canadian Institutes of Health Research and the Drug Safety and Effectiveness Network. Dr. Caldwell is supported by a Medical Research Council Population Health Science Fellowship award (G0902118). Mr. Cameron is a recipient of a Vanier Canada Graduate Scholarship from the Canadian Institutes of Health Research (funding reference number CGV 121171) and is a trainee on the Canadian Institutes of Health Research Drug Safety and Effectiveness Network team grant (funding reference number 116573). The Meta-Research Innovation Center at Stanford (METRICS) is funded by a grant from the Laura and John Arnold Foundation.
Disclosures: Dr. Hutton reports honoraria from Amgen Canada. Dr. Thorlund reports that he is a cofounding partner and majority shareholder of Redwood Outcomes. Dr. Jansen reports that he is a cofounding partner and majority shareholder of Redwood Outcomes. Authors not named here have disclosed no conflicts of interest. Disclosures can be viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum=M14-2385.
Editors' Disclosures: Christine Laine, MD, MPH, Editor in Chief, reports that she has no financial relationships or interests to disclose. Darren B. Taichman, MD, PhD, Executive Deputy Editor, reports that he has no financial relationships or interests to disclose. Cynthia D. Mulrow, MD, MSc, Senior Deputy Editor, reports that she has no relationships or interests to disclose. Deborah Cotton, MD, MPH, Deputy Editor, reports that she has no financial relationships or interest to disclose. Jaya K. Rao, MD, MHS, Deputy Editor, reports that she has stock holdings/options in Eli Lilly and Pfizer. Sankey V. Williams, MD, Deputy Editor, reports that he has no financial relationships or interests to disclose. Catharine B. Stack, PhD, MS, Deputy Editor for Statistics, reports that she has stock holdings in Pfizer.
Corresponding Author: Brian Hutton, PhD, MSc, Ottawa Hospital Research Institute, Center for Practice Changing Research, The Ottawa Hospital–General Campus, 501 Smyth Road, PO Box 201B, Ottawa, Ontario K1H 8L6, Canada; e-mail, bhutton@ohri.
Current Author Addresses:Drs. Hutton and Moher and Mr. Cameron: Ottawa Hospital Research Institute, Center for Practice Changing Research, The Ottawa Hospital–General Campus, 501 Smyth Road, PO Box 201B, Ottawa, Ontario K1H 8L6, Canada.
Drs. Salanti and Chaimani: Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, University Campus Ioannina 45110, Ioannina, Greece.
Dr. Caldwell: School of Social and Community Medicine, Canynge Hall, 39 Whately Road, Bristol BS8 2PS, United Kingdom.
Dr. Schmid: Center for Evidence-Based Medicine, Brown University School of Public Health, Box G-S121-8, Providence, RI 02912.
Dr. Ioannidis: Stanford Prevention Research Center, Stanford University School of Medicine, Medical School Office Building, 1265 Welch Road, Mail Code 5411, Stanford, CA 94305-5411.
Dr. Straus: Li Ka Shing Knowledge Institute, St. Michael's Hospital, 30 Bond Street, Toronto, Ontario M5B 1W8, Canada.
Drs. Thorlund and Jansen: 1505 West Second Avenue, Suite 302, Vancouver, British Columbia V6H 3Y4, Canada.
Dr. Mulrow: American College of Physicians, 190 N. Independence Mall West, Philadelphia, PA 19106.
Dr. Catála-López: Division of Pharmacoepidemiology and Pharmacovigilance, Spanish Medicines and Healthcare Products Agency, Campezo 1, 28022 Madrid, Spain.
Dr. Gøtzsche: Nordic Cochrane Centre, Rigshospitalet, Department 7811, Blegdamsvej 9, DK-2100 Copenhagen, Denmark.
Dr. Dickersin: Johns Hopkins Bloomberg School of Public Health, 615 North Wolfe Street, Room E6152, Baltimore, MD 21205.
Dr. Boutron: Centre d'Epidémiologie Clinique, L'Université Paris Descartes Centre de recherche Epidémiologies et Statistique, INSERM U1153, Equipe: Méthodes en évaluation thérapeutique des maladies chroniques, Hópital Hôtel Dieu, Aile A2 1er étage, 1 Place du parvis Notre Dame, 75181 Paris, Cedex 4, France.
Dr. Altman: Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Botnar Research Centre, Windmill Road, Oxford OX3 7LD, United Kingdom.
Author Contributions:Conception and design: B. Hutton, G. Salanti, D. Moher, C.H. Schmid, A. Chaimani, D.M. Caldwell, K. Thorlund.
Analysis and interpretation of the data: B. Hutton, D. Moher, D.M. Caldwell, A. Chaimani, K. Thorlund, C.H. Schmid, S. Straus, P.C. Gøtzsche.
Drafting of the article: B. Hutton, G. Salanti, D.M. Caldwell, C.H. Schmid, K. Thorlund, D. Moher, C. Cameron, C. Mulrow, F. Catalá-López, P.C. Gøtzsche.
Critical revision of the article for important intellectual content: B. Hutton, G. Salanti, D.M. Caldwell, A. Chaimani, C.H. Schmid, C. Cameron, J.P.A. Ioannidis, S. Straus, K. Thorlund, J.P. Jansen, C. Mulrow, F. Catalá-López, P.C. Gøtzsche, K. Dickersin, I. Boutron, D.G. Altman, D. Moher.
Final approval of the article: B. Hutton, G. Salanti, D.M. Caldwell, A. Chaimani, C.H. Schmid, C. Cameron, J.P.A. Ioannidis, S. Straus, K. Thorlund, J.P. Jansen, C. Mulrow, F. Catalá-López, P.C. Gøtzsche, K. Dickersin, I. Boutron, D.G. Altman, D. Moher.
Statistical expertise: B. Hutton, G. Salanti, A. Chaimani, C.H. Schmid, D.M. Caldwell, K. Thorlund, C. Cameron, D.G. Altman.
Obtaining of funding: B. Hutton, D. Moher.
PRISMA Considerations and Searching the Literature
While we agree with the practices described Drs. Tian, Ge and Li describe regarding searching, we feel they are entirely applicable to traditional systematic reviews and are addressed in the PRISMA Statement’s explanation and elaboration article.(4) The main intent of the PRISMA extension statement for reporting of network meta-analysis is to focus on items that were not addressed in PRISMA and differ substantially from practices for traditional systematic reviews and meta-analyses.
Regarding their first suggestion, we feel ensuring the need for a proposed review is equally important for traditional reviews, and in all cases searching for existing literature should be preceded by in-depth consideration of the clinical importance of the research question in a PICOS (Population-Intervention-Comparator(s)-Outcome(s)-Study design) framework.
Regarding their second suggestion, we believe it has long been common for researchers undertaking reviews of multiple forms to inspect bibliographies of past reviews and included studies as a source for potentially relevant studies; the PRISMA Statement’s Explanation and Elaboration document addresses this issue in Item 7, suggesting ‘In addition to searching databases, authors should report the use of supplementary approaches to identify studies, such as hand searching of journals, checking reference lists, searching trials registries or regulatory agency Web sites, contacting manufacturers, or contacting authors.’(4)
Lastly, we agree that the increased number of interventions in a network meta-analysis can heighten the challenge of completing the systematic search strategy for a review. However, this may also often be true of other reviews not involving a multi-treatment question; for example, reviews involving more than one indication of relevance, or reviews involving complex interventions. In these and other scenarios, we support the practice of peer review of literature searches to maximize their quality. This is addressed in Item 8 in the PRISMA Explanations and Elaborations Statement: ‘We encourage authors to state whether search strategies were peer reviewed as part of the systematic review process.’(4)
Therefore, we support these practices noted by Tian, Ge, and Li. However we feel their importance and existing practice amongst researchers extend to many additional types of reviews, and that guidance from the PRISMA Statement remains highly relevant.
The authors also suggest a potential need for a second guidance document addressing reporting for Bayesian network meta-analyses; we disagree with this perspective at this time. The examples and elaborations provided in our guidance address reporting considerations for the key items that were suggested, while we do not foresee a need for certain suggested components such as specification of starting values or the number of iterations used (we recommend provision of details for convergence assessment already). We are unclear as to the authors’ intended meaning of suggesting sample size, however we hypothesize this is a reference to statistical power in network meta-analyses. We agree this can be of interest, and may be especially so for outcomes with few events. Some research has been conducted in this area,(5) although additional research is needed to inform considerations for reporting guidance.
We believe the current guidance provides a strong set of minimum reporting items for Frequentist and Bayesian NMAs, while authors are certainly encouraged to provide additional information of relevance to readers to support their reviews. As methodologies continue to evolve in this rapidly developing area, we will continue to gather materials for a possible future update of this extension statement which may include guidance that additional statistical considerations be reported.
Brian Hutton, PhD; David Moher, PhD
Ottawa Hospital Research Institute, Ottawa, Canada;
University of Ottawa School of Epidemiology, Public Health and Preventive Medicine, Ottawa, Canada
Chris Cameron, PhD
Ottawa Hospital Research Institute, Ottawa, Canada;
Cornerstone Research Group Inc., Burlington, Canada
Reference List
(1) Tian J, Ge L, and Li L. Searching for previous published and unpublished or ongoing systematic reviews/meta-analyses is very important (Commentary). Annals of Internal Medicine. 2015.
(2) Ge L, Tian J, Li L, and Yang K. The PRISMA Extension Statement for Statistical Analysis Reporting of Network Meta-Analysis is Needed (Commentary). Annals of Internal Medicine. 14-6-2015.
(3) Hutton B, Salanti G, Caldwell D, Schmid C, Chaimani A, Cameron C, Ioannidis J, and et al. The PRISMA Extension Statement for Reporting of Systematic Reviews Incorporating Network Meta-Analyses of Healthcare Interventions: Checklist and Explanations. Annals of Internal Medicine 162(11), 777-784. 2015.
(4) Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche P, Ioannidis J, Clarke M, Devereau PJ, Kleijnen J, and Moher D. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Annals of Internal Medicine 151(4), W65-W94. 2009.
(5) Thorlund K and Mills E. Sample size and power considerations in network meta-analysis. Systematic Reviews 1(41. doi: 10.1186/2046-4053-1-41). 2012.
Searching for previous published and unpublished or ongoing systematic reviews/meta-analyses is very important.
We declare that we have no conflicts of interest.
Jin-hui Tian, PhD
Evidence-Based Medicine Center of Lanzhou University, Lanzhou 730000, China.
Key Laboratory of Evidence-based Medicine and Clinical Translational Research of Gansu Province, Lanzhou 730000.
Long Ge,MD; Lun Li, PhD
The First Clinical Medicine College of Lanzhou University, Lanzhou 730000, China;
Evidence-Based Medicine Center of Lanzhou University, Lanzhou 730000, China;
Key Laboratory of Evidence-based Medicine and Clinical Translational Research of Gansu Province, Lanzhou 730000.
(1)Hutton B, Salanti G, Caldwell DM, Chaimani A, Schmid CH, Cameron C, et al. The PRISMA Extension Statement for Reporting of Systematic Reviews Incorporating Network Meta-analyses of Health Care Interventions: Checklist and Explanations.Ann Intern Med. 2015;162(11):777-84.
(2)Golger S, Wright K. Searching for evidence.In: Biondi-Zoccai G, editor. Network meta-analysis: evidence synthesis with mixed treatment comparison. New York: Nova Science, 2014. P. 63-76.
(3)Li L, Tian J, Tian H, Moher D, Liang F, Jiang T, et al. Network meta-analyses could be improved by searching more sources and by involving a librarian. J Clin Epidemiol. 2014; 67(9):1001-7.
The PRISMA Extension Statement for Statistical Analysis Reporting of Network Meta-Analysis is Needed
Methods section:
The detail of sample size calculation
Direct comparison: Assessment of heterogeneity, model of pooling data,summary measure, assessment of publication bias, sensitivity analysis, other analysis, software applied.
Network meta-analysis: Assessment of heterogeneity, adjustment for covariates, adjustment of multiple arms, code applied, selection of prior distribution, selection of fixed or random effect model, selection of consistency or inconsistency, assessment of inconsistency, assessment of convergence, summary measure (including treatment ranking), assessment of publication bias, sensitivity analysis (based on prior distribution or other), other analysis, software applied.
Results section:
The results of sample size calculation
Direct comparison: Results of heterogeneity assessment, model applied, results of direct comparison, publication bias assessment, sensitivity analysis and other analysis.
Network meta-analysis: Methods and results of heterogeneity, results of model fit tested, number of chains, the staring values for sampling, number of iterations per chain, number of iteration used for final results, results of convergence assessment, prior distributions used, results of indirect comparison, results of network meta-analysis, results of inconsistency assessment, results of ranking, publication bias assessment, sensitivity analysis, other analysis.
The details for each item should also be described in published papers. However, those items are based on published literature. Delphi survey, and face-to-face discussion and consensus meeting should be established to develop another PRISMA extension statement for reporting of statistical analysis and assumptions of NMA (PRISMA-S). We strongly believe that it will play very important parts to improve the quality of NMAs.
We declare that we have no conflicts of interest.
Long Ge
The First Clinical Medicine College of Lanzhou University, Lanzhou 730000, China;
Evidence-Based Medicine Center of Lanzhou University, Lanzhou 730000, China.
Jin-hui Tian, Lun Li, Ke-hu Yang*
Evidence-Based Medicine Center of Lanzhou University, Lanzhou 730000, China;
Key Laboratory of Evidence-based Medicine and Clinical Translational Research of Gansu Province, Lanzhou 730000.
*[email protected]
(1)Hutton B, Salanti G, Caldwell DM, Chaimani A, Schmid CH, Cameron C, et al. The PRISMA Extension Statement for Reporting of Systematic Reviews Incorporating Network Meta-analyses of Health Care Interventions: Checklist and Explanations.Ann Intern Med. 2015;162(11):777-84.
(2)Li L, Tian JH, Yang KH. Current situation of reporting statement for network meta-analysis.Chin J Evid Based Pediatr.2014;9(6):467-71.
(3)Tan SH, Bujkiewicz S, Sutton A, Dequen P, Cooper N.Presentational approaches used in the UK for reporting evidence synthesis using indirect and mixed treatment comparisons.J Health Serv Res Policy. 2013;18(4):224-32.