Critical Appraisal of a Network Meta-Analysis in Emergency Medicine

  

    
GRADE domain 
    
Domain assessment in NMA 
    
Description of procedure 
    
Instructions for downgrading 
  
  

    
Evaluate the confidence in a specific pairwise    effect estimated in NMA 
  
  

    
Study limitations 
    
Study    limitations 
    
Determine which direct comparisons contribute to estimation of the NMA    treatment effect a and integrate risk of bias assessments from these into a    single judgment 
    
Use standard GRADE considerations to inform judgment 
  
  

    
Indirectness 
    
Joint consideration of indirectness and intransitivity 
    
Evaluate indirectness of populations, interventions, and outcomes as    in standard GRADE. Evaluate transitivity by comparing the distribution of    known effect modifiers across comparisons that contribute evidence to    estimation of the NMA treatment effect 
    
If a priori assessment makes a transitivity assumption reasonable and    suggests that effect modifiers are balanced, then do not downgrade.    Otherwise, downgrade (either if a transitivity assumption does not look    reasonable or if there is insufficient evidence to judge) 
  
  

    
Inconsistency 
    
Joint consideration of statistical heterogeneity and statistical    inconsistency 
    
(1) Judge the extent of heterogeneity, considering the comparison-specific    heterogeneity variance, the NMA estimate of variance, a prediction interval    and/or other relevant metrics such as I 2
      
(2) Evaluate the extent to which the comparison under evaluation is    involved in inconsistent loops of evidence 
    
(1) If important heterogeneity is found, downgrade. If heterogeneity    is low, do not downgrade
      
(2) Power to detect inconsistency may be low; downgrade in absence of    statistical evidence for inconsistency when direct and indirect estimates    imply different clinical decisions 
  
  

    
Imprecision 
    
Imprecision 
    
Focus on width of the confidence interval 
    
Assess uncertainty around the pairwise estimate. Downgrade if    confidence interval crosses null value or includes values favoring either    treatment 
  
  

    
Publication bias 
    
Publication    bias 
    
Nonstatistical consideration of likelihood of nonpublication of    evidence that would inform the pairwise comparison. Plotpairwiseestimatesoncontour-enhancedfunnelplot 
    
Use standard GRADE to inform judgment 
  
  

    
Evaluate the confidence in treatment ranking    estimated in NMA 
  
  

    
Study limitations 
    
Study    limitations 
    
Integrate risk of bias assessments from each direct comparison to    formulate a single overall confidence rating for treatment rankingsa
    
Use standard GRADE considerations to inform judgment
  
  

    
Indirectness
    
Joint consideration of indirectness and intransitivity
    
Evaluate indirectness of populations, interventions, and outcomes as    in standard GRADE. Evaluate transitivity across network by comparing the    distribution of known effect modifiers across comparison a
    
If a priori assessment of transitivity suggests effect modifiers are    balanced across the network, do not downgrade. Otherwise, downgrade (either    if a transitivity assumption does not look reasonable or if there is    insufficient evidence to judge)
  
  

    
Inconsistency
    
Joint consideration of statistical heterogeneity and statistical    inconsistency
    
(1) Judge the extent of heterogeneity considering primarily the NMA    variance estimate(s) used and other network-wise metrics such as Q for    heterogeneity in a network
      
(2) Evaluate inconsistency in network using statistical methods (such    as global tests of inconsistency, or global inconsistency parameter)
    
(1) If important heterogeneity is found, downgrade. If heterogeneity    is low do not downgrade.
      
(2) For overall treatment rankings, inconsistency should be given    greater emphasis, since ranks are based on mean effects and the uncertainty    they are estimated with. Downgrade in absence of statistical evidence for    inconsistency when several direct and indirect estimates imply different clinical    decisions
  
  

    
Imprecision
    
Imprecision 
    
Visually examine ranking probabilities (e.g., rank grams) for overlap    to assess precision of treatment rankings
    
If probabilities are similarly distributed across the ranks, downgrade    for imprecision
  
  

    
Publication bias
    
Publication    bias
    
Nonstatistical consideration of likelihood of nonpublication for each    pairwise comparison. If appropriate, plot NMA estimates on a comparison    adjusted funnel plot and assess asymmetry
    
As asymmetry does not provide concrete evidence of publication bias,    downgrading should only be considered jointly with the nonstatistical    assessment
  
  

    
aWhen    integrating assessments about direct comparisons into a judgment about an NMA    treatment effect or the ranking, more weight should be given to assessments    from direct comparisons that contribute more information. We recommend use of    the contributions matrix to quantify how much information each direct    comparison contributes to the estimation of the NMA treatment effect under    evaluation or the ranking. 
      
 
  

Table 1:  Shows a summary critical appraisal of evaluating the quality of evidence from a NMA [1-6].