Meta-Analysis for Imaging Procedure Technical Performance

The article “Meta-analysis of the technical performance of an imaging procedure: Guidelines and statistical methodology,” published in Statistical Methods in Medical Research in 2015, by Erich P Huang and colleagues, addresses the critical need for robust, quantitative methods to summarize the technical performance of medical imaging procedures. This paper is particularly relevant in the context of Quantitative Imaging Biomarkers (QIBs), which are objectively measured characteristics derived from medical images that are increasingly central to patient care and drug development, for roles such as assessing treatment response and guiding clinical decisions.

The Crucial Role of Technical Performance Evaluation: Before a QIB can be accepted for clinical use, the imaging procedure that produces it must undergo a thorough evaluation of its technical performance. This evaluation typically involves assessing performance metrics like repeatability (how close repeated measurements are under the same conditions) and reproducibility (how close measurements are under varying conditions). The authors emphasize that conclusions about an imaging procedure’s technical performance should ideally be based on quantitative summaries from multiple studies. This approach helps to overcome the limitations of small sample sizes often seen in individual technical performance studies and allows for a broader understanding of performance across diverse clinical settings and patient populations.

Key Challenges in Meta-Analysis of Technical Performance: The paper identifies several significant challenges unique to meta-analyses in this domain, differentiating them from traditional meta-analyses of treatment effects or risk factors:

• Small Study Sizes: Many technical performance studies involve as few as 10–20 patients. This small sample size often causes violations of assumptions, particularly the approximate normality of performance metric estimates, which is a cornerstone of standard meta-analysis techniques.
• Non-normality of Metric Estimates: Due to small sample sizes, many common technical performance metrics, such as the repeatability coefficient (RC), are not normally distributed, leading to inaccurate confidence intervals from standard methods. For instance, for the RC, simulation studies showed that it is only approximately normal if a study contains 80 or more subjects.
• Between-Study Heterogeneity: Variability among studies is common due to differences in imaging devices, acquisition protocols, image processing, operator effects, and clinical settings.
• Limitations of Qualitative Reviews: Traditional narrative or prose reviews are difficult to interpret, prone to subjective bias in study selection, and lack a quantitative summary. Systematic reviews improve upon this by using criterion-based searches and rigorous critical review, but meta-analysis takes the crucial extra step of producing a quantitative summary.
• Limited Primary Studies: There is a scarcity of completed studies specifically designed to evaluate the technical performance of imaging procedures.

Proposed Solutions: Guidelines and Statistical Methodology: The article provides a structured approach to conducting meta-analyses for imaging procedure performance, encompassing the systematic review process, advanced statistical methodologies, and reporting guidelines.

1. Systematic Review Process (Section 2):
   • Formulation of the Research Question: This is foundational, requiring careful specification of the clinical context, class of imaging procedures, and specific performance metrics. For example, the Repeatability Coefficient (RC) is highlighted as suitable for determining thresholds for true signal change versus noise in FDG-PET uptake. Researchers must also consider the feasibility of studies, ethical implications (e.g., repeated radiation doses), and trade-offs between narrowly focused versus broader questions.
   • Study Selection Process: This involves defining clear search criteria (e.g., test-retest FDG-PET studies in malignant tumors) and conducting intensive searches across published literature (e.g., Medline, Embase) and unpublished sources (e.g., meeting abstracts, study registries, regulatory summaries, internal reports, public databases like The Cancer Imaging Archive) to mitigate publication bias. Challenges with unpublished data include quality assessment and selective access. Specific inclusion/exclusion criteria (e.g., sample size, device specifications, protocols) are then applied to refine the selection, ensuring studies are appropriate and addressing potential data overlap. Particular attention is paid to the sources of variability captured by the performance metrics.
   • Organizing and Summarizing Data: Retrieved studies should be evaluated by at least two independent reviewers. Key descriptors (e.g., device manufacturer, scan protocols, patient population characteristics) and performance metrics with their uncertainty measures (standard error, confidence intervals) are collected. Forest plots are recommended for graphical display of estimates and confidence intervals across studies.
2. Statistical Methodology for Meta-Analyses (Sections 3 & 4):
   • Fixed-Effects vs. Random-Effects Models:
     • Fixed-effects models assume a single, common true technical performance across all studies (homogeneity), which is rarely realistic in imaging studies due to inherent variability.
     • Random-effects models are generally recommended for QIB applications, as they assume that study-specific performance metrics vary around an overall mean ().
     • Tests for homogeneity (e.g., Q statistic, parametric bootstrap for non-normal data) help determine if a fixed-effects model is appropriate, but their low power with small numbers of studies means failure to reject homogeneity doesn’t confirm its absence. Measures like H and I² quantify heterogeneity.
   • Addressing Non-Normality with Exact Likelihoods: This is a core contribution of the paper. When sample sizes are small, standard methods relying on normal approximations fail. The article proposes using exact distributions of metric estimates if analytically tractable. For example, the squared RC is shown to follow a gamma distribution. Using these exact likelihoods in both frequentist and Bayesian inference improves confidence interval coverage.
   • Expectation-Maximization (EM) Algorithm: For random-effects meta-analysis, the EM algorithm can be used to estimate the underlying distribution of study-specific effects, especially when exact likelihoods are used for non-normally distributed metrics. This approach, combined with nonparametric bootstrapping for confidence intervals, shows improved performance.
   • Meta-Regression (Section 4): This technique extends meta-analysis by allowing researchers to explain between-study variability in performance metrics using study-specific descriptors (covariates) such as slice thickness, training of analysts, or software choice. Both fixed-effects and random-effects meta-regression approaches are described, with adaptations for non-normally distributed data, such as generalized linear models.

Application to Simulations and Actual Data (Section 5): The authors rigorously tested their proposed methodologies through simulation studies and an illustrative application to real-world data:

• Simulation Studies (Section 5.1): These studies demonstrated that standard fixed-effects and random-effects meta-analysis techniques frequently yielded 95% confidence intervals with coverage probabilities noticeably below the nominal 0.95, particularly when some primary studies were small or when the true study effects were non-normally distributed. In contrast, techniques using exact likelihoods (e.g., for the RC) or the EM algorithm with exact likelihoods consistently achieved coverage probabilities close to 0.95, even with small study sizes. The performance of random-effects meta-analysis methods still suffered when the number of studies (K) was very small, regardless of the method.
• FDG-PET SUV Test-Retest Repeatability Example (Section 5.2): The discussed methods were applied to a meta-analysis of five FDG-PET studies examining the repeatability of SUVmean. The results highlighted the practical impact of the proposed methods:
  • Standard fixed-effects meta-analysis (normal approximation) estimated an RC of 0.79 (95% CI: 0.67, 0.92), heavily influenced by a study with a low RC and larger sample size.
  • Using fixed-effects with exact likelihoods produced a noticeably different RC estimate of 1.53 (95% CI: 1.32, 1.74), suggesting the normal approximation was indeed violated due to small sample sizes in the primary studies.
  • Random-effects methods provided similar estimates (e.g., DerSimonian and Laird: 1.25, 95% CI: 0.67, 1.84; EM with exact likelihood: 1.34, 95% CI: 0.52, 1.97), but their wider confidence intervals reflected the heterogeneity.
  • Fixed-effects meta-regression using exact likelihoods suggested that higher median baseline SUVmean was associated with higher RC, and a higher proportion of thoracic lesions was associated with lower RC. These associations were less conclusive or absent when using the normal approximation, underscoring the importance of the appropriate methodology.

Individual Patient-Level Meta-Analysis (Section 6): The article also briefly touches upon individual patient-level meta-analysis as an alternative to study-level methods, particularly advantageous when technical performance is influenced by patient-specific characteristics (e.g., tumor size, baseline uptake, physiological factors). While not always yielding large gains in efficiency, it allows for direct computation of summary statistics and more powerful meta-regression analyses when patient-level covariates are available.

Reporting Guidelines (Section 8): To ensure interpretability and minimize bias, the paper strongly advocates for complete and transparent reporting of meta-analysis results. Recognizing the lack of specific guidelines for imaging procedure technical performance studies, the authors provide a checklist (Table 7). This checklist, structured like a journal article, covers essential elements for the Title, Abstract, Introduction, Methods (e.g., sources, eligibility, validity assessment, performance metrics, heterogeneity, quantitative synthesis), Results (e.g., study flow, characteristics, synthesis), and Discussion. It aims to serve as a starting point for future guideline development and to aid researchers, journal editors, and regulatory bodies.

Future Research and Broader Implications (Section 9): The paper concludes by acknowledging areas requiring further statistical methodological research, particularly for metrics whose exact distributions are not analytically tractable, making fully nonparametric meta-analysis techniques a promising but underexplored avenue. The authors emphasize that robust meta-analysis methods are an essential step towards establishing the clinical utility of QIBs. They call for:

• Increased conduct and publication of imaging procedure performance studies.
• Transparent reporting of these studies.
• Greater coordination and recommendations from professional societies to enhance comparability across studies.
• Continued statistical research to address identified challenges in meta-analysis methodology for imaging technical performance. The work also notes that while the methodology focused on a single QIB, future research could explore multivariate approaches for joint analysis of multiple QIBs from the same image, taking correlations into account.

This comprehensive approach underscores the paper’s significance in providing a framework for rigorously evaluating the technical performance of quantitative imaging biomarkers, thereby advancing their reliability and clinical utility.

Reference for the Article:

Huang, E. P., Wang, X.-F., Choudhury, K. R., McShane, L. M., Gönen, M., Ye, J., Buckler, A. J., Kinahan, P. E., Reeves, A. P., Jackson, E. F., Guimaraes, A. R., & Zahlmann, G. (2015). Meta-analysis of the technical performance of an imaging procedure: Guidelines and statistical methodology. Statistical Methods in Medical Research, 24(1), 141–174. https://doi.org/10.1177/0962280214537394