Systematic Reviews of Healthcare Interventions: Methodology and Practice

This article, titled “Methodology in conducting a systematic review of systematic reviews of healthcare interventions” by Valerie Smith and colleagues, offers a comprehensive guide for conducting systematic reviews of systematic reviews, also known as overviews.

The paper addresses a significant challenge in healthcare: the overwhelming volume of published studies and, more recently, a “plethora of reviews” that vary in quality and scope, often with multiple reviews on the same important topics. This makes it difficult for clinical decision-makers to identify and consider all relevant evidence.

Key aspects of the article include:

• Purpose of Overviews: Systematic reviews of reviews serve as a logical next step to appraise, summarize, and bring together existing studies and reviews in a single place. They allow for the comparison and contrast of findings from separate reviews, providing crucial evidence for clinical decision-makers.
• Detailed Methodology: The authors describe robust methods for systematically identifying and appraising both published and unpublished reviews. This process aims to describe the quality of the evidence base, summarize and compare review conclusions, and discuss the strength of these conclusions.
• Methodological Challenges and Solutions: The article outlines common methodological challenges and offers practical solutions within the context of:
  • Sources and Searching: It discusses strategies for locating relevant literature, including focusing searches on databases specific to systematic reviews (e.g., Cochrane Database of Systematic Reviews, DARE) and using structured search strategies, potentially guided by the PRESS initiative.
  • Review Selection: Guidance is provided on establishing a review team with both methodological and topic expertise, developing clear inclusion/exclusion criteria, and using independent assessment by at least two team members to reduce bias.
  • Quality Assessment of Reviews: Emphasizing the importance of assessing the quality and strength of evidence in included reviews, the article highlights the AMSTAR tool as a validated means to evaluate the methodological quality of systematic reviews. Key domains for quality assessment include the research question, data extraction, comprehensive literature review, assessment of publication bias, and heterogeneity.
  • Presentation of Results: Recommendations include presenting major conclusions alongside their supporting evidence and quality assessment (e.g., using the GRADE approach), and employing summary tables and figures for clarity (examples provided in Tables 1 and 2). The PRISMA statement is suggested as a useful framework for reporting.
  • Implications for Practice and Research: The article explains how systematic reviews of reviews can address inconsistencies in results across individual reviews, assess their quality, and produce definitive summaries to inform clinical practice.
• Challenge of Meta-analysis of Reviews: A specific challenge discussed is the creation of a meta-analysis of reviews without double-counting data from individual studies, which could lead to misleading estimates. Overcoming this would require “unpicking” included reviews, a complex and time-consuming task.
• Conclusion: The paper concludes that systematic reviews of reviews provide an invaluable summary of evidence, aiding “evidence-based clinical decision-making” and serving as an “ideal starting point for knowledge from research”. The described methods offer a practical guide for clinicians and researchers.

Why reviews of reviews are needed

Smith and colleagues developed their approach while summarising evidence on maternity care, where hundreds of trials and many overlapping systematic reviews exist. Decision makers increasingly face several reviews on the same question, of varying quality and scope. A systematic review of reviews, or overview, brings these together so that their findings can be compared and the strength of their conclusions judged.

Methodological challenges and solutions

The paper describes challenges at five points. Sources: searching for published and unpublished reviews across databases and organisations that produce reviews, such as Cochrane, Campbell, the Joanna Briggs Institute and NICE. Study selection: setting clear criteria for what counts as a systematic review. Quality assessment: judging how extensively each review searched, how it described study selection and inclusion criteria, whether included studies were comparable, and whether publication bias and heterogeneity were assessed. Presentation of results: summarising and comparing conclusions across reviews. Implications: drawing conclusions for practice and research.

The meta-analysis problem

A major challenge is combining results from reviews that are themselves meta-analyses. If the same primary study appears in several reviews, using its data more than once inflates statistical power and produces estimates that are misleading and too precise. Avoiding this requires unpicking each review and recombining the individual studies, which is complex and time-consuming, so the value of doing it should be weighed when planning the overview.

Practical value

The authors conclude that a review of reviews offers practitioners an efficient starting point for evidence-based decisions, especially in fields with many competing reviews. They also note that rapid reviews are increasingly requested, but their limitations compared with full systematic reviews still need research.

APA Reference:

Smith, V., Devane, D., Begley, C. M., & Clarke, M. (2011). Methodology in conducting a systematic review of systematic reviews of healthcare interventions. BMC Medical Research Methodology, 11(1), 15.