This article, titled “Opening-up the definition of systematic literature review: the plurality of worldviews, methodologies and methods for reviews and syntheses,” was co-authored by Pierre Pluye, Quan Nha Hong, Paula L. Bush, and Isabelle Vedel. It was accepted on August 13, 2015, and published online on February 18, 2016, appearing in the Journal of Clinical Epidemiology, Volume 73, pages 2-5. This work specifically originates from the Department of Family Medicine, McGill University, Canada.
The article is presented as a commentary within a larger “SERIES: EMERGING KNOWLEDGE SYNTHESIS METHODS FOR INTEGRATING QUALITATIVE AND QUANTITATIVE EVIDENCE“. This nine-article series serves as a crucial foundation for enhancing the understanding of various methods used in reviews and syntheses. Its primary goal is to assist graduate students and researchers in the processes of planning, conducting, and reporting reviews and syntheses that incorporate qualitative research, either independently or in conjunction with quantitative and mixed-methods studies. The authors emphasize that these methods are increasingly vital for addressing complex issues and questions, making them popular among clinicians and managers. While readers of the Journal of Clinical Epidemiology are typically familiar with systematic reviews of randomized controlled trials and meta-analyses, this series aims to introduce them to a broader spectrum of review and synthesis methods. The fact that JCE, despite its usual focus on quantitative studies, is publishing this series is highlighted as noteworthy and commendable, especially given the dramatic increase in publications reporting review methods that include qualitative studies since the 2000s.
A central contribution of this particular article is its urgent call to expand and clarify the definition of a systematic review, a term often used broadly but rarely defined comprehensively. The authors critique the current usage, noting that some researchers reserve the term for systematic reviews of randomized controlled trials, while others might use it loosely, reporting “nonsystematic reviews of convenience samples of studies”. To address this, they propose a three-part definition for a systematic review, aligning with the diverse evidence used in health systems:
- It must be explicit and transparent.
- It must encompass a type of research (empirical, methodological, theoretical) and a type of study method (qualitative, quantitative, mixed), or a combination thereof.
- It must comprise a reproducible process involving specific research question(s) (exploratory, confirmatory, or both), precise eligibility criteria, a comprehensive set of information sources, an exhaustive search strategy (designed with specialized librarians), a reliable selection of relevant articles (based on relevance criteria that may differ between quantitative and purposeful qualitative samples), a quality appraisal of included studies and data extraction (using appropriate validated criteria for each study design), all performed by at least two researchers, and a rigorous synthesis (using qualitative, quantitative, or mixed synthesis methods). The authors also note that this systematic review process may be iterative, particularly in qualitative and mixed-methods reviews.
The article further distinguishes three main types of reviews and three main types of synthesis methods:
- Review Types:
- Quantitative reviews: focus on studies using quantitative methods, such as randomized trials, non-randomized studies, and descriptive studies.
- Qualitative reviews: focus on qualitative research, including ethnography, grounded theory, phenomenography, biography, and qualitative case studies.
- Reviews including qualitative, quantitative, and mixed-methods studies (mixed methods or mixed studies reviews): these encompass empirical studies, and also include integrative reviews that combine empirical and theoretical studies. Mixed studies reviews specifically combine quantitative, qualitative, and mixed-methods studies and utilize either convergence or sequential synthesis designs.
- Synthesis Methods:
- Quantitative synthesis: examples include frequentist (descriptive or inferential), Bayesian, and Boolean syntheses. The “centennial straightforward reliable quantitative content analysis” for coding results is also mentioned, alongside advanced Bayesian and Boolean methods.
- Qualitative synthesis: includes methods such as descriptive, framework, critical interpretive, meta-narrative, and realist synthesis. Thematic synthesis and conceptual mapping are considered the most common and simpler qualitative synthesis methods, while meta-narrative, realist, and critical interpretive syntheses are more advanced and specialized. These methods are rigorous in the same way qualitative data analyses are, involving iterative processes of moving “back and forth between data and ideas” for interpreting or theorizing data.
- Mixed methods synthesis: involves combining qualitative and quantitative synthesis methods.
A crucial aspect common to all methods in this series is the inclusion of qualitative research studies and the understanding that the proposed synthesis methods are fundamentally interpretive processes. This necessitates a team of researchers with appropriate qualitative research expertise. The rigor of these methods is founded on criteria such as “rich” data (e.g., multiple sources of evidence), comprehensive knowledge of the context, researchers’ reflexivity, and in-depth discussions among researchers. Moreover, their coherence and integrity are rooted in the researchers’ epistemology and ontology—ranging from contemporary postpositivism, social constructionism, realism, pragmatism, or critical theory, or even a conciliation of multiple worldviews within a research team—applied at all stages of the research process (planning, implementation, and reporting). For example, while quantitative methods are often associated with postpositivism, and qualitative methods with social constructionism, a qualitative realist synthesis specifically requires adherence to the principles of realism.
The article highlights challenges in identifying qualitative and mixed-methods studies in bibliographic databases, as they often lack specific subject headings like “clinical trial” for quantitative methods, underscoring the vital role librarians and information scientists can play in developing search filters.
To advance the field, the authors propose an agenda for the development and improvement of review and synthesis methods that incorporate qualitative research:
- Build consensus on the definition of systematic reviews and the key stages of a review process.
- Encourage efforts to develop filters and data mining methods for selecting studies with diverse designs.
- Continue to adapt GRADE (Grading of Recommendations Assessment, Development and Evaluation) for different types of evidence, and develop, validate, and reliability test critical appraisal tools for systematic reviews that include diverse study designs.
- Develop and improve guidance for specific synthesis methods, emphasizing the importance of including team members with qualitative research expertise, particularly for advanced qualitative synthesis methods.
- Specify key methodological elements for reporting in publications, potentially by adapting existing statements like PRISMA.
- Design review assessment tools that acknowledge the plurality of review and synthesis methods. The Cochrane collaboration is suggested as a potential supporter for specialized groups focusing on advanced qualitative (e.g., critical interpretive, framework, metasynthesis, realist synthesis) and quantitative (e.g., Boolean, Bayesian syntheses) methods.
Ultimately, this series of articles aims to foster collaborations among qualitative and mixed-methods researchers. By embracing a plurality of review and synthesis methods, the goal is to combine “the power of numbers and the power of stories,” integrating deductive and inductive explanations, measurement and exploration, and objective and subjective data.
Reference. Pluye, P., Hong, Q. N., Bush, P. L., & Vedel, I. (2016). Opening-up the definition of systematic literature review: the plurality of worldviews, methodologies and methods for reviews and syntheses. Journal of Clinical Epidemiology, 73, 2-5. https://doi.org/10.1016/j.jclinepi.2015.08.033