Untrustworthy Evidence in Public Health Policy

The research article titled “Investigating the trustworthiness of research evidence used to inform public health policy: a qualitative interview study on the use of predatory journal citations in policy documents”, authored by Marc A. Albert, Manoj M. Lalu, and Agnes Grudniewicz, and published in Health Research Policy and Systems (2025), critically examines the inclusion of untrustworthy sources in evidence-based policymaking. The study explores how predatory journals—characterized by poor editorial practices and limited peer review—can compromise the integrity of public health policy documents. By relying on such sources, policymakers risk basing decisions on methodologically flawed or misleading evidence, potentially leading to detrimental outcomes at both the patient and population levels.

Background and Purpose: Although evidence-based policymaking has strengthened the ability of decision-makers to incorporate scientific findings into health policy, it also necessitates discernment in source selection. Predatory journals represent a major threat in this context. Defined as “entities that prioritize self-interest at the expense of scholarship,” such journals are marked by deceptive practices, lack of transparency, and absence of rigorous peer review. These journals not only diminish scholarly integrity but have also been found to be widely cited in policy documents—raising concerns about the reliability of the underlying evidence.

The primary objective of this study was to investigate:

1. How research cited in public health policy documents is sourced and evaluated.
2. What factors contribute to the citation of untrustworthy or predatory sources in these documents.

Methodology: A qualitative approach was employed, comprising semi-structured interviews with 15 professionals directly involved in the development of public health policy documents. Interview data were analyzed using thematic analysis to extract key themes and patterns.

Key Findings: Thematic analysis revealed five overarching themes related to how evidence is sourced and evaluated in public health policy documents:

1. Importance of a Clear Plan of Action
   The development of policy documents is influenced by the early establishment of objectives, terms of reference, and methodologies. A well-defined plan enables appropriate evidence synthesis strategies and facilitates allocation of resources, personnel, and funding.
2. Role of Expertise and Collaboration
   The process is inherently collaborative, involving experts in methodology and subject matter. While many contributors have formal training, others acquire competencies through experience. Knowledge of relevant databases and familiarity with high-quality sources are essential for effective evidence retrieval.
3. Balancing Quality and Trustworthiness
   • Common sources include PubMed, Embase, and Epistemonikos, based on assumptions about their reliability.
   • Quality is generally assessed through methodological rigor and completeness of reporting using tools such as AMSTAR, PRISMA, STROBE, and Cochrane Risk of Bias tools.
   • Trustworthiness is associated with the perceived reputability of the journal or database. High-impact journals (e.g., Nature, NEJM) are often seen as inherently credible, despite recognition that unreliable studies may appear even in these venues.
   • Although participants were aware of predatory journals and could identify some based on red flags (e.g., unsolicited emails, rapid peer review), they acknowledged difficulty in definitive identification. Beall’s List, Think.Check.Submit, and similar tools were used, though inconsistently.
   • Notably, most participants prioritized the methodological quality of an article over the source’s overall trustworthiness.
4. Challenges in Evidence Synthesis
   Systematic reviews are considered the gold standard but are often bypassed due to time, cost, and labor constraints. Narrative reviews, while less rigorous, are frequently used in response to practical limitations. Additionally, systematic reviews may exclude regionally relevant or non-English language evidence.
5. Transparency in the Development Process
   All participants emphasized the need for transparency in evidence selection and synthesis methods. This includes detailing both systematic and ad hoc approaches and clearly stating the limitations of the methods employed.

Contributing Factors to Predatory Citations: Several interrelated factors were identified as contributing to the unintentional citation of predatory sources:

• Use of non-systematic assessment techniques.
• Overemphasis on methodological quality without equal consideration of the trustworthiness of publication venues.
• Insufficient methodological training among personnel.
• Lack of access to tools or training in identifying predatory journals.
• Misconceptions about the indexing of predatory journals in reputable databases like PubMed or Embase.

Recommendations: Based on the study’s findings, the authors propose the following strategies to improve evidence quality in health policymaking:

1. Increase Transparency
   Policymakers and researchers should clearly document all steps of evidence sourcing and evaluation. Limitations and methodological decisions must be made explicit for end-users.
2. Assess Trustworthiness Alongside Quality
   Decision-makers should be trained to evaluate both the methodological soundness and the reputability of the sources. This includes recognizing that even legitimate journals may publish flawed or retracted work.
3. Improve Dissemination of Screening Tools
   Greater accessibility and usability of tools such as the Directory of Open Access Journals (DOAJ), Cabell’s Scholarly Analytics, and Think.Check.Submit are essential for improving source vetting processes.

Conclusion: This study offers critical insights into how untrustworthy evidence can infiltrate policy documents and highlights the multifaceted nature of evaluating research for policy. By shedding light on the interplay between source quality, researcher expertise, and institutional practices, the study underscores the necessity of strengthening transparency, trustworthiness assessment, and methodological training to uphold the integrity of public health policy.

Reference: Albert, M. A., Lalu, M. M., & Grudniewicz, A. (2025). Investigating the trustworthiness of research evidence used to inform public health policy: a qualitative interview study on the use of predatory journal citations in policy documents. Health Research Policy and Systems, 23(1), 7.