Evaluating Knowledge Translation Interventions: A Primer

This article, titled “Methodologies to evaluate the effectiveness of knowledge translation interventions: A primer for researchers and health care managers,” was published in 2011 in the Journal of Clinical Epidemiology by Onil K. Bhattacharyya, Elizabeth A. Estey, and Merrick Zwarenstein. The article focuses on the critical importance of methodologies for evaluating the effectiveness of knowledge translation (KT) interventions. The authors emphasize that although the evidence base for interventions aimed at changing clinical practice is modest but growing, the considerable variability in the effects and costs of such interventions makes evaluation essential for both health care managers and researchers. The goal is to advance the KT field and provide a richer evidence base to increase the use of evidence-based implementation strategies, ultimately improving health care quality. The article distinguishes between the need for local versus generalizable knowledge, which plays a central role in guiding evaluation design and focus.

Evaluation Approaches for Managers and Researchers

• For managers: Managers are encouraged to routinely incorporate simple evaluation designs into program implementation to assess and enhance the impact of their services. For institutional-level managers (e.g., clinic, department, or hospital leaders) engaged in quality improvement (QI), locally applicable knowledge is essential. The aim is to determine whether a particular intervention works in the specific context in which it is implemented. Such evaluations are vital to avoid wasting resources and continuing ineffective interventions. Managers should be encouraged to formally evaluate QI interventions they are already implementing, since an uncontrolled evaluation is better than none. These evaluations must remain within reach of local managers and practitioners.

• For researchers: Researchers should focus on building cumulative knowledge by using rigorous designs and rich contextual and implementation descriptions to investigate the reasons behind success or failure. Their primary concern is generalizable knowledge about whether an intervention is likely to work in similar settings. This requires stronger designs that control for bias and robustly establish causality.

Evaluation Study Designs

The article details various study designs that ensure internal validity (the extent to which an observed effect can be attributed to the intervention under study) and applicability or external validity (the extent to which study results can be generalized to other similar settings).

Ensuring Internal Validity (Establishing Causality)

1. Randomized Controlled Trials (RCTs):
   • Considered the gold standard for evaluating causality. Units (individual, group, or clinic level) can be randomized to an intervention or usual care.
   • Control for confounders: Randomization increases the likelihood that groups are initially similar, ensuring that differences in outcomes are attributable to the intervention. Their ability to balance unknown confounders is particularly important given the complex nature of KT interventions and their modest effects (around 10%).
   • Blinding: Reduces bias in cases where outcome measures are subject to interpretation.
   • Placebo effect: Placebo interventions ensure that the only difference between groups is the receipt of the specific intervention.
   • Sample size: A sufficiently large sample size is required to minimize the likelihood of chance findings. Cluster randomization (e.g., by clinic) requires more complex sample size calculations that account for intracluster correlation.
   • Design options: Two-arm trials (to assess superiority over usual care), multi-arm trials (to compare relative effectiveness), and factorial designs (to test synergy between interventions) are available.
   • Randomization unit: Individuals, providers, or clinics can be randomized. Cluster-level randomization reduces contamination and simplifies administration. Example 1 describes a cluster-randomized trial of family practices for childhood asthma management.
2. Non-Randomized Designs (Quasi-Experimental):
   • More prone to confounding and bias compared to RCTs, but often require fewer resources and are easier to implement at smaller scales.
   • Simple before-after designs: Measure outcomes before and after intervention without a control group. These designs are weak for causal inference due to potential trends, chance, or external factors. Researchers should avoid attributing effects to interventions with this design.
   • Controlled Before-After Studies: Introduce a comparable control group in the same context to provide more reliable answers, though at greater data collection cost. Suitable when randomization is infeasible (e.g., small number of units). Example 2 illustrates such a study for diabetes management using multifaceted interventions versus usual care, known as “firm trials.”
   • Interrupted Time Series: Uses multiple measurements before and after intervention to assess whether observed changes exceed underlying trends. Particularly useful when control groups are unavailable (e.g., media campaigns or policy changes). Requires several pre- and post-intervention time points to identify underlying trends or cyclical phenomena. Example 3 analyzes changes in diuretic prescribing following publication of a major clinical trial to assess media impact on evidence-based practice. Limitations include autocorrelation and inability to control for external factors.

Ensuring Applicability (External Validity)

• Pragmatic Designs: A term introduced by Schwartz and Lellouch (1967) in contrast to “explanatory” trials.
• Explanatory trials assess efficacy under ideal conditions (capacity of an intervention to produce beneficial change).
• Pragmatic trials assess effectiveness under routine practice conditions (how well an intervention works in practice). They maximize relevance for real-world decision-making and are suitable for scaling pilot programs regionally or nationally.
• Key feature: Broad eligibility criteria ensuring high inclusion of providers or facilities. However, selecting highly motivated groups can inflate effect estimates or reduce observed effects due to ceiling effects.

Understanding Success and Failure (Qualitative Studies)

Quantitative studies can assess whether interventions improve outcomes, but cannot explain why they succeed or fail or how they can be improved. Qualitative studies complement quantitative evaluations by clarifying processes and contextual factors.
• Examples: Participant observation to document implementation or interviews/focus groups with providers to assess acceptability.
• Qualitative findings help explain multifactorial reasons for success or failure (e.g., drug coverage, availability, patient acceptability) and inform future intervention design.
• Comprehensive evaluations combining qualitative and quantitative methods provide a holistic understanding of intervention value and effects.

Conclusion

The article concludes that given the time and resources required for KT interventions, they must be rigorously evaluated. Complex strategies should first be piloted and, if promising, formally evaluated in trials. Randomized controlled designs are the least biased but require substantial units (patients, providers, clinics) and may be logistically challenging. When randomization is not feasible, controlled before-after designs or interrupted time series are useful alternatives. However, even with minimal bias, variations in implementation and context complicate interpretation. Weak reporting of intervention, design, implementation, and context makes replication difficult. Therefore, qualitative studies should accompany trials to assess implementation and provide insights into reasons for success or failure. Future studies should include multi-arm or active comparisons to learn which approaches work best and to evaluate relative cost-effectiveness. The authors suggest that combining these methods will advance the KT field and enhance adoption of evidence-based implementation strategies, thereby improving health care quality.

Reference Bhattacharyya, O. K., Estey, E. A., & Zwarenstein, M. (2011). Methodologies to evaluate the effectiveness of knowledge translation interventions: A primer for researchers and health care managers. Journal of Clinical Epidemiology, 64(1), 32–40.