In an era demanding rigorous inquiry and evidence-based practice, the capacity to conduct high-quality research is paramount, particularly within healthcare and academia. The development of these capabilities hinges on effective research methodology training. A comprehensive study, “Research methodology workshops evaluation using the Kirkpatrick’s model: Translating theory into practice,” published in Medical Teacher, provides a detailed evaluation of such training, offering a robust model for fostering a strong research culture and empowering individuals with essential skills. This article emphasizes the critical role of systematic evaluation in enhancing academic programs and promoting quality improvement efforts.
The Imperative for Research Training and Evaluation: Healthcare professionals are increasingly required to employ logical and systematic approaches to gather vital information, which is then applied to solve scientific and social problems through experimentation, observation, analysis, comparison, and reasoning. Research methodology, as a systematic way to solve research problems, necessitates an understanding of various techniques and their applicability. Research methodology workshops are specifically designed to assist participants with minimal or no prior research experience, those beginning to formulate research questions, or those already engaged in empirical research. Evaluating such programs is crucial for determining if objectives have been met, assessing efficiency, and responding to growing pressures for significant program objective achievement. This study represents one of the few published evaluations of a faculty development program in research in the Kingdom of Saudi Arabia.
Study Objectives and Methodology: Applying Kirkpatrick’s Robust Framework The primary objective of the study was to evaluate five research methodology workshops by assessing participants’ satisfaction, knowledge and skills gain, and the impact on their professional practices, utilizing the four-level Kirkpatrick’s evaluation model. This model, widely recognized as one of the most comprehensive strategies for evaluating organizational training, measures various aspects of training effectiveness, where each level impacts the next.
The evaluation encompassed:
- Level 1: Reaction – Focusing on participants’ perceptions of the training program, their satisfaction, and how they felt about the training received.
- Level 2: Learning – Measuring the extent to which participants acquired knowledge, skills, and experienced changes in attitudes and behaviors. This level often uses pre- and post-tests to gauge knowledge and cognitive skill acquisition.
- Level 3: Behavior – Assessing whether the learned knowledge, skills, and attitudes were transferable to the workplace, leading to positive changes in behavior and job performance.
- Level 4: Results – Examining the main outcomes and overall organizational benefit stemming from the improved performance of participants. This is considered the most critical and challenging level to measure.
Data Collection and Participants: The study employed both quantitative and qualitative methods for data collection. Evaluation tools included:
- Training feedback questionnaires.
- Pre- and post-tests, consisting of 50 single best answer Multiple Choice Questions (MCQs) designed to assess basic applied knowledge of research methodology.
- Learner development plan reports and behavioral surveys.
- 5-point Likert scale questionnaires to gauge general satisfaction and participants’ self-reported ability and attitude towards research, such as their capacity to formulate research questions, willingness to initiate projects, and ability to develop research tools and conduct data analysis.
- Post-workshop follow-ups conducted via email, phone, or face-to-face communication to survey participants’ ongoing research activities.
The study involved 116 participants across five workshops conducted by the Faculty Development Unit (FDU) at King Saud University (KSU) between 2010 and 2013. Participants came from multi-disciplinary backgrounds, including medicine, nursing, pharmacy, dental, biomedical technology, physical therapy, laboratory science, and clinical research. The majority (87.9%) were from health professions education colleges in Saudi Arabia, with 56.1% female, 70.7% clinicians, 17.2% academicians, and 12.1% researchers. Ethical approval for the study was obtained from the Research Committee Board at the College of Medicine, KSU. Statistical analysis utilized a quasi-experimental design, with t-tests comparing pre- and post-test means using SPSS software.
Detailed Findings Across Kirkpatrick’s Levels:
- Level 1: Participant Reaction (Satisfaction and Engagement) The workshops received overwhelmingly positive feedback. Of the 116 participants, 90 (77.6%) liked the workshops. Specifically, 28 (24.1%) expressed appreciation for activities such as small group sessions (10.3%), workshop organization (5.2%), relevant topics (5.2%), and practical SPSS application (3.4%). A significant 62 (53.4%) liked the workshops but provided valuable suggestions for improvement. These suggestions included requests for more exercises and small group sessions with fewer lectures (24.1%), extending workshop duration from three to five days (14.7%), more hands-on SPSS training (6.0%), improved time management (3.4%), additional facilitators (2.6%), and better materials (2.6%). Critically, the organizing committee proactively reviewed and integrated these suggestions into subsequent workshops, leading to improvements such as increased small group discussions, sessions for individual research proposal discussions, and an interactive presentation on data patterns and handling. This responsiveness fostered participant motivation and a sense of personal recognition. While 26 (22.4%) participants expressed dissatisfaction, reasons cited included low voice/thick accents of speakers (8.6%), short time for lengthy topics (7.8%), workshop organization issues (3.4%), and improper time management (2.6%). The study highlights that participant dissatisfaction is a valuable source for actionable improvements.
- Level 2: Learning Outcomes (Knowledge and Skills Acquisition) The evaluation demonstrated a statistically significant improvement in participants’ basic research knowledge and cognitive skills. Across all workshops, there was an overall mean score gain of 17.67% (p≤0.005). The average pre-test score of all five workshops was 23.30 ± 12.1, which significantly increased to a post-test score of 32.14 ± 11.01 (p=0.005). Individual workshops showed gains ranging from 9.28% to 20.35%. Analyzing by sub-topic, there were statistically significant improvements in SPSS (p=0.001), biostatistics (p=0.015), data collection (p=0.031), study design (p=0.001), and qualitative research (p=0.011). Participants initially struggled with biostatistics and SPSS, scoring below 50% in pre-tests, but showed considerable progress post-workshop. However, improvements in writing manuscripts (p=0.834) and writing proposals (p=0.404) did not reach statistical significance. This suggests that these particular skills may require more intensive or repetitive interventions to develop fully.
- Levels 3 & 4: Behavioral Change and Organizational Results (Impact on Practice) The workshops’ impact extended tangibly into the participants’ professional lives. Post-workshop follow-up revealed that a remarkable 56.9% (66 out of 116) of participants initiated new research projects, while 6.9% (8 out of 116) successfully published their research articles. Furthermore, 79% of participants reported transferring their newly acquired training skills to their workplace. The study acknowledges that while these outcomes are highly encouraging, changes in behavior and performance cannot be attributed solely to the workshops due to potential confounding factors, such as participants’ prior backgrounds. Nevertheless, these findings strongly suggest that the workshops played a significant role in fostering a “research culture” and promoting valuable work-based learning within the institution.
Conclusion and Broader Implications: The rigorous evaluation using Kirkpatrick’s model provides compelling evidence that well-structured and continuously refined research methodology workshops are highly effective. They not only significantly improve participants’ research-related knowledge and skills but also meet international standards for quality education, empowering individual learners to become confident and competent researchers. This, in turn, is expected to increase the institution’s overall research capacity. The inter-professional nature of the participants was identified as a key benefit, facilitating the exchange of diverse perspectives and further promoting a robust research culture. The study underscores that encouraging a “research culture” and embracing work-based learning are among the most powerful determinants for promoting successful research initiatives. Proactively seeking and utilizing learners’ feedback is highlighted as a valuable tool for enhancing enthusiasm and progression in research methodology workshops. The success of these workshops offers valuable guidance for educational planners aiming to improve their training programs.
Reference:
Abdulghani, H. M., Shaik, S. A., Khamis, N., Al-Drees, A. A., Irshad, M., Khalil, M. S., Alhaqwi, A. I., & Isnani, A. (2014). Research methodology workshops evaluation using the Kirkpatrick’s model: Translating theory into practice. Medical Teacher, 36(sup1), S24–S29. https://doi.org/10.3109/0142159X.2014.886012