An Evaluation of Artificial Intelligence Accuracy in Cardiovascular Pharmacology Education
This study, titled “Artificial intelligence in healthcare education: evaluating the accuracy of ChatGPT, Copilot, and Google Gemini in cardiovascular pharmacology,” investigates the performance of three prominent generative artificial intelligence (AI) tools—ChatGPT-4, Microsoft Copilot, and Google Gemini—in answering questions related to cardiovascular pharmacology. Published by Salman, Ameer, Khanfar, and Hsieh in Frontiers in Medicine in 2025, this research addresses the increasing integration of AI into healthcare education and the critical need to understand its capabilities and limitations.
The background of the study highlights that generative AI models like ChatGPT, Copilot, and Gemini are significantly influencing the educational landscape, particularly in healthcare and medical education, where students increasingly use them for studying, tutoring, exam preparation, and assignments. These tools, functioning as large language models (LLMs), are trained on vast datasets to provide real-time assistance and tailored information, thanks to advancements in natural language processing (NLP). While they have diverse primary functions—ChatGPT for general text generation, Copilot for code completion, and Gemini for accurate text responses in specialized fields—all three have been adapted for educational purposes. Cardiovascular pharmacology, a core subject requiring mastery of factual knowledge and its clinical application, serves as an ideal domain for evaluating AI accuracy due to its complexity.
To systematically compare the accuracy of these AI tools, the researchers conducted an experimental study using the free-access versions of ChatGPT (GPT-4o mini), Microsoft Copilot (GPT-4 Turbo), and Google Gemini (Gemini 1.5). The methodology involved administering 45 multiple-choice questions (MCQs) and 30 short-answer questions (SAQs) across three difficulty levels: easy, intermediate, and advanced. Questions, designed by an experienced pharmacology professor and validated by two additional pharmacology professors specializing in cardiovascular pharmacology, assessed basic recall, application, integration, and critical thinking skills. AI-generated answers were reviewed and graded by these three pharmacology experts; MCQs were marked as correct or incorrect, while SAQ responses were rated on a 1–5 scale based on relevance, completeness, and correctness. To ensure unbiased evaluation, SAQ answers were anonymized, and inter-rater reliability among evaluators was assessed using Fleiss’ Kappa, which indicated almost perfect agreement.
The study yielded several key findings:
- MCQ Performance:
- All three AI tools demonstrated high accuracy (87–100%) in easy and intermediate MCQs, suggesting their reliability for foundational and moderately complex factual knowledge.
- However, a decline in performance was observed for all AI models on advanced MCQs.
- ChatGPT scored highest (73%) in advanced MCQs, while Copilot achieved 53% accuracy and Google Gemini notably struggled with 20% accuracy.
- Copilot and Gemini had significantly lower scores on advanced MCQs compared to their easy-intermediate performance, whereas ChatGPT’s decline was less pronounced.
- Overall, ChatGPT and Copilot performed similarly, with Gemini showing the lowest accuracy, particularly when compared to ChatGPT.
- SAQ Performance:
- ChatGPT and Copilot demonstrated consistently high accuracy scores (overall 4.7 ± 0.3 and 4.5 ± 0.4, respectively) across all SAQ difficulty levels, with no significant differences between them or across difficulty levels. This indicates their robust capability to provide detailed explanations for question-based learning.
- In contrast, Gemini’s SAQ performance was markedly lower across all levels (overall 3.3 ± 1.0), performing significantly worse than both ChatGPT and Copilot. Its performance was particularly poor in the advanced SAQ section.
In conclusion, ChatGPT-4 emerged with the highest overall accuracy in both MCQ and SAQ cardiovascular pharmacology questions, irrespective of difficulty. Copilot ranked second, showing competitive performance, while Google Gemini exhibited significant limitations in handling complex MCQs and providing accurate SAQ responses. The study suggests that while AI tools are promising for basic factual learning, their limitations become more evident with increasing subject complexity. Therefore, educators and students should exercise caution when relying on AI for complex tasks and use AI as a supplement to, rather than a replacement for, traditional learning methods, especially for advanced or clinical material. The findings underscore the importance of ongoing assessment and refinement of AI models to optimize their reliability in specialized medical education.
Reference:
Salman, I. M., Ameer, O. Z., Khanfar, M. A., & Hsieh, Y-H. (2025). Artificial intelligence in healthcare education: evaluating the accuracy of ChatGPT, Copilot, and Google Gemini in cardiovascular pharmacology. Frontiers in Medicine, 12, 1495378. https://doi.org/10.3389/fmed.2025.1495378
