The research article titled “Identifying healthcare needs with patient experience reviews using ChatGPT” by Li et al. (2025) addresses the critical challenge of efficiently assessing and distributing medical resources, which is often hampered by traditional assessment tools that fail to capture actual patient demands. The authors propose a novel methodology that leverages the recent breakthroughs in Large Language Models (LLMs), specifically ChatGPT, to effectively understand healthcare needs directly from patient experience reviews. This approach aims to improve the quality of care, enhance the patient visit experience, and ultimately avoid unnecessary waste of healthcare resources.
Key Aspects of the Study:
- Objective: The primary objective was to develop a method that uses LLMs to analyze patient feedback and identify specific healthcare needs.
- Methodology: The study utilized a substantial dataset of 504,198 patient reviews collected from haodf.com, a prominent online medical platform in mainland China. The core of their methodology involves the development and application of Aspect-Based Sentiment Analysis (ABSA) templates. These templates categorize patient reviews into three clinically significant areas of concern: “patient experience,” “physician skills and efficiency,” and “infrastructure and administration”. To enhance the LLM’s analytical capabilities, these ABSA templates were embedded into prompts for ChatGPT-4o, along with the Chain of Thought (CoT) technique, which guides the model to perform step-by-step logical reasoning for sentiment analysis.
- Performance and Reliability: The proposed method demonstrated outstanding performance, achieving a weighted total precision of 0.944, a weighted total recall of 0.884, and a weighted F1 score of 0.912. These results significantly surpassed those obtained from “direct narrative tasks in ChatGPT”. Furthermore, the model exhibited high reliability and stability, with an average accuracy of over 91.5% across three different random sampling methods.
- Practicality and Cost-Effectiveness: The study also evaluated the practical applicability of their approach, noting that the average cost of processing 1,000 reviews was approximately USD 1.9, with an average processing time of 55 minutes per 1,000 reviews, or about 3.5 seconds per entry. This suggests the method is promising for real-world applications in healthcare settings.
- Contributions and Applications: This research makes significant contributions by providing a feasible framework for clinical applications and introducing robust evaluation methods for LLMs in healthcare. It sheds light on understanding patient and health consumer demands and can contribute to enhancing patient experience and better healthcare resource allocation. The framework can also be used by governments and healthcare organizations for macro-assessment of healthcare resource distribution, policy reliability assessment, and policy refinement by comparing patient sentiments before and after policy implementations.
Limitations: The authors acknowledge limitations, including the potential for a more granular categorization scheme, the possibility of censorship affecting review representation on the platform, and the exclusion of the latest CoT advancements like Tree of Thought (ToT) or Graph of Thought (GoT). They also suggest future work on hyperparameter optimization and automatic labeling of large datasets.
In summary, this study highlights the significant potential of integrating LLMs with structured analytical frameworks like ABSA templates for deep, efficient, and reliable sentiment analysis of patient feedback, offering a valuable tool for improving healthcare quality and resource management.
Reference: Li, J., Yang, Y., Chen, R., Zheng, D., Pang, P. C.-I., Lam, C. K., Wong, D., & Wang, Y. (2025). Identifying healthcare needs with patient experience reviews using ChatGPT. PLoS ONE, 20(3), e0313442. https://doi.org/10.1371/journal.pone.0313442
