The primary objective of this study was to comprehensively assess ChatGPT as a data analysis tool and to establish a practical framework for researchers to effectively apply it to various data-related tasks, including data management, descriptive statistics, and inferential statistics. The overarching aim is to contribute to “a more equitable research landscape without the barriers presented by coding knowledge,” thereby fostering broader access and innovation within the research community.
To achieve this, the researchers employed a rigorous methodology:
- Dataset: A subset of the 2019 National Inpatient Sample from the National Healthcare Cost and Utilization Project was selected, representing real-world observational data typical of clinical research studies. This dataset included 2740 observations with five key attributes: age, gender, race, length of stay, and total charges.
- Assessments: The study evaluated ChatGPT’s capabilities across several data analysis domains:
- Data processing, categorization, and tabulation: Including tasks like reclassifying variables, subsetting variables, and presenting data.
- Descriptive statistics: Calculating mean, standard deviation (SD), median, and interquartile range (IQR).
- Inferential statistics: Covering both parametric tests (chi-square, Pearson correlation, independent 2-sample t-test, 1-way ANOVA) and their nonparametric equivalents (Fisher exact, Spearman correlation, Mann-Whitney U test, Kruskal-Wallis H test).
- Prompt Specificity: A crucial aspect of the inferential statistics assessment involved formulating prompts at three distinct levels of specificity: “Basic,” “Intermediate,” and “Advanced.” These levels reflected varying degrees of user familiarity with statistics, ChatGPT, and Python, impacting the guidance provided to the AI.
- Validation: ChatGPT’s outputs were meticulously validated against expected statistical values generated by established statistical software: Python (Python Software Foundation), SAS (SAS Institute), and RStudio (Posit PBC).
The key findings of the study are particularly insightful:
- ChatGPT demonstrated perfect accuracy in performing data processing, categorization, tabulation, and descriptive statistics (means, SDs, medians, and IQRs) across all trials.
- However, for inferential statistics, accuracy varied significantly based on the specificity of the prompts:
- “Basic” prompts resulted in a low accuracy of 32.5%.
- “Intermediate” prompts significantly boosted accuracy to 81.3%.
- “Advanced” prompts achieved the highest accuracy at 92.5%.
- The authors noted that most calculation errors encountered were attributable to challenges in implementing user instructions and interpreting intermediary outputs, rather than intrinsic issues with the underlying Python-based analytical frameworks, suggesting that precise instructions can mitigate inaccuracies.
In conclusion, the paper asserts that ChatGPT holds “significant potential as a tool for exploratory data analysis,” particularly beneficial for researchers who possess some statistical knowledge but limited programming expertise. Nevertheless, its effective deployment necessitates “careful prompt construction and human oversight to ensure accuracy“. The authors clarify that while ChatGPT can accelerate research by assisting with preliminary analyses and serve as an educational resource, it is explicitly “not intended to be a standalone tool for data analysis”. The study advocates for its use as a “supplementary tool” and provides concrete best practices for researchers, including:
- Formulating specific and comprehensive prompts (at a minimum, resembling the “Intermediate” level).
- Engaging in a dialogue with ChatGPT to refine queries and understand its decisions.
- Attempting each trial at least three times for consistency.
- Crucially, ensuring that all statistical outputs and interpretations are verified by a statistician before publication.
- Maintaining transparency about the use of ChatGPT in research by sharing prompts used in analyses.
REFERENCE:
Ruta, M. R., Gaidici, T., Irwin, C., & Lifshitz, J. (2025). ChatGPT for univariate statistics: Validation of AI-assisted data analysis in healthcare research. Journal of Medical Internet Research, 27, e63550. https://doi.org/10.2196/63550
