JOMPAC

Journal of Medicine and Palliative Care (JOMPAC) is an open access scientific journal with independent, unbiased, and double-blind review under international guidelines. The purpose of JOMPAC is to contribute to the literature by publishing articles on health sciences and medicine.

EndNote Style
Index
Original Article
Comparative evaluation of large language models for preliminary diagnosis and initial treatment decision-making in intensive care unit emergencies
Aims: Timely and accurate decision-making is essential in intensive care unit (ICU) emergencies, where delays or errors in preliminary diagnosis and initial treatment may lead to adverse outcomes. Large language model–based Artificial Intelligence (AI) systems have shown promise in clinical reasoning tasks, yet their performance in high-acuity, time-sensitive ICU emergencies remains insufficiently studied.
Methods: This simulation-based observational study evaluated the performance of three large language model–based AI systems (Gemini, DeepSeek, and ChatGPT) in preliminary diagnosis and initial treatment decision-making across 84 simulated ICU emergency scenarios. Scenarios reflected real-world ICU practice and covered bleeding, cardiac, electrolyte imbalance, neurologic, respiratory, and other critical conditions. Each scenario included two multiple-choice questions (diagnosis and treatment). Responses from AI models were compared with the expert-derived reference standards established by two boardcertified anesthesiology specialists, with adjudication by a third expert in cases of disagreement. Model performance was assessed using accuracy, macro-averaged sensitivity, specificity, F1 score, and Cohen’s kappa coefficient. Subgroup analyses were performed by emergency type. The study was designed and reported in accordance with the TRIPOD-LLM statement.
Results: Overall diagnostic accuracy was 0.93 for Gemini, 0.92 for DeepSeek, and 0.65 for ChatGPT. Treatment decision accuracy was 0.98 for Gemini, 0.95 for DeepSeek, and 0.68 for ChatGPT. Agreement with the expert-derived reference standard was substantial for Gemini (?=0.743 for diagnosis; ?=0.741 for treatment), substantial for DeepSeek diagnosis (?=0.710) and moderate for DeepSeek treatment (?=0.584), but low for ChatGPT (?=0.144 and ?=0.085, respectively). Gemini and DeepSeek demonstrated consistently high subgroup accuracy across all emergency categories, including perfect performance in respiratory emergencies, whereas ChatGPT showed greater variability and lower accuracy across subgroups.
Conclusion: In simulated ICU emergency scenarios, Gemini and DeepSeek demonstrated high diagnostic and treatment decision accuracy, although agreement with the expert-derived reference standard varied by model and task, whereas ChatGPT showed lower and more variable performance. These findings suggest that selected large language models may have potential as decision-support tools in critical care settings; however, performance varies markedly between models. Further validation using real-world clinical data is required before clinical implementation.


1. Dost B, Turan Eİ, Aydın ME, et al. Artificial Intelligence in anaesthesiology: current applications, challenges, and future directions. Turk J Anaesthesiol Reanim. 2025;53(6):282-292. doi:10.4274/TJAR.2025.252320
2. Turan Eİ, Baydemir AE, Özcan FG, Şahin AS. Evaluating the accuracy of ChatGPT-4 in predicting ASA scores: a prospective multicentric study ChatGPT-4 in ASA score prediction. J Clin Anesth. 2024;96:111475. doi:10.1016/j.jclinane.2024.111475
3. Turan Eİ, Baydemir AE, Balıtatlı AB, Şahin AS. Assessing the accuracy of ChatGPT in interpreting blood gas analysis results ChatGPT-4 in blood gas analysis. J Clin Anesth. 2025;102:111787. doi:10.1016/j.jclinane.2025.111787
4. Abad-Gurumeta A, Gómez-Ríos MÁ. Artificial Intelligence in anesthesiology and critical care: transformative potential and unresolved ethical challenges. Not all that glitters is gold. Minerva Anestesiol. 2025;91(4):254-256. doi:10.23736/S0375-9393.25.18911-6
5. Abdel Malek M, van Velzen M, Dahan A, et al. Generation of preoperative anaesthetic plans by ChatGPT-4.0: a mixed-method study. Br J Anaesth. 2025;134(5):1333-1340. doi:10.1016/j.bja.2024.08.038
6. Ismaiel N, Nguyen TP, Guo N, Carvalho B, Sultan P; study collaborators. The evaluation of the performance of ChatGPT in the management of labor analgesia. J Clin Anesth. 2024;98:111582. doi:10.1016/j.jclinane.2024.111582
7. Ruan Q, Shi J, Dai Y, Yang P, Zhu N, Wang S. Performance of large language models in complex anesthesia decision-making: a comparative study of four LLMs in high-risk patients. J Med Syst. 2025;49(1):122. doi:10.1007/s10916-025-02247-3
8. Guerrier G, Cargeac A, Courtois F, et al. Effectiveness of artificial intelligence as a support tool in preoperative anaesthesia assessments for cataract surgery: a step towards enhancing patient management. Br J Anaesth. 2025;135(5):1383-1385. doi:10.1016/j.bja.2025.08.013
9. Gisselbaek M, Suppan M, Minsart L, et al. Representation of intensivists' race/ethnicity, sex, and age by artificial intelligence: a cross-sectional study of two text-to-image models. Crit Care. 2024;28(1):363. doi:10.1186/s13054-024-05134-4
10. Jois SM, Rangalakshmi S, Iyengar SMJ, Mahesh C, Devi LD, Namachivayam AK. Artificial Intelligence enhanced Chatbot boom: a single center observational study to evaluate assistance in clinical anesthesiology. J Anaesthesiol Clin Pharmacol. 2025;41(2):351-356. doi:10.4103/joacp.joacp_151_24
11. Balta KY, Javidan AP, Walser E, Arntfield R, Prager R. Evaluating the appropriateness, consistency, and readability of ChatGPT in critical care recommendations. J Intensive Care Med. 2025;40(2):184-190. doi:10.1177/08850666241267871
12. Gallifant J, Afshar M, Ameen S, et al. The TRIPOD-LLM reporting guideline for studies using large language models. Nat Med. 2025;31(1): 60-69. doi:10.1038/s41591-024-03425-5
13. Hoppe JM, Auer MK, Strüven A, Massberg S, Stremmel C. ChatGPT With GPT-4 outperforms emergency department physicians in diagnostic accuracy: retrospective analysis. J Med Internet Res. 2024;26: e56110. doi:10.2196/56110
14. Park C, An MH, Hwang G, Park RW, An J. Clinical performance and communication skills of ChatGPT versus physicians in emergency medicine: simulated patient study. JMIR Med Inform. 2025;13:e68409. doi:10.2196/68409
15. Wang J, Shue K, Liu L, Hu G. Preliminary evaluation of ChatGPT model iterations in emergency department diagnostics. Sci Rep. 2025;15(1): 10426. doi:10.1038/s41598-025-95233-1
16. Dinc MT, Bardak AE, Bahar F, Noronha C. Comparative analysis of large language models in clinical diagnosis: performance evaluation across common and complex medical cases. JAMIA Open. 2025;8(3):ooaf055. doi:10.1093/jamiaopen/ooaf055
17. Urda-Cîmpean AE, Leucu?a DC, Drugan C, Du?u AG, Calinici T, Drugan T. Assessing the accuracy of diagnostic capabilities of large language models. Diagnostics (Basel). 2025;15(13):1657. doi:10.3390/diagnostics15131657
18. Singh J, Bohra R, Mukhtiar V, et al. Diagnostic accuracy of a large language model (ChatGPT-4) for patients admitted to a community hospital medical ıntensive care unit: a retrospective case study. J Intensive Care Med. 2026;41(5):413-420. doi:10.1177/08850666251368270
19. Turan EI, Baydemir AE, Şahin AS, Özcan FG. Effectiveness of ChatGPT-4 in predicting the human decision to send patients to the postoperative intensive care unit: a prospective multicentric study. Minerva Anestesiol. 2025;91(4):259-267. doi:10.23736/S0375-9393.24.18587-2
20. Biesheuvel LA, Workum JD, Reuland M, et al. Large language models in critical care. J Intensive Med. 2024;5(2):113-118. doi:10.1016/j.jointm.2024.12.001
21. Berkhout WEM, van Wijngaarden JJ, Workum JD, et al. Operationalization of Artificial Intelligence applications in the ıntensive care unit: a systematic review. JAMA Netw Open. 2025;8(7):e2522866. doi:10.1001/jamanetworkopen.2025.22866
Volume 7, Issue 5, 2026
Page : 985-992
_Footer