Is AI better than doctors and nurses at triaging ED patients?

A study comparing emergency medicine doctors and nurses working in the ED with AI found that doctors and nurses were better at triaging patients in the ED. For the study, presented in September at the European Emergency Medicine Congress in Vienna, a paper and digital questionnaire was distributed by the researchers to six emergency medicine doctors and 51 nurses working in the ED of Vilnius University Hospital Santaros Klinikos.

The participants were asked to triage clinical cases selected randomly from 110 reports cited on the internet in the PubMed database. Using the Manchester Triage System, the clinical staff were required to classify the patients according to urgency — placing them in one of five categories from most to least urgent. The same cases were analysed by ChatGPT (version 3.5). In total, 100% of the doctors (6) and 86.3% of the nurses (44) completed the questionnaire.

“We conducted this study to address the growing issue of overcrowding in the emergency department and the escalating workload of nurses,” said Dr Renata Jukneviciene, a postdoctoral researcher at Vilnius University, Lithuania who presented the study. “Given the rapid development of AI tools like ChatGPT, we aimed to explore whether AI could support triage decision-making, improve efficiency and reduce the burden on staff in emergency settings.”

As to the findings, Jukneviciene explained: “Overall, AI underperformed compared to both nurses and doctors across most of the metrics we measured,” adding: “For example, AI’s overall accuracy was 50.4%, compared to 65.5% for nurses and 70.6% for doctors. Sensitivity — how well it identified true urgent cases — for AI was also lower at 58.3% compared to nurses, who scored 73.8%, and doctors, who scored 83.0%.” In all the areas and categories of urgency that the researchers analysed, doctors had the highest scores.

“However, AI did outperform nurses in the first triage category,” Jukneviciene said, “which are the most urgent cases; it showed better accuracy and specificity, meaning that it identified the truly life-threatening cases. For accuracy, AI scored 27.3% compared to 9.3% for nurses, and for the specificity AI scored 27.8% versus 8.3%.” While AI could be useful when used in conjunction with clinical staff, Jukneviciene said it should not be used as a standalone triage tool.

“These results suggest that while AI generally tends to over-triage, it may be somewhat more cautious in flagging critical cases, which can be both a strength and a drawback,” Jukneviciene said. “While we anticipated that AI might not outperform experienced clinicians and nurses, we were surprised that in some areas AI performed quite well. In fact, in the most urgent triage category, it demonstrated higher accuracy than nurses. This indicates that AI should not replace clinical judgement, but could serve as a decision-support tool in specific clinical contexts and in overwhelmed emergency departments.

“AI may assist in prioritising the most urgent cases more consistently and in supporting new or less experienced staff. However, excessive triaging could lead to inefficiencies, so careful integration and human oversight are crucial. Hospitals should approach AI implementation with caution and focus on training staff to critically interpret AI suggestions,” Jukneviciene concluded. Follow-up studies using newer versions of AI and AI models that are fine-tuned for medical purposes are being planned by the researchers, who also want to test them in larger groups of participants, include ECG interpretation, and explore, specifically for triage and incidents involving mass casualties, how AI can be integrated into nurse training.

Image credit: iStock.com/AJ_Watt