Photo: MoMo Productions/Getty Images
ECRI, the independent and nonpartisan patient safety organization, published its annual list of Top 10 health technology hazards this past week. Unsurprisingly, as it is in so many other areas of healthcare, generative AI is making its presence felt.
WHY IT MATTERS
The safety group places artificial intelligence chatbots in healthcare atop its 2026 list, noting that large language models such as ChatGPT, Claude, Copilot, Gemini and Grok, which "produce human-like and expert-sounding responses to users’ questions," are nonetheless not regulated as medical devices even as they're used more and more by patients and providers alike.
ECRI points out that more than 40 million people daily turn to ChatGPT for health information, for instance, despite the fact that genAI chatbots can provide false or misleading information or exacerbate healthcare disparities.
"For example, one chatbot gave dangerous advice when ECRI asked whether it would be acceptable to place an electrosurgical return electrode over the patient’s shoulder blade," the organization noted. "The chatbot incorrectly stated that placement was appropriate – advice that, if followed, would leave the patient at risk of burns."
ECRI, which is hosting a webcast to discuss its list on Jan. 28, says the risk to patients could increase as more provider organizations close their doors, potentially leading more people to rely on AI for medical advice.
Here's ECRI's full Top 10 list of healthcare technology hazards:
- Misuse of AI chatbots in healthcare
- Unpreparedness for a "digital darkness" event, or a sudden loss of access to electronic systems and patient information
- Substandard and falsified medical products
- Recall communication failures for home diabetes management technologies
- Misconnections of syringes or tubing to patient lines, particularly amid slow ENFit and NRFit adoption
- Underutilizing medication safety technologies in perioperative settings
- Inadequate device cleaning instructions
- Cybersecurity risks from legacy medical devices
- Health technology implementations that prompt unsafe clinical workflows
- Poor water quality during instrument sterilization
THE LARGER TREND
New products and integrations from Anthropic's Claude AI and OpenAI – which launched separate services known as ChatGPT for Healthcare and ChatGPT Health – have made news in recent weeks.
Despite the potential for improved patient engagement and reduced clinician workload, concerns have also been aired about fundamental challenges related to governance.
Past ECRI health tech hazard lists have included risks such as other AI-enabled health technologies, which topped the December 2024 edition, and home-based medical devices, which led the previous edition.
ON THE RECORD
"Medicine is a fundamentally human endeavor. While chatbots are powerful tools, the algorithms cannot replace the expertise, education, and experience of medical professionals," said Dr. Marcus Schabacker, president and chief executive officer of ECRI. "Realizing AI’s promise while protecting people requires disciplined oversight, detailed guidelines, and a clear-eyed understanding of AI’s limitations.
"AI models reflect the knowledge and beliefs on which they are trained, biases and all," said Schabacker. "If healthcare stakeholders are not careful, AI could further entrench the disparities that many have worked for decades to eliminate from health systems."
Mike Miliard is executive editor of Healthcare IT News
Email the writer: mmiliard@himss.org
Healthcare IT News is a HIMSS publication.
