Photo: Cottonbro Studio/Pexels
Two of the leading companies of the generative AI race, Anthropic and OpenAI, have made news this month with splashy announcements of their own respective offerings tailored for the healthcare space.
But as new dedicated health artificial intelligence tools are tapped for use by health systems and are integrated with electronic health records and other health IT tools, providers – and patients – face increased risks.
AI may solve some of healthcare's greatest challenges, but experts who spoke with Healthcare IT News this week say there's also plenty of cause for concern.
For instance, the potential for AI to deliver inaccurate medical information – those infamous "hallucinations" – combined with some models' false sense of diagnostic certainty, remains a big challenge.
Moreover, there's very often a lack of clinical liability for these tools, which creates significant legal and ethical gaps related to who's held accountable when an algorithm-influenced recommendation results in patient harm.
As consumer genAI tools seek wider acceptance in the provider space, there's a lot to think about when it comes to where and how they're deployed.
Risks for providers and patients
In the past week, OpenAI and Anthropic announced dedicated AI aiming to improve the use of medical records by healthcare organizations, including doctors and hospitals.
With the launch of OpenAI's Healthcare product suite, several health systems will use ChatGPT for Healthcare to support patient care and reduce administrative burdens, the AI company said.
Clinical teams at health systems, including AdventHealth, Boston Children's, Cedars-Sinai, HCA, Memorial Sloan Kettering, Stanford Medicine and others, can use ChatGPT for Healthcare to integrate medical evidence into their patient care work, OpenAI said in its announcement.
Elation Health, which offers an electronic health record platform, said this week that it has integrated Anthropic's Claude AI into its clinical insights program and that doctors are using it to create instant summaries of complete patient records.
Medical professionals using clinical insights software that taps into Claude AI are getting 61% faster answers to their questions, according to the EHR vendor.
OpenAI and Anthropic are aligning their AI development with the Trump Administration's AI priorities and action plans to harness technology and empower patients to make better decisions regarding their health, according to Michael Abrams of Numerof & Associates, a life sciences industry consultant.
Patients can upload medical records directly to a separate product called ChatGPT Health or connect their wellness app data and ask the AI for insights. They can also connect to Claude as a HealthEx app subscriber and ask questions about their medical history across multiple providers.
"First, we have to acknowledge the math: There simply aren't enough physicians and medical staff to meet global demand," Adam de la Zerda, CEO and founder of Visby Medical, a medical diagnostics company, said in response to our query.
"Access to care is a crisis, and AI is the only tool capable of scaling that expertise, so this is a terrific step in the right direction."
While the foundational large language models like OpenAI's ChatGPT and Anthropic's Claude can be used to gather and analyze patient information and improve the understanding of available treatment options, there are also plans to use the LLMs "to automate record-keeping and form-filling, to develop process improvement plans, to improve operational efficiency, and more," Abrams said.
"Responsible boards and executives must recognize the risks associated with these new tools and take steps to protect the institution and staff that will be involved," Abrams told Healthcare IT News on Tuesday.
Remaining realistic on AI
Use of health LLMs requires healthcare leaders to be realistic and disciplined when it comes to governance and accountability.
"While data privacy is table stakes, the real governance challenge is the decoupling of certainty from accountability," de la Zerda said Wednesday.
"Success won't come from model quality alone," Dr. Chase Feiger, CEO and cofounder of Ostro, an AI software company, pointed out. "It'll come from whether organizations bring real discipline around governance, accountability and how medicine actually works."
"All of the stakeholders understand that the greatest interest in these tools will be for diagnosis, by consumers, patients and physicians," Abrams said. "They also must realize that for diagnostic purposes, these tools, despite the testing that has been done, carry risk for users and for the institutions of which they are a part."
In creating dedicated health use cases for these frontier LLMs, the companies state that they encrypt health data by default, both at rest and in transit, and store the data separately so it is not used to train the models.
However, a more critical concern might be that an LLM offers answers with the tone of diagnostic certainty, "yet it bears no clinical liability," de la Zerda noted.
"This creates a dangerous gap where a patient might anchor on an authoritative-sounding AI summary that lacks professional nuance," he said. "Governance must ensure that we don't replace human judgment with an algorithm that is confident, but ultimately unaccountable."
"Natural language processing capabilities enable AI to go beyond simply sharing information," Ali Diab, CEO and co-founder of Collective Health, a health benefits platform developer, said in response to our query.
"That introduces interesting new questions about who is responsible if that analysis or recommendation is wrong."
Questions to ask first
"In an industry like healthcare, where information informs life or death decisions, safety and accuracy is essential," Simon Kos, Heidi Health's global chief medical officer, said in a statement shared with Healthcare IT News. "That's why doctors go to medical school for years."
But "just meeting existing regulations like HIPAA is insufficient to ensure safety," he said.
"We do need to acknowledge the significant challenges in the current system before we hold AI to a higher bar," Kos said. "Human mistakes occur today, and the delays caused by workforce shortages and process inefficiencies adversely affect patient outcomes."
Abrams said he is advising healthcare delivery executives and boards to approach governance in advance of rollout from a position of mitigating tool risks.
"This requires procedures for dealing with situations that are likely to occur, like when the AI tool renders a diagnosis that is provably wrong," Kos explained.
Other questions he advises healthcare leaders to answer:
- Who will be accountable for decisions influenced by the tool – the clinician, department, vendor or hospital?
- What will be needed to defend AI-influenced decisions in malpractice or peer-review proceedings, and how do we ensure we can meet those requirements?
- Are there domains in which the use of the tools should be restricted until more is understood about capabilities and accuracy?
The upside is upstream
When used thoughtfully, LLMs like GPT and Claude belong before and after the point of care, "not autonomously in it," Feiger said.
"What matters is that OpenAI is finally treating healthcare as a category with different rules, lower tolerance for error and real consequences when things go wrong," he said.
While helping patients use and understand their medical records and clinicians to improve the speed at which they use patient data are advantageous to the health sector, healthcare leaders must understand where the line should be drawn.
"What concerns me isn't the model itself; it's how easily tools like this can be over-trusted," Feiger said. "They sound confident, even when they're wrong, and in medicine, that’s dangerous if boundaries aren’t explicit."
If AI influences a clinical action, "responsibility can't be vague," he continued. "Consumer use and enterprise use also can't be blurred."
When embedding AI into workflows, "accuracy failures are usually subtle, not obvious, especially in edge cases like pregnancy, pediatrics, rare disease or complex medication regimens," he noted.
Diab said that he believed inherent risks on consumer use of health LLMs could be managed "with the appropriate level of disclosure and, potentially, regulation," and offered China as a potential instructive example.
The nation's AI regulatory framework classifies some AI applications as Class III medical devices that need the equivalent of a U.S. Food and Drug Administration approval process, he explained.
Developers should not shy away from regulatory oversight, he said, and recommends they proactively engage with governing agencies.
"If American consumers start to rely heavily on these AI-based tools to self-diagnose, and if that leads to consumers potentially making deductions or decisions regarding their health that don’t end up being the right ones, and in doing so, incur harm to themselves or a loved one, I think we'll face a reckoning for these tools that the excitement around them right now may overlook," he said.
Andrea Fox is senior editor of Healthcare IT News.
Email: afox@himss.org
Healthcare IT News is a HIMSS Media publication.
