Dr. Adam Rodman, director of AI programs for the Carl J. Shapiro Center for Education and Research at Beth Israel Deaconess Medical Center
Photo: Adam Rodman
More than one-third of U.S. citizens say they look to artificial intelligence large language models for medical advice. These LLMs, such as ChatGPT and Claude, can offer mounds of information in answer to just about any question. But is that information always correct?
Getting accurate health advice can be challenging, whether it's a question asked of Google or – as in becoming more common – a person uploading a de-identified medical record into ChatGPT.
"People simply Google things, and they use LLMs," said Dr. Adam Rodman. "In terms of exactly how they're using artificial intelligence, we know some of the methods, but we certainly can't yet say what percentage of people are copying in their records or imaging. That remains unknown at this point. But anecdotally, it's a lot – my patients do this all the time."
Rodman is director of AI programs for the Carl J. Shapiro Center for Education and Research at Beth Israel Deaconess Medical Center, and assistant professor at Harvard Medical School. He's also a general internist and a top healthcare AI researcher.
Healthcare IT News spoke with him recently to discuss the issues – good and bad – surrounding patients' use of AI to help them manage their own healthcare.
Q. The 21st Century Cures Act guarantees patients access to their records. You suggest that, rather than try to dig through all of this information, patients remove identifiable information and upload the rest into an AI chatbot to summarize what they're looking for or ask questions about the data. Do you think many patients have the technical chops to know how to download all their health data from a patient portal and then do all that you suggest?
A. No. The 21st Century Cures Act is a fantastic law, and the intent is great. But we know that a tiny minority of patients actually sign up for portals, no less have the technical chops to download their records. This takes a high amount of digital health literacy.
As the technology offers more beneficial things to patients, it's going to widen the gaps for people who know how to do it and people who don't. Looking at my patients, some of them can, but there are very many who don't have the ability. A lot of them don't have portal access or don't even know that there's such a thing as portal access. The majority of people still never even sign up for portals.
One of the challenges with LLMs is they're potentially incredibly helpful, but they're not actually intelligent. If you effectively ask the wrong questions, use them the wrong way, you can get useless or even harmful information.
There's two pieces here: Do I know how to use a portal? Am I good at prompting LLMs and knowing how to talk to them as a patient? I don't think these are skills very many people have.
I talk with my patients who use LLMs. If they use LLMs and show results to me, I talk with them about how they should use AI. My assumption is at the baseline level that knowledge is very low, and there are safety risks with that. Healthcare needs to accept that this is happening and try to educate people about the safest ways to do it.
Q. Do you think patients will have the trust to upload their private medical data to an AI chatbot, even when identifiable data is removed? Might not they believe the chatbot can still somehow connect the data with the user?
A. Trust is really interesting when it comes to AI because there's actually relatively low levels of trust in general, including in healthcare. What is very interesting with AI and digital interventions, the trust level increases dramatically if it's something that's part of your doctor's workflow, if it's something that connects with your doctor. We see this with telehealth interventions.
But no, there's a low level of trust with AI. I will tell you, I don't fully trust the AI companies. That's why if you're going to do this, you need to strip out identifiable information as much as possible. We have mechanisms in this country, like HIPAA. I would like to see healthcare organizations creating institutional chatbots for their patients that have the level of trust that flows from the health system.
It's more complicated than, Do I trust AI or not? It's, Do I trust AI being used by certain institutions? We sympathize with anybody who says I don't trust OpenAI or I don't trust xAI to handle my health data. That's a fair point.
But we already trust health systems to handle our health data. It's all already in cloud storage, and there's a business associate agreement. That is what I would prefer to see happen. It's not happening, which is why people are using public chatbots.
Q. Please explain "cyberchondria" when it comes to patients using AI for medical advice. And what can patients and doctors do to overcome this problem?
A. "Cyberchondria" is old. Some of the early papers quantifying cyberchondria come from 2008 and 2009. Cyberchondria is a phenomenon that exists outside of LLMs.
So, think about how search engine page ranking works. It tries to guess what you want. So, if you search for something benign, like I have a headache, the second link could be dangerous causes of headaches. That algorithm is going to serve up things you're concerned about.
The same phenomenon happens in LLMs. The companies have put lots of safeties in there. But early on, ChatGPT 4 could tell you crazy things, like you have cancer. It doesn't do that anymore. It's actually gotten much, much better.
But still, if you're worried about something subconsciously, you're going to guide the model in that way, the same way as a search algorithm worked on Google back in the late 2000s. One of the things that is frustrating about cyberchondria is while it is a tech problem, it's also a human problem.
It's playing on our anxieties, and the models pick up on this. What I like, because this happens, this is probably the most common thing I need to talk to my patients about, is they start to get really worried about things in AI. And there's nothing you can really do to mitigate it other than to tell people that this is a phenomenon and that the models are going to have a tendency to figure out what you're worried about.
It is a problem that predates LLMs, and it's not going away. It probably existed even before search engines. We just didn't have a good word for it.
Q. What is the most important piece of advice you can give to clinical and IT leaders at hospitals, health systems and group practices when it comes to patients using AI for medical advice?
A. I have a very concrete piece of advice. Legacy IT vendors have a sense of thinking of health tech as something that the hospital buys that goes through a pure procurement process and has knowable failure points. We're in a world where people bring their own health IT. The doctors are bringing their own health IT; the patients are bringing their own health IT – LLMs.
It's really uncomfortable for health IT people. I understand why. This is the new world we live in. If we want to mitigate this somehow, we actually have to embrace the technology and bring it into our health systems, because if we don't, people are going to use it on their own.
Follow Bill's health IT coverage on LinkedIn: Bill Siwicki
Email him: bsiwicki@himss.org
Healthcare IT News is a HIMSS Media publication.
WATCH NOW: The model context protocol at work in AI and clinical workflows
