From “Dr. Google” to AI-powered chatbots, the internet is full of health advice. But for a safe checkup or diagnosis, experts say it’s still best to leave it to the professionals.
One of those experts is Ahmed Abdeen Hamed, a research fellow at Binghamton University who recently led a study to evaluate how well artificial intelligence can perform in the medical space. His team focused on the most talked-about tool today: ChatGPT. Published in iScience this summer, the study served to answer a question about AI: Just how well does ChatGPT really know its stuff?
“People talk to ChatGPT all the time these days, and they say: ‘I have these symptoms. Do I have cancer? Do I have cardiac arrest? Should I be getting treatment?’” Hamed said in a news release.
Atlanta-area Wellstar physician Dr. Andrew Thornton emphasized that while it’s possible to use AI and the internet safely for health information, patients should never rely on them during medical emergencies or spend excessive time researching online.
“That is the time to call 911, or get someone to take you to a hospital immediately,” Thornton said.
ChatGPT can’t replace your doctor
ChatGPT is not ready to provide accurate medical advice, though it is showing promise in some specific aspects of medical knowledge. During their study, Hamed’s team made an interesting discovery about those certain aspects, as well as proof that the tool can’t diagnose users effectively.
When it came to identifying disease terms, prescription drugs and information about genes, ChatGPT performed its tasks with 88% to 97% accuracy. However, AI users are often looking for health guidance as well, not just health facts. This is where the large language model — or LLM — began to crack.
“The diseases were really very easy to identify, so ChatGPT was very good at identifying and producing diseases that are actually known in the disease ontology,” Hamed told the AJC.
“(It was the) same story with the drugs, same story with genes, but not the same story with symptoms. And that was a little interesting, because the implications of that are really huge.”
ChatGPT struggled with the complexity of user queries that used vague or casual language to describe symptoms. When questions were conversational, the AI often failed to accurately link symptoms to their potential medical causes.
“ChatGPT uses more of a friendly and social language, because it’s supposed to be communicating with average people,” Hamed said in a news release. “In medical literature, people use proper names. The LLM is apparently trying to simplify the definition of these symptoms, because there is a lot of traffic asking such questions, so it started to minimize the formalities of medical language to appeal to those users.”
The AI’s reluctance to acknowledge when it doesn’t know the correct answer to health-related questions concerned Thornton.
“I think it’s very important for patients to understand that ChatGPT is not going to tell you the confidence with which it is presenting certain bits of information to you,” he said. “It’s going to present it in a way that it sounds very confident in what it’s telling you, and it will do so the same way with inaccurate information as it does with accurate information.”
That convincing AI confidence can lead to significant health risks.
How often do people seek AI health advice?
Artificial intelligence usage in general is growing rapidly.
According to a survey from the Pew Research Center, around 34% of U.S. adults have used ChatGPT at some point in their lives — twice as many as in 2023. Many internet users are also likely receiving information from AI without necessarily seeking it.
Examining 2.5 million webpage visits from March 2025, the center determined that 93% of featured users encountered AI-related content at some point during their web searches. Around 60% of those users visited search result pages featuring AI-generated summaries by simply using Google’s search engine.
Wanted or not, AI information is prominent online these days — and that exposure has led many to ask the bleeding-edge tech for health advice.
About 17% of U.S. adults participating in the 2024 KFF Health Misinformation Tracking Poll said they used AI chatbots at least once a month for health information and medical advice. About a quarter of adults under the age of 30 used AI for health guidance.
Thornton has seen an uptick in patients freely talking about their health-related internet search habits during their checkups, too, something he says happens in an urgent care setting all the time.
“I find that patients are more forthcoming now than 10 years ago, as far as looking things up on the internet,” he said. “I think because they know we expect it, and it’s such a common thing that they go ahead and tell us the concerns that they have based on things they’ve looked up.”
Use it wisely
While ChatGPT is not ready to be your doctor, it still has potential to be a powerful health care tool one day. For now, though, it’s best to leave it to the professionals.
“I think that the internet and AI platforms can be used to sort of add information, to give them more context about different disease states, as well as maybe medications that they are taking,” Thornton said. “I don’t think it should ever be used to narrow down possible diagnoses to what the patient is likely experiencing.
“It really needs to be used for general knowledge about different diseases or medications.”
About the Author
Keep Reading
The Latest
Featured
