Picture this: you’re sixty years old, trying to live a healthier life. Maybe you’ve just gotten a blood test showing high chloride levels, or maybe you read an article about electrolytes and got worried about your salt intake. You decide to ask ChatGPT for advice because it feels responsive, private, and impossibly knowledgeable. That is exactly what happened to one 60-year-old man, and it turned into a nightmare. He wanted to reduce chloride in his diet, and the chatbot suggested replacing ordinary table salt with sodium bromide. To a person without a chemistry degree, that sounds like a clever, science-backed hack. But sodium bromide is not a seasoning or a diet aid; it’s a chemical compound that can build up in the body and cause bromism, a form of poisoning marked by fatigue, vomiting, memory loss, hallucinations, paranoia, seizures, and even psychosis. The man followed the advice, and before long he began to lose touch with reality. He became suspicious, confused, and terrified, and his family had no choice but to rush him to a hospital. There, doctors faced a second problem: they couldn’t retrieve the ChatGPT conversation to see exactly what advice had caused the damage. No transcript, no record, no way to trace the poisoning back to its source. The case, later detailed in a research paper on the hidden dangers of AI-generated health advice, captures the strange new vulnerability of our digital age. We put our trust in a machine that speaks like a human, acts like an expert, and obeys no rules of professional accountability. It’s easy to imagine making the same mistake. You ask a simple question, you get a confident answer, and you don’t think twice until it’s too late.
The paper was written by researchers from Binghamton University, Stanford University, Texas A&M University, and Indiana University, and its central argument is unsettling: the dangers of AI health advice are often invisible to everyone except the victim. If a family doctor gives you wrong advice, there is a record. You can demand an explanation, file a complaint, or seek a second opinion. But when a chatbot gives you wrong advice, the conversation can disappear as if it never happened. That leaves clinicians, researchers, and regulators in the dark. They cannot track dangerous patterns, warn potential victims, or measure how often the problem occurs. And the problem is growing fast. A Gallup survey of more than 5,500 U.S. adults, conducted between October 27 and December 22, 2025, found that about 14 million adults reported skipping a visit to a health-care provider after using AI for health information. That is an enormous number of people making medical decisions based on algorithms that are not licensed, monitored, or audited. As Kaicheng Yang, one of the paper’s authors, told Newsweek, AI can be very helpful for health questions, but it can also state wrong answers with the same confidence as correct ones. The consequences, he said, might include delaying treatment and taking the wrong medication. The scariest part, Yang admitted, is that nobody knows how dangerous it is right now, because the harms are so often hidden. We only see the rare cases that end badly enough to make the news, not the thousands of quieter moments where AI gives bad advice and no one ever finds out.
OpenAI, the company behind ChatGPT, has responded to cases like this with caution. A spokesperson told Newsweek that ChatGPT is not a doctor and should never be used as a substitute for medical care, diagnosis, or treatment. They noted that their terms clearly state ChatGPT isn’t intended for medical diagnosis or treatment, but they also acknowledged that people still use it for health information, and they take the need to make those outputs as reliable and safe as possible. The spokesperson emphasized progress: ChatGPT is powered by OpenAI’s latest underlying models, which continuously improve in recognizing when a situation may need urgent attention, handling uncertainty with better judgment, and giving people clearer, more useful guidance. They also mentioned close partnerships with hundreds of physicians from 60 countries, representing dozens of specialties, who help define, measure, and improve health responses in ChatGPT. That all sounds reassuring, but it doesn’t solve the transparency problem. When a person types “how can I reduce chloride?” into a chat box at two in the morning, they aren’t reading the terms of service. They’re anxious and looking for help. When the chatbot answers in a calm, confident tone, they trust it. A disclaimer somewhere in the interface doesn’t protect someone from a hallucinated recommendation that lands them in a hospital. And even if OpenAI has made newer models safer, the man in this case can’t be helped by that progress. His conversation was never saved in a way his doctors could access, so his case remains a throwaway example rather than a lesson with clear evidence. The larger issue remains: AI systems operate with little visibility into the health advice they provide. If users can’t access their own health-related chatbot interactions, they can’t share them with doctors when something goes wrong. If regulators can’t inspect what these systems recommend, they can’t hold anyone accountable. It’s a black box with a smiling face.
So what should ordinary people do? The researchers aren’t calling for a complete rejection of AI. Yang was explicit: “We are not asking users to stop using AI for health questions.” Instead, the advice is to treat the answer as a starting point, not a diagnosis. Use AI to understand unfamiliar medical terms, prepare a list of questions for your doctor, or get a general sense of what a test result might mean. But before making any actual changes to your diet, medication, or treatment, verify the information. Ask the AI for its sources. Check those sources yourself. Talk to a real clinician, especially if the advice involves changing how you take a prescribed drug or adding a supplement to your routine. If an answer sounds too simple, too extreme, or too good to be true, it probably is. The paper also suggests a set of practical safeguards. Users should have access to records of their health-related chatbot conversations, so that if harm occurs, they can share exact transcripts with their doctors. There should be a system for reporting AI-generated health harm, similar to the way people report faulty medical devices or adverse drug reactions. Independent third parties should be able to monitor and evaluate the health advice these systems give, not just in controlled labs but in the messy reality of everyday use. Social media companies should clearly label AI-generated health content, so people know when they are looking at algorithmic output rather than professional expertise. Search engines should rely on medically vetted information in their AI summaries instead of pulling from random websites and forums. These aren’t radical demands. They are basic protections, the kind we already rely on in food safety, pharmaceutical regulation, and consumer rights. Without them, the next bad piece of advice may be impossible to trace until it has already caused real damage.
Of course, expecting AI companies to adopt these safeguards voluntarily is naïve. Yang said it plainly: “We do not think this is something we should rely on the companies to do, because their incentive is always to make more money. Building such a system goes against that incentive.” That is why the researchers believe policymakers need to step in. One of their most controversial proposals is extending physician malpractice liability to AI chatbot companies. If an AI gives medical advice and someone is hurt as a result, the company behind it should face legal consequences in the same way a doctor or hospital would. That may sound aggressive, but it’s how we treat other dangerous products in society. We don’t let pharmaceutical companies regulate their own safety data. We don’t let food manufacturers decide whether their ingredients are safe to consume. Yet with AI, we are allowing technology companies to offer personalized health advice to millions of people, and then we’re erasing the evidence when something goes wrong. The researchers argue that we need independent third-party monitoring systems, staffed by people who are not financially connected to the AI industry. Without that independence, any accountability mechanism is just theater. The goal isn’t to crush innovation or punish companies for trying to be helpful. It’s to make sure that a health question doesn’t turn into a life-altering mistake because of a confident hallucination. If a chatbot can tell you to replace salt with sodium bromide, or to stop taking your blood pressure medication, we need to know about it. We need to be able to investigate it, report it, and prevent it from happening to someone else.
The risk is broader than just the people who deliberately ask AI for health advice. In today’s internet, AI-generated content can reach you even when you’re not looking for it. Search engines are increasingly placing AI summaries at the top of their results. Social media platforms are embedding AI features into their apps. You could be scrolling through your feed for entertainment and stumble onto a video or a post that claims a miracle supplement cures arthritis, or that a particular vaccine is dangerous, or that a common medication should be avoided. This information may have been generated by an algorithm, but to the average eye it looks just as legitimate as anything else. Yang noted that even he encounters health information when he isn’t searching for it. “Sometimes I’m not even looking for health information,” he said, “but just by browsing my social media feeds, it’s there. It just shows up.” He worries that this could have undesirable outcomes, especially if there is medical misinformation or if state actors are trying to manipulate online discussions. The researchers argue that whether people actively seek health guidance or accidentally encounter it, harmful outcomes often leave little evidence that can be independently reviewed or verified. A social media post that convinces someone to stop taking their medication disappears into the algorithmic void. A chatbot answer that sends someone to the hospital vanishes from the conversation history. By the time the damage becomes visible, it may be impossible to trace back to its source. This is why the paper’s call for transparency is so urgent. The 60-year-old man recovered enough to be discharged, but his story could have ended much differently. As AI becomes an even larger part of our daily lives, we need to remember that chatbots are tools, not doctors. They can help us think, but they cannot think for us. And they certainly should not make decisions about our bodies behind a curtain of silence. The next person who asks a simple dietary tip deserves better.
