ChatGPT: Can you trust its advice about diet & health

We asked ChatGPT what foods can lower the risk of six health problems--arthritis pain, cataracts, hair loss, and breast, pancreatic, or prostate cancer. Here's what we found...and how to get better answers to your questions.

1. Ask for ChatGPT's evidence

Unlike Google AI's overviews, which often grab advice about diet and health from random internet sites (like a company that sells kitchen wares or a website for learning Spanish), ChatGPT makes it easier to drill down to a scientific paper or other source of its advice.

Let's say you ask ChatGPT a question like "What foods can lower the risk of prostate cancer?" You can follow up by asking "What's the evidence for your advice?" (ChatGPT often starts its reply with "Great question." Gee, thanks!).

The chatbot then supplies a list of studies that may look impressive. But there's no guarantee that those studies are the most trustworthy.

2. Check the sources

Click on every link to evidence. Don't be surprised if you find mistakes.

For example, some links go to studies that contradict ChatGPT's advice (oops!) or to animal or test-tube studies (not the strongest evidence). We also found broken links or studies on the wrong topic.

If ChatGPT cites a health authority (like the American Cancer Society), check the link to make sure the advice is really there.

But even when ChatGPT accurately cites studies, you can't be sure that they offer the best evidence. For example, it seems to love meta-analyses and reviews (which typically combine the results of dozens of studies). But those overall results sometimes gloss over weaknesses in the original studies.

Meanwhile, ChatGPT sometimes misses top-notch evidence from gold-standard randomized clinical trials that fail to support its advice.

3. Remember: ChatGPT aims to please

When it comes to questions like "What foods can lower the risk of [insert illness]?" ChatGPT rarely seems to say "Sorry, foods can't help."

For example, when we asked, "What foods can prevent Covid?" the chatbot said, "No food can prevent Covid-19 on its own, but a healthy diet can support your immune system, which may help your body better fight infections, including Covid-19."

Do they, though? ChatGPT's list of "Immune-Boosting Nutrients & Food Sources" starts with vitamins C and D. Yet in clinical trials, neither has helped speed recovery from Covid.

4. Ask for contradictory evidence

When we asked "What evidence contradicts your advice?" about pancreatic cancer, ChatGPT hedged. "Here's where the evidence becomes inconclusive, limited, or contradictory," it replied.

Another tip: Repeat the same questions to see if you get somewhat different answers.

For example, when we repeated the question "Which foods can lower the risk of pancreatic cancer?" several times, mushrooms, onions, and garlic made the list for a few of our queries and didn't make it for a few others.

"ChatGPT can make mistakes," the chatbot often notes. Indeed.

These are the responses we got to several queries.

What foods can lower the risk of recurring breast cancer?

Foods may help, said ChatGPT.

Fruits & vegetables (especially cruciferous). Their antioxidants, carotenoids, and indoles can help, we were told.

But when we asked for evidence, ChatGPT noted that a 2013 study that pooled data on 11,390 women "found no significant link" between cruciferous vegetables and cancer recurrence.

And the WHEL trial, which randomly assigned roughly half of 3,088 breast cancer survivors to eat more fruits and vegetables for seven years, "found no benefit" for recurrence, noted ChatGPT. Hello?

Omega-3 fats. Fatty fish's "anti-inflammatory omega-3s (EPA and DHA) may reduce cancer-promoting inflammation," said ChatGPT.

Its evidence? A meta-analysis that didn't look at cancer recurrence. Maybe that's why the chatbot added that "evidence in recurrence is not as strong as in primary prevention but promising."

Promising? Note to ChatGPT: Fish oil had no impact on breast cancer risk in the largest trial done.

Legumes & whole grains. "High fiber intake has been associated with reduced breast cancer mortality," asserted ChatGPT.

It cited a 2012 "key meta-analysis" that reported that women who ate the most fiber had a 7 percent lower risk of being diagnosed with breast cancer than those who ate the least. (So much for recurrence or "mortality.")

What ChatGPT missed: The WHEL trial, which encouraged breast cancer patients to consume 30 grams of fiber a day for seven years, came up empty.

Bottom line: ChatGPT also recommended green tea, turmeric, garlic, onions, nuts, and seeds, but the evidence that any food can lower the risk of breast cancer recurring is limited.

After we probed, the chatbot's "Final Thoughts" changed its tune:

"The strongest evidence for diet and breast cancer recurrence comes from cohort and observational studies...but results are inconsistent and sometimes small. Large randomized trials like WHEL didn't show benefit, highlighting how hard it is to prove causation in nutrition." Coulda said that earlier...

What foods can lower the risk of osteoarthritis pain?

Try these, suggested ChatGPT:

Fatty fish. They're "rich in omega-3 fatty acids, which reduce joint inflammation and stiffness."

The chatbot cited a 2021 meta-analysis of six clinical trials on 454 people. Unfortunately, differences between the six studies and other limitations "indicate low-quality evidence and do not provide a basis for clinical guidelines," said the meta-analysis's authors.

Worse yet, ChatGPT missed the VITAL trial, which gave 1,398 people with knee pain fish oil (840 mg of EPA plus DHA) or a placebo every day for five years. Pain scores? No different.

And when we scrolled down far enough, ChatGPT told us that "omega-3 and vitamin E showed no significant changes" in a 2022 meta-analysis. So much for reducing joint inflammation and stiffness.

Leafy greens. "A study in American Journal of Medicine found that low vitamin K levels were associated with more severe OA" (osteoarthritis), said ChatGPT. (Wrong journal, but whatever.)

Too bad the chatbot missed a clinical trial that gave 378 people with hand osteoarthritis a daily multivitamin with or without vitamin K (which is abundant in leafy greens). After three years, "there was no overall effect of vitamin K," concluded the authors.

Low-fat dairy. "Vitamin D is crucial for bone and joint health," said ChatGPT. "Low levels are linked with increased OA pain and progression."

Umm... Remember the VITAL trial on 1,398 people? After five years, those who were randomly assigned to take vitamin D (2,000 IU a day) had no less knee pain than those who got a placebo.

Bottom line: ChatGPT also recommended turmeric, citrus fruits, berries, nuts, seeds, olive oil, whole grains, water, and green tea, but so far, there's no solid evidence that any food curbs osteoarthritis pain.

What foods can lower the risk of cataracts?

Foods rich in these components may matter, said ChatGPT:

Lutein & zeaxanthin. "These carotenoids accumulate in the lens and retina and filter harmful blue light," it declared. But in the AREDS2 trial-in people with intermediate or advanced age-related macular degeneration (AMD)-those who got lutein and zeaxanthin supplements had no lower risk of cataract surgery.

Among people in the trial with the lowest intakes of lutein and zeaxanthin from foods (like leafy greens), those who got the supplements were 32 percent less likely to need cataract surgery. But researchers would need a new trial on people with low intakes to know if lutein and zeaxanthin matter.

Vitamin C. In studies that track people for years, those who eat vitamin C-rich foods (like broccoli, berries, and citrus fruit) have a lower risk of cataracts, noted ChatGPT.

But vitamin C failed to prevent cataracts in clinical trials, suggesting that something else about fruit-and-vegetable eaters curbs their risk.

Vitamin E. It's a "powerful antioxidant" in foods like almonds and avocados that's "associated with a lower risk of cataracts," said ChatGPT. But in clinical trials, vitamin E hasn't helped.

Omega-3 fats. They're "linked to lower incidence of nuclear cataracts, though evidence is more robust for age-related macular degeneration than cataracts," said the AI tool.

More robust? It's more like a double bust. In the AREDS2 trial, omega-3 fats had no impact on macular degeneration or the risk of cataract surgery.

Zinc. "While primarily focused on AMD, the protective role of zinc in oxidative stress supports its inclusion in cataract prevention," said ChatGPT.

Translation: We've got no good evidence that zinc protects against cataracts.

Bottom line: To help prevent cataracts, wear sunglasses with UV protection year-round, don't smoke, and get some lutein from foods like leafy greens, peas, broccoli, and yellow corn.

What foods can lower the risk of  pancreatic cancer?

These could do the trick, we were told.

Fruits & vegetables. They topped the list. But when asked for studies, ChatGPT said that only "cruciferous vegetables, not total produce, were inversely associated with pancreatic cancer" in a 2006 Swedish study.

But, as it turns out, the link with cruciferous vegetables wasn't statistically significant either. Oops.

ChatGPT also cited two meta-analyses to back up its advice on cruciferous vegetables. But the links were only significant in "case-control" studies, which compare what people with cancer (cases) say they typically ate before their diagnoses to what people without cancer (controls) say they typically ate.

That's weaker evidence because of "recall bias"-that is, a diagnosis may alter what people remember eating.

In a stronger 2012 meta-analysis, researchers pooled data on 862,584 people from 14 studies that asked people what they ate and then waited years to see who got cancer.

"Fruit and vegetable intake...is not associated with a reduced pancreatic cancer risk," concluded the scientists.

Fatty fish. Omega-3 fats curbed the growth of pancreatic cells in test tubes and in mice in a 2011 study, said ChatGPT. Evidence in humans? Scant.

Bottom line: ChatGPT also recommended whole grains, beans, nuts, and green tea, but there's little evidence that any food can lower pancreatic cancer risk.

The chatbot did recommend limiting red and processed meats to reduce the risk of pancreatic cancer, citing the International Agency for Research on Cancer and the American Institute for Cancer Research. At least that advice comes from human experts who can weigh the strength of the evidence!

What foods can lower the risk that  early-stage prostate cancer gets worse?

Here's some of ChatGPT's advice:

Tomatoes. They're rich in lycopene, "a powerful antioxidant," noted the chatbot. "Some small trials showed that lycopene supplements or tomato products might reduce PSA levels."

(Elevated PSA, or prostate-specific antigen, is a marker of prostate cancer.)

Hmm. In the trial that we think ChatGPT meant to cite-we're not sure because the link was broken-PSA levels were not significantly lower in lycopene takers.

Cruciferous vegetables. They "contain sulforaphane, which may inhibit cancer cell growth," said ChatGPT.

"May" is right. "This study does not provide compelling evidence of a protective influence of cruciferous vegetables on prostate cancer risk," said the study that we think ChatGPT meant to cite. (Bad link again.)

Soy foods. They "contain isoflavones (e.g., genistein), which may have anti-androgenic effects," explained ChatGPT.

We gave up trying to find the studies that it meant to cite.

But we did find a 2021 review of randomized controlled trials on isoflavones. They "seem to have no influence on PSA levels in localized prostate cancer," it concluded.

Bottom line: ChatGPT also recommended green tea, pomegranates, legumes, berries, and fatty fish, despite weak evidence.

What's more, ChatGPT ignored the MEAL trial, which randomly assigned 443 men with early-stage prostate cancer to a control group or to eat at least seven daily servings of vegetables (especially cruciferous, leafy green, or carotenoid-rich ones). After two years, the veggies had no impact on cancer progression.

Veggies can still help protect your heart and waist. Prostate? Not clear.

What foods can lower the risk of hair loss?

We got some interesting answers on this one.

Protein. "Hair is primarily made of a protein called keratin, so a protein-rich diet is essential," explained Chat-GPT. According to its source, "protein malnutrition, such as in kwashiorkor and marasmus, can result in hair changes that include hair thinning and hair loss."

No doubt. But what if you're not severely malnourished?

"It is unclear what role, if any, is played by amino acid and protein supplementation in the absence of known deficiency," said the source.

Zinc. "Lower zinc levels were found in patients with alopecia areata," noted ChatGPT. "Supplementation showed improvement in some cases."

Alopecia areata is an autoimmune disease in which the immune system mistakenly attacks hair follicles, causing inflammation and hair loss.

We couldn't find the cited study on zinc. (ChatGPT's link went to a rat study on selenium nanoparticles.) But the only double-blind trial testing zinc on alopecia areata came up empty.

Vitamin A. "A 2019 review noted both deficiency and excess can harm hair," cautioned ChatGPT.

Its link sent us to a retracted 2019 study on osteosarcoma cells, but we managed to find the right one. The review's advice: "As a general rule, consuming too much or over-supplementing vitamin A can cause hair loss." In other words, vitamin A is not good for preventing hair loss.

Biotin. It "only helps those with a diagnosed deficiency (which is rare)," said ChatGPT.

Correct! So why list eggs as a biotin-rich food that can help prevent hair loss?

Bottom line: ChatGPT also recommended foods rich in selenium, omega-3 fats, and vitamin C, but don't count on any food to prevent or treat hair loss. Ditto for the chatbot's "Bonus Tips" like stay hydrated, limit processed foods and sugar, and check for iron, vitamin D, zinc, and B-12 deficiencies.

How about evidence deficiencies?

Tags

• Cancer
• Arthritis
• Supplements

Topics

• Preventing Disease
• Fact vs. Fiction

More on food & health

Nutrition Action