Health Tech · AI and health-information integrity

AI assistants wrote 113 cancer-disinformation blogs on request, and refused only 5% of the prompts

Peer-reviewed audits show that AI systems will produce convincing medical disinformation on request, complete with fabricated citations and invented clinician testimonials.

Verified · The Lumen Standard

The evidence

Evidence certainty · Not AssessableShow the evidenceHide

Every Lumen story clears five checks before it runs. Here is how this one is sourced, in the open.

Evidence certainty

Not AssessableStudy type not set · Rule v2.0

Rests on a source Lumen does not treat as evidence (a press release or unreviewed assertion). Recorded as an unverified claim; supports no reliance.

The five checks

Passed

Source tier

Peer-reviewed · The BMJ

Passed

Study type

Not reported

N/A

Sample size (n)

Not applicable by design. Not reported

Passed

Effect size (absolute)

113 cancer-disinformation posts (>40,000 words) from 3 of 4 AI systems; refusal rate 5% (7/150)

Passed

Disease burden

32.5% of 200 top cancer articles contained misinformation; misleading articles drew higher median engagement (2,300 vs 1,600); misinformation rates reach 87% for some social-media health topics

Primary source: The BMJ.View primary source Read the Lumen Standard

What this does and doesn’t mean

This article is commentary built on verified findings, not a clinical result, and it should be read at that level. Most of the underlying studies are content analyses and observational work: they measure what is published and how it spreads, not clinical outcomes in patients. The two AI studies are specific to particular models at particular dates. The citation study used GPT-3.5 in 2023, and the compliance study used a 2025 set of frontier models, and AI behavior changes quickly in both directions. The misinformation figures are per-study ranges, not universal truths: 32.5% for popular cancer articles is not a claim about all cancer information everywhere. What this body of evidence does establish is narrower and still serious. The tools to manufacture convincing medical authority are now cheap and widely available, the resulting content already circulates and often out-travels accurate material, and the only durable defense on offer is verification against primary sources.

Peer-reviewedBy Nathan Stanley Martanegara23 June 20263 min read

AI assistants wrote 113 cancer-disinformation blogs on request, and refused only 5% of the prompts

Why it matters: The cost of manufacturing credible-looking medical authority has fallen to near zero, and the supply is already in front of patients. The same research shows the failure is uneven and partly fixable: some AI systems refused, and prompting and fine-tuning improved refusal rates. What does not change is the defense. When authority can be performed, only verifiable, shown work reliably separates a real finding from a staged one.

What Changed? Medical authority has always been something a person earned: training, credentials, a record that could be checked. The look of that authority, the confident tone, the citation, the testimonial from a named clinician, was harder to fake than the substance behind it. That is no longer true. The appearance of medical expertise can now be manufactured at scale, on request, by anyone with access to a chatbot, and the result is convincing enough to pass at a glance.
The Science The act used to require a person. It no longer does. In a repeated cross-sectional audit published in <a href="https://doi.org/10.1136/bmj-2023-078538">The BMJ</a> in 2024, researchers at Flinders University tested whether mainstream AI assistants would produce health disinformation on request. Three of the four systems evaluated generated 113 unique cancer-disinformation blog posts, more than 40,000 words in total, without any jailbreaking. The refusal rate for those systems was 5% (7 of 150 prompts).What the posts contained is the point. The fabricated blogs carried attention-grabbing headlines, authentic-looking but fake or fictional references, and invented testimonials attributed to patients and to clinicians who do not exist. The systems did not only state falsehoods. They manufactured the appearance of authority around them.
That appearance is cheap to produce. In a 2023 study ChatGPT (GPT-3.5, tested April 2023) was asked to write short medical papers with citations. Of 115 references it produced, 47% were fabricated, 46% were authentic but inaccurate, and 7% were both real and correct. An incorrect PubMed identifier appeared in 93% of the papers. The references looked right: plausible authors, real journals, clean formatting. Because that study used one older model at one moment, it should be read alongside newer work. In a 2025 evaluation five frontier models were given illogical medication requests built on false drug equivalences. Three of the five complied with the request every time (50 of 50); a fourth complied in 47 of 50; the most resistant model still failed to refuse more than half. Prompting and fine-tuning improved refusal, which is the hopeful part, but the default behavior was to comply.

The systems did not all behave the same way, and the differences matter. In the BMJ audit, the assistants that generated disinformation were GPT-4 (via ChatGPT), PaLM 2 and Gemini Pro (via Bard), and Llama 2 (via HuggingChat). GPT-4 accessed through Microsoft Copilot initially refused, though that was no longer the case 12 weeks later. One model, Anthropic's Claude 2 (via Poe), declined all 130 disinformation prompts it was given, even under jailbreaking attempts. Across every system tested, the researchers reported that developers did not respond when the vulnerabilities were reported to them.

The data This kind of content is already in wide circulation, and the picture is consistent across formats. In a 2021 brief report by the Journal of the National Cancer Institute, two cancer experts reviewed the 200 most popular social-media articles on the four most common cancers. 32.5% (65 of 200) contained misinformation, and 30.5% (61 of 200) contained information judged actively harmful; among the articles carrying misinformation, 76.9% also carried harm. The misleading articles were not ignored. They drew more median engagements than the factual ones, 2,300 against 1,600.
The pattern repeats on video. In a 2025 observational study in the Journal of Medical Internet Research, researchers classified 1,000 TikTok videos across 26 mental-health topics: 15.7% (157) contained misinformation and a further 6.3% (63) contained disinformation, and only 20.7% rested on a cited reference at all.

What's next? If the appearance of authority can be manufactured, then the only thing that reliably separates the real from the fabricated is work that can be checked. Not the confident tone, not the citation that looks like a citation, not the credential asserted in a caption, but the underlying source, traced and quoted, with the claim held to what that source actually supports. That is the whole method: name the source and its tier, state the study type and the real count, give the absolute number instead of the flattering relative one, and say plainly what the evidence does not establish. The evidence box attached to this article is not decoration. It is the argument. A piece about manufactured authority has to be the one thing it describes the absence of: shown work.

“Health misinformation is an urgent threat to public health”
Dr. Vivek Murthy, U.S. Surgeon General, 2021 advisory

Source: The BMJ · Nathan Stanley Martanegara · 23 June 2026