Why AI makes up facts and fake sources, and how to stop it

What is actually happening when AI invents a source

A language model completes text. Given "according to a 2021 study in the", the highest-probability continuations are real-sounding journal names, and the model produces one. No lookup happens. No database is consulted. The output is a citation-shaped string assembled from patterns, and its formatting is impeccable precisely because formatting is the part the model learned best.

Calling the behaviour "hallucination" makes it sound like a malfunction, which misleads people into thinking a better prompt or a newer model will eliminate it. A more useful description is unfounded generation: the model had no grounding for the claim and produced one anyway, because producing something is what the system does. Understanding the mechanism changes the fix. You stop asking the model to be honest and start removing the situations where invention is the only available move.

Two situations account for most fabrications in business use. The first is asking for evidence the model was never given, such as "cite three studies showing this". The second is asking about a specific document that was mentioned but not attached, such as "using our Q3 report". In both cases the model has a request it cannot satisfy and no mechanism for refusing, so it produces the expected shape.

The Avianca case, and why it keeps happening to careful people

In Mata v. Avianca, Inc., before the United States District Court for the Southern District of New York, two attorneys submitted a brief citing six judicial decisions that did not exist. Judge P. Kevin Castel imposed a $5,000 sanction in June 2023, noting that the lawyers continued to stand behind the citations after their authenticity was questioned. The fabricated opinions included quotations and internal citations, which is what made them convincing.

The instructive detail is not that the lawyers used AI. The instructive detail is that when challenged, they asked ChatGPT whether the cases were real, and it confirmed that they were. Asking a model to verify its own output is not verification: the confirmation is generated text produced by the same process that produced the citation, with no connection to any case reporter.

The pattern generalises well beyond law. Marketing teams cite statistics that trace to nothing, analysts quote reports that were never published, and students cite papers with real authors attached to invented titles. In each case the artifact looked correct, and the correctness of the format was mistaken for correctness of the content.

How often it happens, according to the research that measured it

Rates vary sharply by domain and by whether the model has retrieval. Work from the Stanford RegLab and Institute for Human-Centered AI — Dahl, Magesh, Suzgun and Ho, published in the Journal of Legal Analysis in 2024 — tested general-purpose models on legal queries and reported hallucination rates in the range of roughly 58% to 82% depending on the model and question type, with performance worst on questions about specific, lower-profile cases.

Two findings inside that work matter more than the headline number. Models were most likely to fabricate exactly where a user is least able to check, namely obscure matters rather than famous ones. And models frequently failed to recognise when a premise in the question was false, accepting and building on it instead of correcting it.

Treat any single published rate as domain-specific rather than universal. The honest summary for everyday work is narrower and more useful: fabrication rises as specificity rises, as subject obscurity rises, and as available grounding falls. A question about the boiling point of water is safe. A question about your competitor's 2026 pricing is not.

Grounding beats instruction: put the evidence inside the request

Telling a model to be accurate does not constrain it, because no draft announces its own inaccuracy. Supplying the document does constrain it, because the output can then be compared against something. Lewis and colleagues formalised the approach as retrieval-augmented generation at NeurIPS 2020 (arXiv:2005.11401), fetching relevant passages at query time and generating from those rather than from parameters alone.

In everyday tools the same principle has three practical forms. Paste the source text directly into the prompt, which works everywhere and is the most reliable option for a single document. Upload the file where the product supports it, then confirm the upload landed by asking the model to quote its first line verbatim before you ask anything else. Or use a tool with live retrieval and require that every claim carry the URL it came from.

Grounding narrows the job from "recall the world" to "read this and report". That is a task the technology is genuinely good at, and it converts an unverifiable claim into one you can check against a document sitting in front of you.

The UNKNOWN rule, which is the highest-value clause in the prompt

Models fabricate partly because nothing in the request permits an incomplete answer. Give the model a legitimate way to report absence and much of the invention stops. The clause is short:

Three parts are doing work here. Restricting the model to attached material closes the door on parametric memory. Inline attribution makes each claim auditable at a glance instead of requiring a second pass over the draft. And the closing gap list turns absences into a visible worklist, so a missing number becomes a research task rather than a silent invention.

The same structure adapts to citations. Requiring that every reference include a working URL the model was given, and that references not present in the supplied material be omitted entirely, removes the most common route to a fake bibliography.

Why asking "are you sure?" does not work

Pressing a model on whether it is certain produces a confidence statement, and confidence statements are generated the same way everything else is. The model has no separate faculty that inspects the world and reports back. In the Avianca matter the follow-up question produced an assurance that the fictional cases were genuine, which is the clearest documented example of the failure mode.

Worse, the question is unstable in both directions. Challenge a correct answer firmly enough and many models will retract it and substitute a wrong one, a capitulation pattern sometimes described as sycophancy. Huang and colleagues at Google DeepMind, in "Large Language Models Cannot Self-Correct Reasoning Yet" (ICLR 2024, arXiv:2310.01798), found that without external feedback, self-correction could turn correct answers into incorrect ones.

What does work is asking the model to do something checkable. "Quote the exact sentence from the attached document that supports this claim, and give its page number" either produces a quotation you can find or exposes that no such sentence exists. The request has an external referent, so the answer can be wrong in a way you can detect.

A verification pass that takes four minutes

Build the check into the workflow rather than relying on vigilance. For any draft carrying numbers or references, run this sequence before the work leaves your hands.

1. Extract every claim that carries a number, date, name or citation. Ask the model to list them in a table with the exact sentence and the source it attributes each to.
2. Open every URL. Not skim — open. A fabricated reference usually has a plausible DOI-shaped string and a dead or unrelated destination.
3. Match each figure to the source document by searching the document for the number itself, not for the claim around it.
4. Recompute any arithmetic in a calculator or spreadsheet. Language models handle multi-step arithmetic unreliably, and a confident total is not a verified one.
5. Flag anything that survived without a source and either cut it or research it properly.

Four minutes of this catches the failures that cost reputations. The A-Z AI Prompt Encyclopedia builds the same discipline into its structure: every one of the 1,200 workflows pairs a Create card with a Review card that asks for the exact passage, the evidence and the recommended correction rather than a general impression.

Prompts that reduce fabrication, and one that increases it

Three phrasings measurably shift behaviour in everyday use. Requiring inline attribution — "name the source and year inside the sentence that carries the figure" — makes unsupported claims visually obvious. Requiring quotations — "support each claim with a verbatim quotation from the attached material" — forces the model back into the document. Requiring an explicit gap list surfaces what is missing rather than papering over it.

The phrasing that reliably makes things worse is any instruction that rewards volume of evidence. "Cite at least five authoritative studies" tells a model with no retrieval to produce five citation-shaped strings, and it will. If the model cannot search, asking for more sources asks for more fiction.

A related trap is the false premise. Asking "why did the 2025 FTC ruling ban this practice?" invites the model to explain a ruling that may not exist, because the question presupposes it. Ask instead: "Does a 2025 FTC ruling on this practice exist in the attached material? If not, say so and stop."

Where the stakes justify more than a prompt fix

Some categories of work should not depend on prompt discipline alone. Legal filings, medical guidance, financial advice, safety instructions, regulatory submissions and anything creating a contractual obligation need a qualified human in the loop regardless of how well the prompt was written. The NIST AI Risk Management Framework (AI RMF 1.0, January 2023) treats this as a governance question rather than a tooling one, and that framing is the right one.

The practical implication for teams is a policy rather than a habit. Decide in advance which document types may never be published without a named reviewer, and record who reviewed what. The FTC's 2024 rule on fake reviews and testimonials is a reminder that fabricated content carries regulatory exposure and not merely embarrassment.

For marketing and content work, the proportionate rule is simpler: no number and no citation goes out without a human having opened the source. That single gate would have prevented every publicised AI citation failure of the last three years.

What this looks like in a finished prompt

Combining the moves gives a request that is hard to answer with invention:

The result is not guaranteed to be perfect, and the verification pass still runs. What changes is that the model no longer has a comfortable path to invention, and any fabrication that does occur is trivially detectable because the audit trail was specified into the output.

Telling a real citation from a fabricated one at a glance

Fabricated references share recognisable tells once you know them. The author names are usually real researchers who work in roughly the right field, because those names are frequent in training data. The journal is real and prestigious. The year is recent and plausible. The title, however, is generic and reads like a summary of your question rather than a paper title.

Digital identifiers are the fastest check. A DOI either resolves at doi.org or it does not, and a fabricated one typically does not. An arXiv identifier follows a strict format of four digits, a dot, and four or five digits; a made-up one often resolves to an unrelated paper, which is a stronger tell than a dead link because it looks like success.

The final tell is quotation. Ask for the exact sentence being relied on, with its page number. Real sources yield a quotation you can find with a text search. Fabricated ones yield a fluent paraphrase of your own question, which is the moment to delete the reference and do the research yourself.

A ninety-second verification order for any reference

Checking a reference is faster than most people expect once the order is fixed. Search the exact title in Google Scholar first; a genuine paper from 2020 onward will have citation counts and a publisher link attached to it, while an invented title returns nothing or returns something unrelated.

Then resolve the identifier. A DOI opens at doi.org and lands on the paper itself. An arXiv identifier opens at arxiv.org/abs/ followed by the number, and the title on that page must match what was claimed. For a court decision, search the case name together with its reporter citation on CourtListener or PACER. For an official statistic, go to the issuing body — the Bureau of Labor Statistics, Eurostat and the Office for National Statistics all publish their own figures alongside methodology notes, and the methodology note is where a weak number reveals itself.

Two traps deserve naming. A URL that loads is not a verified source, because a fabricated reference sometimes points at a real paper on an unrelated subject, and the page loading feels like confirmation. And a figure quoted on a vendor blog is not a primary source: trace it to the study, check the sample size and the year, and if no methodology was ever published, drop the figure rather than repeat it.

What to do when you have already published something fabricated

Act on the artifact before the explanation. Correct or remove the claim, note the correction visibly if the piece was published, and check whether the same figure was reused in other assets — fabricated statistics propagate through decks, one-pagers and social posts within days of being drafted.

Then trace the origin. In most cases the number entered the workflow in a prompt that requested evidence the model did not have, which tells you which prompt to fix rather than which person to blame. Add the UNKNOWN rule and the attachment requirement to that prompt and the specific failure does not recur.

Then write the check into the process. A one-line rule in a style guide — no figure ships unless a named person opened its source — outlasts any resolution to be careful, and it survives staff turnover in a way that individual vigilance does not.

Scale the response to the exposure. A fabricated statistic in a blog post needs a correction note and a scan of the assets that reused it. A fabricated figure in an investor update, a regulatory filing, a court submission or a medical communication needs disclosure to the recipient, because the downstream reliance is the harm rather than the error itself. The FTC's 2024 rule on consumer reviews and testimonials is a useful reminder that invented supporting material carries regulatory exposure and not only reputational cost.

Keep a short record of what was corrected and when. If the same number surfaces again six months later in a deck someone forked, the record is what stops it being republished as fact. Teams that log corrections in the same place they log content briefs catch repeats far more often than teams relying on memory.