How do you catch AI hallucinations before they reach a client?

Catching AI hallucinations in a draft means separating three defects that need three different checks. The source may not exist at all. The source may exist while the number or quotation attached to it appears nowhere inside it. Or — the case most checklists skip — the source may exist, be quoted accurately, and still not support the sentence it is attached to. This page is the pass that finds all three and ends with a publish-or-return decision on every claim.

When should you use this — and when should you not?

Run this pass on anything AI helped write that carries a number, date, currency figure, named source, direct quotation or link, and that will reach a client, a publication or a decision.

The failure is not confined to careless amateurs. Springer Nature retracted the book Mastering Machine Learning in 2025; its retraction notice states the publisher could not verify the source of 25 of the book's 46 references (Retraction Watch). An audit of nearly 2.5 million biomedical papers found that about one paper in 277 published in the first seven weeks of 2026 cited a reference that does not exist, against one in 2,828 in 2023 — as reported by Retraction Watch and The Scientist, though Cochrane's Ella Flemyng told Retraction Watch the analysis lacked methodological detail. Peer review is the most heavily checked environment there is, and that is what gets through it.

Do not run the full pass when:

• The draft has no checkable objects. Opinion and narrative need an editorial read instead — see edit-ai-draft.
• The text is an internal working artefact. A sketch for yourself is not worth the cost.
• Every figure came from your own data. If AI only rephrased your CRM export, the risk is transfer error, not invention: check against the export.
• The domain is regulated. In medicine, finance or law this pass is the floor, not the ceiling, and does not replace a qualified reviewer.
• You must decide now and cannot check. Cut every unverified number and make the point without it. A thinner checked argument beats a fast check.

This is one component of a wider release gate; the surrounding steps live in ai-draft-pre-publish-check and, for regulated marketing claims, ai-content-pre-publish-compliance.

What do you need before you start?

The draft has to be editable so you can mark claims in place; a flattened PDF forces you to hold state in your head, which is where coverage silently drops.

Keep the chat transcript if you have one. Which assistant wrote the draft, and whether web search or a deep-research mode was on, changes the defect profile. Across two benchmarks covering more than 200,000 cited URLs, 3–13% had no record in the Wayback Machine — the authors' criterion for "probably never existed" — and 5–18% failed to resolve at all; deep-research agents cited more and fabricated URLs more often than search-augmented models (Rao, Wong and Callison-Burch, April 2026, preprint, not peer-reviewed).

Beyond that you need a browser and two habits: exact-phrase search in quotation marks, and find-on-page. Optional and free: doi.org to resolve a DOI, web.archive.org to tell a deleted page from one that never existed, and the Retraction Watch database, which Crossref acquired in 2023 and made publicly available, updated every working day as of 2026-07-27 (Crossref documentation).

No paid tool is required, and none of them does the hardest part of this job.

How do you do it, step by step?

Plan for this to take real time. We have found no measurement of how long the pass runs or how much it catches for a first-time user, so treat any duration — including your own first run — as a working assumption to calibrate, not a standard.

Step 1. Build the surface list. Mark every checkable object in the draft: number, percentage, currency figure, date, named study, organization cited as a source, direct quotation, link. Count them. That count is your denominator for everything below.

This step exists because fabrication is scattered, not clustered. An audit of about 2.5 million papers and 111 million references across arXiv, bioRxiv, SSRN and PubMed Central, with data through August 2025, found the growth is in papers where 0–10% of references are fabricated, not in papers where half of them are; the share of papers with at least one fabricated reference rose by 2–23 percentage points depending on the corpus (Zhao et al., preprint, not peer-reviewed). Checking three references out of twenty is therefore a procedure that reliably returns "clean".

An assistant can build the list for you, as long as it never gets to judge anything:

You are a fact-checking triage assistant. Do NOT verify anything.
Do NOT add, remove, or rewrite any content.

From the draft below, extract EVERY verifiable claim into a table.
A verifiable claim = any number, percentage, currency figure, date,
named study, named organization cited as a source, direct quotation,
or URL.

Output a markdown table, one row per claim, with columns:
| # | claim_text_verbatim | type (number/quote/source/url/date) |
| named_source_as_written | risk (high/med/low) |

risk = high if the claim carries the argument OR attributes a specific
figure to a specific organization.

If a claim has NO named source in the draft, write "NONE" in
named_source_as_written. Do not guess a source.

DRAFT:
[PASTE DRAFT]

Step 2. Does the source exist? Paste any DOI into doi.org and check it resolves to that exact work; with no DOI, search the exact title in quotation marks. Open every URL, and when one fails, look for it in the Wayback Machine: a snapshot means the page existed and later moved or died, no snapshot at all means it probably never existed. For a company or agency report, go to that organization's own site — a summary of a report proves nothing about the report.

Step 3. Is the number actually inside the source? Open the source itself and find-on-page the figure, including its variants (56.9, 56,9, 57%); for a quotation, search a five-to-seven-word fragment. Absent figure: flag. Present but attached to a different denominator, population or period: also flag — that is the common version of this defect.

If the draft names no source, work backwards: search the number as an exact phrase, then climb. Articles citing each other are circular reporting, not confirmation. A number counts as confirmed only when you reach a document reporting how it was measured — sample size, period, who ran it.

For each row below, generate search strings I will run MYSELF.
Do NOT search. Do NOT answer whether the claim is true.
Do NOT invent sources.

For each claim produce exactly:
1. exact_phrase_query: the number or quote in quotation marks, plus
   2-3 disambiguating keywords
2. origin_query: a query aimed at the ORIGINAL measurement
   (e.g. "<topic> survey methodology <year> sample size")
3. what_would_confirm: one sentence naming the document and the
   section that would count as confirmation (must mention sample
   size or method)

Output as a markdown table. No commentary.

CLAIMS:
[PASTE ROWS FROM STEP 1]

Step 4. Does the source actually say this? This is the step that decides whether the pass was worth running. For every claim that survived steps 2 and 3, read the paragraph around the figure and the limitations or methods section, and look for five substitutions:

Substitution	What the source says	What the draft says
Dropped qualifier	"among respondents who already use the tool"	applies to everyone
Reversed direction	"associated with"	"leads to"
Swapped population or period	one country, one year	a global, current figure
Wrong document type	a press release or vendor landing page	"a study found"
Stale cut	true on an earlier date	presented as current

The evidence that this is the dominant defect is unusually consistent. In CITETRACE — 761,495 citation-source pairs from ten models across five providers, collected in a fifteen-day window in March–April 2026 — 30.6% of citations misrepresented the source they pointed to and 27.1% cited a source from a domain unsuited to the question, with up to 96% of answers containing at least one structurally misleading citation, all while the cited sources were real and retrievable (CITETRACE, preprint, not peer-reviewed). Three limits travel with those numbers: the judging was done by a model, only 58.3% of the unique cited URLs could be downloaded so the authors call their figures a conservative lower bound, and the questions came from English-language Stack Exchange, so transfer to marketing or editorial drafts is not demonstrated.

A separate study covering more than 3,000 assistant responses, run by 22 public-service media organizations in 18 countries and published in October 2025, found 31% of responses had serious sourcing problems — missing, misleading or incorrect attributions (EBU); it was commissioned by broadcasters, an interested party in the dispute with AI assistants over news. The authors of the largest citation audit say the same thing about their own blind spot: real references attached to claims the cited work never made are "more prevalent and harder to detect", and detection methods for it "remain under active development" (Zhao et al.). SMU Libraries put the rule in one line: "a verified citation proves a paper exists. It does not prove the paper supports the sentence it sits in" (SMU Libraries).

Step 5 (academic sources only, detachable). Search each surviving scholarly work by DOI or title in the Retraction Watch database. Skip this step for a piece citing only industry reports and news.

Step 6. Decide and rewrite. One rule: a flag means the number leaves the text — not softened, not hedged with "some reports suggest". Either you found the original and rewrote the sentence to match its real wording, qualifier, population and period, or the number is gone and the sentence makes its point without it.

Rewrite the marked sentences using ONLY the verified facts below.

RULES:
- Use the exact figure, population, time period and qualifier from
  verified_statement. Do not round. Do not generalize.
- If a sentence's supporting claim is marked REMOVED, rewrite the
  sentence so it makes its point WITHOUT any number. Do not
  substitute a different number.
- Do not add any new claim, number, source or example.
- Preserve the author's voice and paragraph structure.

VERIFIED FACTS:
[claim_id | verified_statement | source_url | population | period]

REMOVED CLAIMS:
[claim_id | original_text]

DRAFT SECTION:
[PASTE]

What does a good result look like?

Two artefacts come out: a rewritten draft, and a log that makes the check auditable.

surface	checked	no source	number not in source	source does not support	removed
18	18	2	3	4	6

Read that row as a normal outcome, not a disaster: nine of eighteen claims had a defect, six could not be repaired and were cut, three were rewritten to the original's actual wording.

Per claim, a finished row reads like this. Say the draft credits a 2025 report from an analyst firm with the figure that 68% of marketing teams use generative AI for campaign copy. (Hypothetical — attaching an invented number to a real firm is the behaviour this page exists to catch.) The row records three findings: the report resolves; 68% appears inside it; the report attributes it to survey respondents at companies above a revenue threshold, in one region, in the first half of 2025. The sentence is rewritten to carry that population and period, with the direct URL beside it. Had any of the three come back empty, the row would read removed.

How do you know the output is good?

• Checked count equals surface count — every claim on the surface list was checked, not a sample.
• Every number left in the text has a direct URL to a document where find-on-page locates that number.
• Every surviving source has the population and time period from the original recorded next to the claim.
• No claim is both unverified and still in the text.
• A check log records six counts: surface, checked, no-source, number-not-in-source, source-does-not-support, removed.

Each criterion is checkable by someone who did not do the work. That is the design: "I checked it" is not evidence.

Coverage is arithmetic — the number of rows you checked against the number of rows on the surface list. This is the criterion people negotiate with, and the one the evidence defends most directly: because fabrication concentrates in drafts where a small minority of references are wrong, a clean sample tells you almost nothing about the rest (Zhao et al.).

The link test is mechanical: open what sits next to a number, run find-on-page, see the number. A link to an article that merely mentions the study is not a pass. The recorded population and period are what prove step 4 happened at all — without them, you checked existence and stopped.

The last criterion governs the others. A pass that ends in "I could not confirm this, but it reads plausibly" leaves the weakest claims standing exactly where the evidence is thinnest. That is why step 6 has one rule and no softening option.

Where does this usually break?

• Spot-checking. Three of twenty references check out and the draft is declared clean — a conclusion the shape of the distribution does not support (Zhao et al.).
• Stopping at "the paper exists". This catches the easiest defect and misses both harder ones.
• Treating repetition as confirmation. Five blogs carrying the same figure may be five retellings of one press release. Citation chains broke this way long before AI: Rekdal followed the standard explanation that a misplaced decimal point created the "spinach is rich in iron" myth back through decades of citations to one undocumented sentence in a 1981 BMJ article, and found no evidence the decimal error ever occurred — the explanation itself had spread as an academic urban legend (Social Studies of Science, 2014).
• Asking the same model to check itself. It will confirm, confidently. Researchers including three from OpenAI argue that hallucination is a statistical consequence of training and evaluation that reward confident guessing over admitting uncertainty (Kalai et al., preprint) — an argument advanced partly by a model vendor, but with a clear practical reading: the model generates search strings, a human opening the source decides.
• Confusing a dead link with an invented one. Unresolvable URLs outnumber fabricated ones in measurement, 5–18% against 3–13% (Rao et al.). The Wayback Machine separates them; skipping that manufactures false alarms and burns the credibility of the whole pass.
• Trusting a vendor checker's accuracy figure. One vendor advertised 99% accuracy for its detector as of 2026-07-27 (SwanRef); the page carries no methodology behind that figure, and we found no independent replication of it. Paperpile — which sells a citation checker — documented a false positive from its own tool (Paperpile). A checker is a first-pass filter, not a verdict.
• Checking after publication. A Deloitte report for an Australian government department, contracted at AU$440,000, was found to contain more than a dozen false references and a fabricated quotation from a Federal Court judgment — real court, real decision, invented words — after an outside academic examined it; the firm repaid the final instalment (Large Language Law, summarising AFR and AP reporting, October 2025). The alternative to an hour of checking is being priced by whoever finds the error first.

What else do people ask?

Does web search or deep research mode fix this?

No, and it changes which defect you get rather than removing it. Retrieval makes the cited sources real and openable — which is precisely the condition under which CITETRACE measured 30.6% of citations misrepresenting their source (CITETRACE), and under which deep-research agents fabricated URLs more often than search-augmented models (Rao et al.). Retrieval moves the work from step 2 to step 4.

Does this apply to text a human wrote?

Yes. The defect is a property of unchecked citation chains, not of AI — the spinach case ran on human authors for decades (Rekdal). AI made it cheaper and faster to produce, which is why the pass now has to be routine rather than occasional. Once the facts hold, the remaining work is voice and structure: see edit-ai-draft and score-ai-output, and for research write-ups specifically, customer-research-synthesis. The full tracks are ai-for-marketers and ai-for-writers.