The AI reported a number that's wrong. How do I tell if it hallucinated or if the data feeding it was bad?
Re-run the same request twice. A hallucination usually changes between runs; a bad join or stale sync produces the identical wrong number every time. Then check whether the number appears anywhere in the source rows. If it does, the model read real data and the pipeline is wrong. If it appears nowhere, the model invented it. Those two failures have completely different fixes.
Two failures that look identical on a dashboard
An owner sees a brief saying last month's booked revenue was some figure that doesn't match the field service system. That single symptom has two very different causes, and treating one as the other wastes weeks.
A hallucination means the model generated a plausible value that was never in its input. A pipeline error means the model faithfully summarized data that was already wrong: a broken join, a timezone boundary, a filter that dropped voided invoices, a sync that ran before the day closed.
The three-step diagnostic
Run these in order. Most cases resolve at step two.
- Re-run it. Same question, same period, same configuration. Generated fabrications tend to vary; a deterministic query bug reproduces exactly. Identical wrong answers point at the data.
- Search the source for the number. If the figure exists somewhere in the retrieved rows — a subtotal, a different period, another branch — the model found something real and applied it wrongly. That is a retrieval or scoping problem, not invention.
- Recompute by hand from the same extract. Not from the source system, from the exact extract the model saw. If your hand calculation matches the model, the extract is the problem and the model is fine.
What each answer tells you to fix
If the number is genuinely invented, the fix is structural: stop asking a language model to do arithmetic over raw text. Numbers should be computed in the data layer and handed to the model as facts to narrate. A model that writes the sentence around a pre-computed figure has nothing to invent. This is the core discipline behind automated intelligence briefs that survive scrutiny.
If the extract was wrong, the fix is in the sync and the join — usually definitional. Which date field, which record states, which locations. That work looks like data integration, not AI work, and it is where most of the real effort lives.
A tell worth knowing
Fabricated numbers are usually suspiciously round or suspiciously precise, and they are internally consistent — the fake total matches the fake parts, because the model generated them together. Pipeline errors are the opposite: the total and the components disagree, because they came from different broken paths.
If the components don't reconcile to the total, stop looking for a hallucination and go read the query.
Preventing the ambiguity in the first place
Every generated statement that contains a number should carry a reference back to the rows that produced it. Not a citation in the academic sense — a record identifier, a filter definition, a row count. When someone questions a figure, you should be able to open the underlying set in seconds.
Systems built this way rarely have hallucination arguments, because the argument resolves itself. See how we structure AI business intelligence so every claim has a traceable source.
Topics: hallucination · diagnostics · data quality · debugging
Have a version of this question about your own business?
The useful answer usually depends on which systems you run and how they're connected. That's a conversation, not a blog post.