What is an AI hallucination, and how is it controlled?

An AI hallucination is output from an AI system that reads as fact and is not true: a date that appears in no document, a policy number that was never issued, a confident summary of a clause the contract does not contain. The UK government's AI Playbook describes it as content that appears plausible but may actually be factually incorrect. It cannot be cured, but it can be controlled, by making the system work from your records and not from memory, checking its output with ordinary rules, and putting a person in front of anything that matters.

Why does AI make things up?

The Playbook's explanation is that a model returns the most likely output for what it is given, from patterns in the data it was trained on, without an actual understanding of the content. Most of the time the likely answer is the right one. When the model has nothing to go on, the likely answer is still fluent and still confident, and the model does not flag the difference, because to the model there is none.

So a hallucination is neither a lie nor, strictly, a fault. The system has done what it does, in a place where it had no facts to work from.

Why does it matter more when AI acts?

A hallucination in a draft is caught, or not, by the person who reads the draft. A hallucination inside an AI agent can be acted on. The Information Commission's Office, in its report on agentic AI, says such systems increase the chance of "cascading hallucinations". It reports the view of the people it consulted that a wrong detail held in an agent's memory is likely to affect several later decisions, and says the number of steps an agent takes could reduce a person's opportunity to step in and correct it.

Where people's details are involved, the law has a say. The UK GDPR's accuracy principle, which applies across the UK, requires personal data to be accurate and every reasonable step to be taken to erase or correct, without delay, what is not. The ICO's guidance says the principle covers personal data that comes out of an AI system as well as what goes in, while noting that this does not mean a system must be 100% statistically accurate. That guidance was marked as under review when we read it on 3 October 2026.

What does a hallucination look like in an office?

A training provider keeps a register of delegates' certificates and their expiry dates. An AI worker is asked to update it from the certificates that awarding bodies send by email. On one poor scan the expiry date is half hidden by a stamp.

An uncontrolled system fills the gap. Most of the certificates it has seen run for three years, so it writes a date three years after the issue date. The date sits in the register, and the refresher reminder goes out on the wrong day or not at all.

A controlled system is required to take the date from the document and to show where it found it. It cannot, so it leaves the field alone and puts the certificate in front of the course administrator with a note that the expiry date is not legible. The work, as a role, is renewals and expiry tracking.

How do you stop AI making things up in business work?

1. Work from the source. The worker answers from the document or record in front of it, not from what the model remembers of the world, and keeps a pointer to where each fact came from.

2. Keep the job narrow. A worker with one defined role and the same kinds of document every day has far fewer gaps to fill than a general assistant asked anything.

3. Make "I do not know" a permitted answer. The role says what to do when something is missing or unclear, which is to stop and ask. A system with no such route will produce something.

4. Check with rules that are not AI. A date must be a real date and later than the issue date. A total must equal its lines. A reference must exist in the system it claims to come from. These checks are ordinary software and do not hallucinate.

5. Add the three controls every AI worker has. Its guardrails limit what it can do, so that a wrong belief cannot become a costly action. Human in the loop puts a person in front of what matters. The audit trail keeps a record, so that an error can be traced and its cause removed.

Is it a hallucination, out-of-date knowledge or a misread?

• Out-of-date knowledge. A model that does not know about last month's rule change is behind rather than hallucinating, and the remedy is the same: work from the current source.

• A misread document. Reading a 1 as a 7 on a bad scan is a reading error, and a person would make it too. Validation rules catch only the misreadings that break a rule, which is why the approver sees the value beside the document it was read from.

What should you ask a supplier?

Be wary of anyone who says their system does not hallucinate. Ask what it does when the information is not there, how each fact it writes is tied to a source, and which outputs a person sees before they are used. An honest answer describes the controls and promises no perfection.

Professional judgement sits outside all of this: advice, sign-off and decisions about people stay with a person whatever the system's accuracy. On which admin suits AI, see whether AI can do back-office admin work. Our starting price for a managed AI worker is on the homepage.

The worker would have one job and work from your own records, and where it is unsure it would stop and ask your people. If accuracy is what has kept you from trying AI on your admin, talk to our agent.