Before you start
You need one chat window and a reply you can check against something real, such as your own CV, a book you know well or a public web page. There is no code. The exercises build a checking habit and do not score any vendor.
Check an AI answer in five steps
- Mark the claims. Underline every name, number, date, quote and link in the reply. Those are the parts you can test against a source.
- Ask where each claim comes from, then open the source. Google's Gemini help says "Not all responses include related links or sources", so a missing link is normal. A link that is there still has to contain the sentence the answer attributes to it.
- Give the model a way out. Add "If you are not sure of something, say so instead of guessing." Anthropic's guide says explicit permission to admit uncertainty can drastically reduce false information.
- When you supply a document, ask for word-for-word quotes first and the answer second. The same guide suggests asking the model to retract any claim it cannot back with a quote.
- Run the question again in a new chat and compare. The guide says inconsistencies across repeated runs could indicate hallucinations. When the runs agree, still check the facts that matter, because agreement is not proof.
Why a model makes things up
Google's machine learning course says it plainly: "LLMs hallucinate, meaning their predictions often contain mistakes." A model writes the most likely next tokens, as the first lesson showed, so a sentence can be fluent and false at once.
Researchers at OpenAI went further in a 2025 paper. They define a hallucination as a plausible falsehood and argue that models hallucinate because training and evaluation reward guessing over acknowledging uncertainty. The comparison they draw is to students facing hard exam questions. Under a scoring rule that gives zero for a blank and zero for a wrong answer, guessing never costs anything. In their opening example, a language model gave three wrong dates on three attempts when asked for a person's birthday. Their proposed fix is to change how existing benchmarks score answers.
The paper backs this with a review of ten widely used benchmarks, shown below. Nine grade each answer only as right or wrong and give no credit for "I don't know". The tenth, WildBench, gives partial credit, though the authors note that its scale may still rate "I don't know" below a fair answer that contains a hallucination.
Anthropic's interpretability study looked inside Claude. It found that declining to answer is the default behavior, and that a "known entities" feature can switch that default off. The misfire happens when Claude recognizes a name but knows nothing else about the person.
| Source | What it says | What to do with it |
|---|---|---|
| Google machine learning course | LLMs hallucinate, meaning their predictions often contain mistakes | Read fluent text as unchecked text |
| OpenAI researchers, 2025 | Training and scoring reward guessing over admitting doubt | Give the model explicit permission to say it does not know |
| Anthropic interpretability study, 2025 | Declining is the default, and a half-known name can switch it off | Be most careful with obscure people, places and numbers |
| Graded only right or wrong | Credit for "I don't know" | |
|---|---|---|
| GPQA | Yes | No |
| MMLU-Pro | Yes | No |
| IFEval | Yes | No |
| Omni-MATH | Yes | No |
| WildBench | No | Partly |
| BBH | Yes | No |
| MATH (L5 split) | Yes | No |
| MuSR | Yes | No |
| SWE-bench | Yes | No |
| HLE | Yes | No |
Source: Kalai, Nachum, Vempala and Zhang (OpenAI): Why language models hallucinate, table 2 (read November 5, 2026)
Why a source link does not end the checking
Microsoft's application card says Copilot aims to respond with reliable sources where grounding information is sought, yet AI can still fabricate content that sounds reasonable but is factually inaccurate. Even when it draws on high-authority web data, a response might misrepresent that content and might not be completely accurate, up to date or reliable.
The card adds a second case. When Copilot summarizes content from several sources, it may include information that is not in them. Anthropic gives a general rule in the last lines of its guide.
although these techniques significantly reduce hallucinations, they don't eliminate them entirely. Always validate critical information, especially for high-stakes decisions.
How much checking a task needs
Checking costs time, so match it to the stakes. The grading below is our own reading of the vendors' warnings, not a vendor standard. Microsoft's card names medical and financial work as high-stakes areas, and it warns against uses that could affect a person's legal status or access to credit, employment, housing or insurance.
| Task | Risk | What to do |
|---|---|---|
| Brainstorming names, outlines or ideas | Low | Read it and keep what you like |
| Summarizing a document you pasted | Medium, because a summary can add details the document lacks | Spot-check three claims against the text |
| Quotes, numbers, dates or links you will publish | High | Open every source and find the line |
| Health, money, law or safety decisions | Highest | Treat the AI as a way to form better questions for a professional |
Andrej Karpathy on dream machines, and Rob Freund on fake cases in court
Andrej Karpathy, an AI researcher and educator and a founding member of OpenAI, wrote on X on 9 December 2023 that in some sense hallucination is all a large language model does. A prompt starts a dream, and most of the time the dream lands somewhere useful. We call it a hallucination only when it drifts somewhere we judge to be false.
Among the fixes he lists for chat assistants are comparing several samples and anchoring the answer in real documents, which are steps 4 and 5 above. Rob Freund, a lawyer for brands, agencies and creators, posted on 30 June 2025 about a Georgia trial court that relied on made-up cases from AI in an order. The appeal court vacated it, and the other side's lawyer then cited even more made-up cases in the appeal briefs.
A court order belongs in the highest-risk row of the table above, and those cases got past a trial judge. Before a name, case, quote or number goes into something you sign, file or publish, open the source and find it yourself.
Hallucination is not a bug, it is LLM's greatest feature. The LLM Assistant has a hallucination problem, and we should fix it.
Screenshots of X's own embeds of the posts, taken on 5 November 2026, with the time in UTC. X cuts Karpathy's post short with Show more. Select one to open the post on X.
Common mistakes
- Asking the same model "are you sure?" and counting a yes as a check. It is more generated text, and Anthropic's study found Claude can make up plausible-sounding steps.
- Trusting a citation because it looks formal. Open it and find the sentence.
- Taking agreement between two runs as proof. A repeated wrong answer is still wrong.
- Treating a half-famous name as a safe question. Anthropic ties made-up answers to a name the model recognizes but knows little about.
Try it yourself
- Ask your assistant for a short biography of someone semi-public whom you know well. Mark each claim right, wrong or unverifiable, then repeat with the "say so if you are not sure" line and compare.
- Paste a paragraph, ask for a summary and check three claims against your text.
- Ask for sources on a topic you know, open each link and count how many contain the sentence the answer attributed to them.
Next: what you share with the model
Checking the output covers only what comes back. The next lesson, AI privacy, covers what you send: what the big chat apps keep from your messages and what never to paste.
The first lesson, how large language models work, explains the guessing behind a wrong answer and is worth a second look after this one.
Check yourself
Which sentence best describes an AI hallucination?
Show answer
C. A fluent reply that states something false as if it were true The OpenAI researchers define a hallucination as a plausible falsehood.
You paste a long report and want a summary you can trust. What does Anthropic's guide suggest asking for first?
Show answer
A. Word-for-word quotes from the report The guide says quotes ground the response in the actual text, and a claim the model cannot back with a quote should be retracted.
A chatbot attaches a link to a statistic. What should you do next?
Show answer
D. Open the link and find the sentence that supports the statistic Microsoft says even answers drawn from high-authority web data may misrepresent that content, so the check happens at the source.
According to the OpenAI researchers, why do models keep guessing?
Show answer
B. Training and scoring reward guessing over admitting uncertainty The paper argues that models hallucinate because training and evaluation procedures reward guessing over acknowledging uncertainty.
Frequently asked questions
Do newer models still hallucinate?
Yes. Anthropic's guide opens by saying that even the most advanced language models can generate text that is factually incorrect or inconsistent with the given context. Its techniques reduce the problem and do not remove it.
Does turning on web search stop hallucinations?
It helps and does not end the problem. Search hands the model pages to draw on, but the model still writes the answer, which is why the card quoted above warns about misrepresented content. Gemini's help adds that not every response carries sources, so open the links that do come with an answer.
Is a hallucination the same as a lie?
No. The OpenAI researchers describe it as a plausible falsehood that comes from how models are trained and scored, much like a student guessing on a hard exam question. Nothing in that account involves intent to deceive.
Why does the AI sound so sure when it is wrong?
Fluent text is what a model produces, and the OpenAI paper argues that scoring rewards a confident guess over an honest "I don't know". Anthropic's study says something similar about training: models are always supposed to give a guess for the next word.
What can I ask the AI to cut down on made-up answers?
Anthropic lists several instructions. Allow "I don't know" and ask for word-for-word quotes first. Ask it to back each claim with a quote or remove it. Tell it to use only the documents you provided and not its general knowledge.
Can I use AI for a health, money or legal question?
As preparation, yes. Ask it to explain a term, list questions for your doctor or adviser and sort what to bring. The decision belongs with a professional, and the table above rates these tasks highest. That grading is our reading, not a vendor rule.
Sources
- Anthropic: Reduce hallucinations https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/reduce-hallucinations
- Anthropic Research: Tracing the thoughts of a large language model https://www.anthropic.com/research/tracing-thoughts-language-model
- Anthropic: Glossary https://platform.claude.com/docs/en/about-claude/glossary
- Kalai, Nachum, Vempala and Zhang (OpenAI): Why language models hallucinate https://arxiv.org/abs/2509.04664
- Google: Machine Learning Crash Course, Large language models https://developers.google.com/machine-learning/crash-course/llm/transformers
- Google Gemini Apps Help: View related sources https://support.google.com/gemini/answer/14143489
- Microsoft Learn: Application card, Microsoft Copilot (for organizations) https://learn.microsoft.com/en-us/copilot/plan/microsoft-365-copilot-application-card
- Andrej Karpathy on X, 9 December 2023 https://x.com/karpathy/status/1733299213503787018
- Rob Freund on X, 30 June 2025 https://x.com/RobertFreundLaw/status/1939805015423820183