Which of the big three chatbots makes things up the most? It is the question every team asks before signing a licence, and the one nobody can answer with a single number. Grok, ChatGPT, and Gemini all hallucinate. They just do it differently, at different rates, depending on what you ask them to do. So instead of hunting for a magic percentage, this guide shows how to build an evaluation that actually holds up in 2027, when model versions change every few months and leaderboards go stale within weeks.
Why a Single Hallucination Rate Is a Myth
Hallucination means a model produces fluent, confident output that is not supported by facts or by the source it was given. Researchers split it into two types: intrinsic hallucination, where the model contradicts its source, and extrinsic hallucination, where it adds claims the source never made. That distinction comes from the widely cited survey of hallucination in natural language generation, and it matters because the two types need different tests.
Here is the trap. A benchmark that measures summarization faithfulness is not measuring the same thing as a closed-book trivia quiz. A model can score brilliantly on one and poorly on the other. When someone quotes “Grok hallucinates 3% of the time,” ask three follow-up questions: which benchmark, which model version, and which date? Without those answers, the number is decoration.
The Benchmarks That Actually Get Cited
Four evaluation frameworks dominate the conversation. Each one tests a different skill, which is why their rankings rarely agree.
| Benchmark | What it measures | Best used for |
|---|---|---|
| HHEM (Vectara) | Whether a summary introduces unsupported claims | Document summarization and RAG pipelines |
| SimpleQA (OpenAI) | Short-form factual accuracy, with abstention allowed | Trivia-style fact recall |
| FACTS Grounding (Google DeepMind) | Whether answers stay grounded in long provided documents | Long-context research tasks |
| TruthfulQA | Resistance to common misconceptions | Historical baseline, now largely saturated |
The Vectara hallucination leaderboard based on the HHEM model is the most quoted source for summarization faithfulness, and it updates constantly, so any figure you cite must be pulled on the day you publish. OpenAI’s SimpleQA factuality benchmark deliberately includes questions that strong models get wrong, and it rewards models for saying “I don’t know” rather than guessing. Google DeepMind’s FACTS Grounding benchmark scores grounding and comprehensiveness separately, which is genuinely useful because a cautious, empty answer should not earn full marks. TruthfulQA, from the original TruthfulQA paper, is still referenced but is too easy for current models to be a fair ranking tool.
How Grok, ChatGPT, and Gemini Differ in Practice
Each vendor optimizes for something different, and that shapes where errors show up.
ChatGPT tends to be conservative on factual questions. It hedges, adds caveats, and sometimes refuses when it could answer. That lowers its visible error rate on trivia but frustrates users who want a direct response. In summarization tasks it is generally strong, though it will occasionally smooth over gaps in a source document by inventing a plausible transition sentence.
Gemini leans hard on grounding and long-context work. Given a 200-page report, it stays anchored to the text more reliably than most rivals, which is exactly what FACTS Grounding was built to measure. Its weak spot is confident extrapolation: ask it about something adjacent to the document and it may blend source material with general knowledge without flagging the shift.
Grok is the wildcard. It is tuned for a more conversational, opinion-forward style, and that personality cuts both ways. Users report it is willing to commit to an answer where others hedge, which reads as confidence but occasionally produces assertions with thin backing. Its real-world knowledge of very recent events depends heavily on how its live search features are wired at any given moment.
None of these patterns is permanent. Model updates can flip behaviour overnight, which is why the practical comparison articles worth reading, such as this Grok vs ChatGPT vs Gemini comparison for 2026, date every claim they make.
Building a 2027 Evaluation That Survives Model Updates
The teams that get reliable answers stop asking “which model hallucinates least” and start asking “which model hallucinates least on our tasks.” Here is a workflow that works.
- Collect 50 to 100 real prompts from your own work. Support tickets, contract questions, product spec summaries. Public benchmarks tell you about the benchmark; your prompts tell you about your job.
- Write the correct answer for each prompt by hand. This is the slow part and the part people skip. Without a gold answer, you cannot measure anything.
- Run every prompt through all three models on the same day. Version drift is real, so a test spread across two weeks is not a comparison.
- Score each output on three axes: factually correct, fully supported by any source you provided, and honest about uncertainty. A confident wrong answer scores worst; a clear “I can’t verify this” scores better than a guess.
- Log the model version string for every run. When a vendor ships an update, you will want to know whether your results moved because of the model or because of your prompts.
- Re-run quarterly. Treat it like a security audit, not a one-off purchase decision.
Practical Tips That Save Time
Blind your scoring. Have a colleague rate outputs without knowing which model produced them, because brand expectations skew judgement badly. Keep a running list of the specific failure modes you see, such as invented citations, wrong dates, or merged facts from two documents, and count them separately. And test the abstention behaviour explicitly: ask questions with no answer and see which model admits it. A model that says “I don’t know” is more valuable than one that always has a confident reply.
Common Mistakes to Avoid
The biggest error is trusting a vendor’s own evaluation of its own model. The second is comparing numbers from different benchmarks as if they share a scale. The third is testing once and assuming the result holds for a year. And the fourth, which catches even experienced teams, is using prompts that leaked into training data, so the model is reciting rather than reasoning. Hold back a private test set that never appears in public.
Where Governance Fits In
Evaluation now runs as both an engineering exercise and a risk-management exercise. Risk frameworks treat confabulation as a managed risk rather than an unavoidable quirk. The NIST AI Risk Management Framework sets out how organizations should map, measure, and manage risks including fabricated output, and it pairs naturally with an internal evaluation log. If your organization has to justify model choices to a client, a regulator, or an internal audit team, a dated, documented test set is the evidence that carries weight. No binding numeric hallucination threshold exists in law, so your own measurements are the standard you will be held to. (NIST Artificial Intelligence overview)
What to Watch Through 2027
Three trends will shape how this comparison looks a year from now. First, grounding-first design is becoming the default: more products will refuse to answer without a retrieved source, which lowers hallucination on document tasks while pushing errors into the retrieval layer instead. Second, abstention will become a feature, with models scored on knowing when to stop. Third, benchmark contamination will keep eroding trust in public leaderboards, pushing serious buyers toward private test sets. The comparison articles that stay useful, like this ChatGPT vs Gemini vs Claude breakdown of context limits, coding, and cost, will be the ones that separate capability from reliability.
It also helps to compare models on the task you actually care about. If your work is research-heavy, the Perplexity vs ChatGPT comparison of citation quality and speed is a more relevant read than a general leaderboard, because citation accuracy is a hallucination metric in disguise.
Conclusion
Grok, ChatGPT, and Gemini all produce confident output that is sometimes wrong, and no single percentage captures the difference between them. The honest answer to “which one hallucinates least” is: it depends on the task, the model version, and the day you tested. What you can control is your own evaluation. Build a small, private, hand-scored test set from real work, run all three models on the same day, log the versions, and repeat every quarter. That practice will tell you more about output quality than any leaderboard screenshot, and it will still be valid long after the 2027 model releases have come and gone.
Frequently Asked Questions
Which model has the lowest hallucination rate?
There is no stable answer. Rates shift with every model update and depend entirely on which benchmark you use. A model that leads on summarization faithfulness may trail on short-form factual recall. Test on your own tasks rather than trusting a single published number.
Is Grok more prone to hallucination than ChatGPT?
Not in any consistent, measurable way across all tasks. Grok tends to answer more directly and with more personality, which can make its errors feel more confident. ChatGPT hedges more often, which hides some errors behind caveats. Both make things up, and the gap between them changes with each release.
What is the difference between intrinsic and extrinsic hallucination?
Intrinsic hallucination means the model contradicts the source it was given, such as summarizing a report and stating the opposite of what it says. Extrinsic hallucination means the model adds claims the source never made. Both damage trust, but they need separate tests. (Hallucination (artificial intelligence)))
How often should I re-test model accuracy?
Quarterly is a sensible baseline for most teams, plus a fresh run whenever a vendor announces a major model update. Model behaviour can shift noticeably between versions, so results older than three months should be treated as historical rather than current.
Can I reduce hallucination without switching models?
Yes. Grounding the model in retrieved documents, asking it to cite passages, and instructing it to say “I don’t know” when unsure all reduce fabricated output. Human review on high-stakes answers remains essential, because no configuration eliminates the problem entirely.
Are public hallucination leaderboards reliable?
They are useful starting points, not verdicts. Public benchmarks can leak into training data, and leaderboards measure narrow tasks. Use them to shortlist models, then run your own blinded test on real prompts before making a decision.
Check the leaderboard’s date before you quote it. Vectara’s numbers move week to week, and a figure lifted from a cached page can be months stale. If you must cite a public score, link to the live page and note the retrieval date beside it. Readers who later find a different number will trust your analysis more, not less, because you showed your work.
Keep a failure journal alongside a score. When a model invents a citation or merges two documents, paste the prompt and the output into a shared log. Over a quarter, patterns emerge that no single test run reveals: one model may fail repeatedly on dates, another on attributions. That log becomes your most persuasive internal document when someone asks why you chose one vendor over another.
Revisit your gold answers too. Facts change, and a test set written in early 2027 may contain claims that were true then and are wrong now. Schedule a light annual review of your answer key so you are not penalizing a model for being right about a world that moved on.
None of this requires a large team. A spreadsheet, a shared folder, and a recurring calendar block will outperform most purchased evaluation suites, because the prompts come from your actual work and the scoring reflects your actual standards. The models will keep changing. Your method does not have to.
