The question isn’t whether models hallucinate—they all do—but how the failure is distributed. Some models fabricate rarely but grandly; others hedge on everything.1
Fabrication rate across eight models
Every model was evaluated on the same 1,200 closed-book factual questions with temperature fixed to zero and no tool access.2
Claude Opus 4.5
6.1%
Gemini 3 Pro
8.4%
GPT-5
9.8%
Qwen3 Max
12.7%
Grok 4
14.2%
Mistral Large 3
15.1%
DeepSeek V3.2
16.5%
Llama 4 Maverick
18.9%
Show data
| Model | Fabrication rate (%) | 95% CI | n |
|---|---|---|---|
| Claude Opus 4.5 | 6.1% | 4.8% – 7.5% | 1200 |
| Gemini 3 Pro | 8.4% | 6.9% – 10% | 1200 |
| GPT-5 | 9.8% | 8.2% – 11.6% | 1200 |
| Qwen3 Max | 12.7% | 10.9% – 14.7% | 1200 |
| Grok 4 | 14.2% | 12.3% – 16.3% | 1200 |
| Mistral Large 3 | 15.1% | 13.1% – 17.2% | 1200 |
| DeepSeek V3.2 | 16.5% | 14.5% – 18.7% | 1200 |
| Llama 4 Maverick | 18.9% | 16.7% – 21.2% | 1200 |
Full snapshot table
The table below preserves 95% Wilson confidence intervals alongside sample counts for each evaluated checkpoint.
| Model | Fabrication rate (%) | 95% CI | n | Note |
|---|---|---|---|---|
| Claude Opus 4.5 | 6.1% | 4.8% – 7.5% | 1200 | Highest abstention precision |
| Gemini 3 Pro | 8.4% | 6.9% – 10% | 1200 | — |
| GPT-5 | 9.8% | 8.2% – 11.6% | 1200 | — |
| Qwen3 Max | 12.7% | 10.9% – 14.7% | 1200 | — |
| Grok 4 | 14.2% | 12.3% – 16.3% | 1200 | — |
| Mistral Large 3 | 15.1% | 13.1% – 17.2% | 1200 | — |
| DeepSeek V3.2 | 16.5% | 14.5% – 18.7% | 1200 | — |
| Llama 4 Maverick | 18.9% | 16.7% – 21.2% | 1200 | — |
Patterns in false-premise prompts
Roughly a third of fabrications occur on questions built around plausible-sounding but non-existent papers or statutory clauses. Models that abstain early on malformed premises cut their overall fabrication rate nearly in half.