- Primary metric
- Fabrication rate (lower is better)
- Repository
- https://github.com/LostBee/halluc-bench
- Tracks
- closed-book, false-premise
Methodology
Hallucination Bench separates confident fabrication from explicit abstention and calibrated hedging across 1,200 verifiable prompts. Prompts span archival records, bibliographic citations, and deliberately unanswerable false premises.
Latest results
From No. 002 (2026-11-14).
Claude Opus 4.5
6.1%
Gemini 3 Pro
8.4%
GPT-5
9.8%
Qwen3 Max
12.7%
Grok 4
14.2%
Mistral Large 3
15.1%
DeepSeek V3.2
16.5%
Llama 4 Maverick
18.9%
Show data
| Model | Fabrication rate (%) | 95% CI | n |
|---|---|---|---|
| Claude Opus 4.5 | 6.1% | 4.8% – 7.5% | 1200 |
| Gemini 3 Pro | 8.4% | 6.9% – 10% | 1200 |
| GPT-5 | 9.8% | 8.2% – 11.6% | 1200 |
| Qwen3 Max | 12.7% | 10.9% – 14.7% | 1200 |
| Grok 4 | 14.2% | 12.3% – 16.3% | 1200 |
| Mistral Large 3 | 15.1% | 13.1% – 17.2% | 1200 |
| DeepSeek V3.2 | 16.5% | 14.5% – 18.7% | 1200 |
| Llama 4 Maverick | 18.9% | 16.7% – 21.2% | 1200 |
| Model | Fabrication rate (%) | 95% CI | n | Note |
|---|---|---|---|---|
| Claude Opus 4.5 | 6.1% | 4.8% – 7.5% | 1200 | Highest abstention precision |
| Gemini 3 Pro | 8.4% | 6.9% – 10% | 1200 | — |
| GPT-5 | 9.8% | 8.2% – 11.6% | 1200 | — |
| Qwen3 Max | 12.7% | 10.9% – 14.7% | 1200 | — |
| Grok 4 | 14.2% | 12.3% – 16.3% | 1200 | — |
| Mistral Large 3 | 15.1% | 13.1% – 17.2% | 1200 | — |
| DeepSeek V3.2 | 16.5% | 14.5% – 18.7% | 1200 | — |
| Llama 4 Maverick | 18.9% | 16.7% – 21.2% | 1200 | — |
Run timeline
- No. 002