Primary metric
Fabrication rate (lower is better)
Repository
https://github.com/LostBee/halluc-bench
Tracks
closed-book, false-premise

Methodology

Hallucination Bench separates confident fabrication from explicit abstention and calibrated hedging across 1,200 verifiable prompts. Prompts span archival records, bibliographic citations, and deliberately unanswerable false premises.

Latest results

From No. 002 (2026-11-14).

Show data
ModelFabrication rate (%)95% CIn
Claude Opus 4.5 6.1% 4.8% – 7.5% 1200
Gemini 3 Pro 8.4% 6.9% – 10% 1200
GPT-5 9.8% 8.2% – 11.6% 1200
Qwen3 Max 12.7% 10.9% – 14.7% 1200
Grok 4 14.2% 12.3% – 16.3% 1200
Mistral Large 3 15.1% 13.1% – 17.2% 1200
DeepSeek V3.2 16.5% 14.5% – 18.7% 1200
Llama 4 Maverick 18.9% 16.7% – 21.2% 1200
Fig. 1 — Fabrication rate on Hallucination Bench. Lower is better.
Model Fabrication rate (%) 95% CI n Note
Claude Opus 4.5 6.1% 4.8% – 7.5% 1200 Highest abstention precision
Gemini 3 Pro 8.4% 6.9% – 10% 1200 —
GPT-5 9.8% 8.2% – 11.6% 1200 —
Qwen3 Max 12.7% 10.9% – 14.7% 1200 —
Grok 4 14.2% 12.3% – 16.3% 1200 —
Mistral Large 3 15.1% 13.1% – 17.2% 1200 —
DeepSeek V3.2 16.5% 14.5% – 18.7% 1200 —
Llama 4 Maverick 18.9% 16.7% – 21.2% 1200 —

Run timeline