The question isn’t whether models hallucinate—they all do—but how the failure is distributed. Some models fabricate rarely but grandly; others hedge on everything.1

Fabrication rate across eight models

Every model was evaluated on the same 1,200 closed-book factual questions with temperature fixed to zero and no tool access.2

Show data
ModelFabrication rate (%)95% CIn
Claude Opus 4.5 6.1% 4.8% – 7.5% 1200
Gemini 3 Pro 8.4% 6.9% – 10% 1200
GPT-5 9.8% 8.2% – 11.6% 1200
Qwen3 Max 12.7% 10.9% – 14.7% 1200
Grok 4 14.2% 12.3% – 16.3% 1200
Mistral Large 3 15.1% 13.1% – 17.2% 1200
DeepSeek V3.2 16.5% 14.5% – 18.7% 1200
Llama 4 Maverick 18.9% 16.7% – 21.2% 1200
Fig. 1 — Fabrication rate on closed-book factual questions (placeholder data). Lower is better.

Full snapshot table

The table below preserves 95% Wilson confidence intervals alongside sample counts for each evaluated checkpoint.

Model Fabrication rate (%) 95% CI n Note
Claude Opus 4.5 6.1% 4.8% – 7.5% 1200 Highest abstention precision
Gemini 3 Pro 8.4% 6.9% – 10% 1200 —
GPT-5 9.8% 8.2% – 11.6% 1200 —
Qwen3 Max 12.7% 10.9% – 14.7% 1200 —
Grok 4 14.2% 12.3% – 16.3% 1200 —
Mistral Large 3 15.1% 13.1% – 17.2% 1200 —
DeepSeek V3.2 16.5% 14.5% – 18.7% 1200 —
Llama 4 Maverick 18.9% 16.7% – 21.2% 1200 —

Patterns in false-premise prompts

Roughly a third of fabrications occur on questions built around plausible-sounding but non-existent papers or statutory clauses. Models that abstain early on malformed premises cut their overall fabrication rate nearly in half.