DeepSeek v4 coding scores conflict with broader capability assessments
Is this a scandal?
No longer — the story has resolved. Noise 1/100, cooling down, across 0 sources.
Evaluation standards will likely shift toward more dynamic, private benchmarks to prevent models from over-optimizing on static public coding tests. Over the next few quarters, we will see an increased emphasis on agentic and multi-step reasoning evaluations rather than raw code completion.
Noise 1/100 — louder than 87% of tracked AI controversies.
Why it matters
The discrepancy highlights how narrow optimization on public benchmarks can mask broader gaps in agentic reasoning, safety, and cybersecurity capabilities, complicating how enterprises evaluate open-weights models.
Key points
- DeepSeek v4's Pro configuration scored 80.6 on SWE-bench Verified and 93.5 on LiveCodeBench, placing it near the top of global coding leaderboards.
- A comprehensive evaluation by CAISI placed DeepSeek v4 eight months behind the US frontier, citing deficiencies in cybersecurity and abstract reasoning.
- DeepSeek originally marketed the v4 model as being only two months behind the leading frontier models at launch.
- The launch of the closed-source Fable 5 model has further stretched the gap between open-weights models and the absolute frontier.
- Developers running quantized or 'Flash' versions of the model locally report a noticeable performance drop compared to the headline 1.6-trillion-parameter Pro configuration.
The story
An industry debate has emerged over the performance profile of DeepSeek's newly released DeepSeek v4 model. While the model's 1.6-trillion-parameter Pro configuration achieved near-top scores on coding benchmarks—specifically 80.6 on SWE-bench Verified and 93.5 on LiveCodeBench—a broader evaluation by the Center for AI Safety and Innovation (CAISI) concluded that the model remains roughly eight months behind the current US frontier. DeepSeek had previously claimed the model was only two months behind leading systems. Analysts suggest that the high coding scores reflect targeted optimization for specific developer benchmarks, whereas the CAISI evaluation exposed performance gaps in more complex domains, including cybersecurity and abstract reasoning. Additionally, the recent launch of the closed-source Fable 5 model is expected to widen this frontier gap further.
Who's involved
Claims DeepSeek v4 is highly competitive, sitting only two months behind the global frontier at launch.
Assesses DeepSeek v4 as being approximately eight months behind the US frontier due to gaps in abstract reasoning and cybersecurity.
Noise Level
The timeline
Community debate highlights benchmark divergence
Users and researchers analyze the gap between DeepSeek v4's elite coding scores and CAISI's broader capability evaluations.
Fable 5 released
A new closed-source frontier model, Fable 5, is launched, advancing the state of the art.
The forecast
Evaluation standards will likely shift toward more dynamic, private benchmarks to prevent models from over-optimizing on static public coding tests. Over the next few quarters, we will see an increased emphasis on agentic and multi-step reasoning evaluations rather than raw code completion.
Forecast, not fact — an editorial estimate we score when this resolves.
That's the complete picture as of — nothing more to know right now. We'll update this page the moment it changes.
Join the Discussion
Discuss this story
Community comments coming in a future update
Be the first to share your perspective. Subscribe to comment.