# Benchmark saturation: why leaderboard scores say less than they used to

- **Published:** 2026-01-23
- **Updated:** 2026-10-12
- **Category:** Models & Benchmarks
- **Author:** Sinthetix
- **URL:** https://sinthetix.com/blog/benchmark-saturation-what-to-trust

> AI benchmarks are saturating and scores depend on test setup. Three published examples, and how to read a model announcement without being misled.

Model announcements still lead with benchmark tables. The tables are getting less informative, for two published reasons: the tests are running out of headroom, and the score depends on how the test is run.

**Key takeaways**

- Scores on well-known benchmarks rose by tens of percentage points in a single year, leaving little room to separate top models.
- Vendors disclose that setup details, such as the hosting environment, can change a model's score.
- A grading error can hide most of a model's real ability, so a score is only as good as the test behind it.

## The tests are filling up

Stanford's 2025 AI Index reported that, in one year, scores rose by 18.8 percentage points on MMMU, 48.9 on GPQA and 67.3 on SWE-bench, three benchmarks introduced in 2023 [1]. The gap between the top-ranked and tenth-ranked model fell from 11.9 percent to 5.4 percent in a year, and the top two models were separated by just 0.7 percent [1].

**One-year score gain on three benchmarks**

Percentage points, per Stanford's 2025 AI Index

Unit: pp

| Category | Score gain |
| --- | --- |
| MMMU | 18.8 |
| GPQA | 48.9 |
| SWE-bench | 67.3 |

*Year-over-year improvement as reported in the AI Index takeaways; these benchmarks were introduced in 2023. Source: [1], Stanford HAI*

When several models cluster near the top, a one-point lead is within the noise of how the test was run.

## The setup changes the score

Anthropic's Opus 4.5 announcement shows how much fine print sits behind a table. Most of its evaluations used a 64K thinking budget and were averaged over five trials, while SWE-bench Verified used no thinking budget [2]. A footnote says Anthropic improved its hosting environment, which raised Gemini 3 to 56.7 percent and GPT-5.1 to 48.6 percent from the values their developers reported, using the Terminus-2 harness [2]. The same post says the model's capabilities "outpace some of the benchmarks" Anthropic uses, citing a case where a creative solution was scored as a failure because the test did not anticipate it [2].

## The grader can be the problem

Anthropic's engineering team described a related case in a January 9 post. On CORE-Bench, Opus 4.5 initially scored 42 percent. After fixing grading and task-specification issues and using a less constrained scaffold, it reached 95 percent [3]. The model had not changed; the test had.

## How to read a model announcement

- Ask for the settings: reasoning budget, number of trials, tools allowed, and who ran the test.
- Prefer a spread of results over a single headline score.
- Discount differences of a point or two between top models.
- Be skeptical of benchmarks unrelated to your work.

## The practical implication

Public benchmarks are useful for shortlisting, not for deciding. Build a small test set from your own documents and tasks, and let the decision rest on how candidate models perform on it.

## Sources

1. [The 2025 AI Index Report: Top Takeaways](https://hai.stanford.edu/ai-index/2025-ai-index-report), Stanford HAI, 2025-04-07
2. [Introducing Claude Opus 4.5](https://www.anthropic.com/news/claude-opus-4-5), Anthropic, 2025-11-24
3. [Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents), Anthropic Engineering, 2026-01-09

## How Sinthetix helps

**Generative AI & LLM**: Integrate powerful AI models to turn unstructured data into insights and power chatbots.

- RAG systems
- LLM-powered chatbots
- Document analysis

[Explore Generative AI & LLM](https://sinthetix.com/services/generative-ai-llm) | [Talk to us](https://sinthetix.com/contact)

Sinthetix is an AI engineering studio. We ship agents, automation, and internal tools for operators who need results, not reports. [About Sinthetix](https://sinthetix.com/about)