Benchmark saturation: why leaderboard scores say less than they used to

Model announcements still lead with benchmark tables. The tables are getting less informative, for two published reasons: the tests are running out of headroom, and the score depends on how the test is run.

The tests are filling up

Stanford's 2025 AI Index reported that, in one year, scores rose by 18.8 percentage points on MMMU, 48.9 on GPQA and 67.3 on SWE-bench, three benchmarks introduced in 2023 ^[1]. The gap between the top-ranked and tenth-ranked model fell from 11.9 percent to 5.4 percent in a year, and the top two models were separated by just 0.7 percent ^[1].

When several models cluster near the top, a one-point lead is within the noise of how the test was run.

The setup changes the score

Anthropic's Opus 4.5 announcement shows how much fine print sits behind a table. Most of its evaluations used a 64K thinking budget and were averaged over five trials, while SWE-bench Verified used no thinking budget ^[2]. A footnote says Anthropic improved its hosting environment, which raised Gemini 3 to 56.7 percent and GPT-5.1 to 48.6 percent from the values their developers reported, using the Terminus-2 harness ^[2]. The same post says the model's capabilities "outpace some of the benchmarks" Anthropic uses, citing a case where a creative solution was scored as a failure because the test did not anticipate it ^[2].

The grader can be the problem

Anthropic's engineering team described a related case in a January 9 post. On CORE-Bench, Opus 4.5 initially scored 42 percent. After fixing grading and task-specification issues and using a less constrained scaffold, it reached 95 percent ^[3]. The model had not changed; the test had.

How to read a model announcement

• Ask for the settings: reasoning budget, number of trials, tools allowed, and who ran the test.

• Prefer a spread of results over a single headline score.

• Discount differences of a point or two between top models.

• Be skeptical of benchmarks unrelated to your work.

The practical implication

Public benchmarks are useful for shortlisting, not for deciding. Build a small test set from your own documents and tasks, and let the decision rest on how candidate models perform on it.