Reasoning models and test-time compute: what changed, and what it costs

For most of the last decade, the main route to a better language model was a bigger one: more parameters, more training data, more training compute. Reasoning models add a second lever. Instead of only improving the model before deployment, they spend more computation at the moment a question is asked, thinking through intermediate steps before answering. This week's release of Kimi K2 Thinking, an open-weights reasoning model from Moonshot AI, is a good snapshot of where that approach stands and what it means for buyers.

What "test-time compute" means

A conventional model produces an answer in a single pass. A reasoning model generates a long internal chain of thought, and may explore, check or revise candidate answers before it responds. The extra work happens at inference time, which is why researchers call it test-time compute.

The idea was formalized in an August 2024 paper from Snell, Lee, Xu and Kumar. They found that a "compute-optimal" strategy, which allocates test-time compute adaptively according to how hard each prompt is, made test-time scaling more than 4x more efficient than a best-of-N baseline ^[1]. In a comparison matched on total computation, a smaller model given extra test-time compute outperformed a model 14 times its size, on problems where the smaller model already had a non-trivial success rate ^[1]. Two caveats from the same paper matter for buyers: the benefit depends heavily on prompt difficulty, and it is conditioned on the smaller model having some success to begin with ^[1]. Thinking longer does not rescue a model that cannot do the task at all.

What Kimi K2 Thinking adds

Moonshot released K2 Thinking around November 6, 2025; a same-day blog post by Simon Willison reports the release ^[3], though the model card carries no date. It is a Mixture-of-Experts model with 1 trillion total parameters and 32 billion activated per token, with a 256K-token context window ^[2]. Its distinguishing claim is not raw reasoning but endurance: it interleaves chain-of-thought with function calls and, according to the model card, maintains stable tool use across 200 to 300 sequential calls ^[2]. That matters because agents are chains of tool calls, and reliability tends to degrade as chains grow.

Moonshot reports benchmark results with tools enabled. Its model card marks the GPT-5 and Claude Sonnet 4.5 HLE scores with an asterisk, meaning no public scores were available and Moonshot re-tested them under the same conditions as K2 Thinking ^[2].

The shape of the results is more informative than any single number. K2 Thinking leads on Humanity's Last Exam and BrowseComp, which reward multi-step research and web navigation, and trails both comparison models on SWE-bench Verified, a software engineering task ^[2].

The cost side: tokens, not just price

Test-time compute is, by definition, extra computation per question ^[1]. In practice that shows up as longer outputs, more time to an answer and a larger bill per task, even when the price per token is low. The model card does not publish per-task token counts or latency, so this post does not quote any; measure them yourself on your own workloads.

Open weights add a deployment angle. The model ships with INT4 weight-only quantization applied through quantization-aware training, which Moonshot says gives roughly a 2x generation speedup, and all of its benchmark results are reported at INT4 precision ^[2]. The weights and code are under a modified MIT license, so read its terms before building a commercial product on it ^[2].

What this does not show

• All of the benchmark numbers above come from Moonshot. Independent replication of the tool-use endurance claim is not something the cited sources provide.

• Benchmarks measure fixed tasks. A model that wins on a web research test can lose on your documents.

• Token use, speed and price for the model were not available from the sources we reviewed, so cost comparisons here are directional only.

How to decide when to pay for thinking

1. Match effort to difficulty. Route simple lookups and formatting to a fast model; reserve reasoning models for multi-step analysis, planning and research.

2. Measure cost per completed task. Count total tokens and wall-clock time, including retries.

3. Test long tool chains. If an agent needs dozens of sequential calls, test that length specifically.

4. Check the hosting needs (GPU memory, serving stack) before choosing open weights for a product.

Practical implication

Reasoning models are a tool for the hard minority of tasks, not a default. Evaluate them on your own workloads, using cost and time per finished task as the metric rather than price per token.