# Reasoning models and test-time compute: what changed, and what it costs

- **Published:** 2025-11-07
- **Updated:** 2026-10-12
- **Category:** Models & Benchmarks
- **Author:** Sinthetix
- **URL:** https://sinthetix.com/blog/reasoning-models-test-time-compute

> Moonshot's open-weights Kimi K2 Thinking, released this week, shows where reasoning models stand: stronger on research tasks, weaker on software engineering.

For most of the last decade, the main route to a better language model was a bigger one: more parameters, more training data, more training compute. Reasoning models add a second lever. Instead of only improving the model before deployment, they spend more computation at the moment a question is asked, thinking through intermediate steps before answering. This week's release of Kimi K2 Thinking, an open-weights reasoning model from Moonshot AI, is a good snapshot of where that approach stands and what it means for buyers.

**Key takeaways**

- Test-time compute lets a model "think longer" on hard problems, trading latency and tokens for accuracy.
- Research from 2024 found that allocating this compute adaptively can beat simply using a much larger model on suitable problems.
- Kimi K2 Thinking, an open-weights model, leads on research-style benchmarks but trails on software engineering, by Moonshot's own numbers.
- The bill changes: reasoning models spend extra computation per task, so cost per completed task matters more than price per token.

## What "test-time compute" means

A conventional model produces an answer in a single pass. A reasoning model generates a long internal chain of thought, and may explore, check or revise candidate answers before it responds. The extra work happens at inference time, which is why researchers call it test-time compute.

The idea was formalized in an August 2024 paper from Snell, Lee, Xu and Kumar. They found that a "compute-optimal" strategy, which allocates test-time compute adaptively according to how hard each prompt is, made test-time scaling more than 4x more efficient than a best-of-N baseline [1]. In a comparison matched on total computation, a smaller model given extra test-time compute outperformed a model 14 times its size, on problems where the smaller model already had a non-trivial success rate [1]. Two caveats from the same paper matter for buyers: the benefit depends heavily on prompt difficulty, and it is conditioned on the smaller model having some success to begin with [1]. Thinking longer does not rescue a model that cannot do the task at all.

## What Kimi K2 Thinking adds

Moonshot released K2 Thinking around November 6, 2025; a same-day blog post by Simon Willison reports the release [3], though the model card carries no date. It is a Mixture-of-Experts model with 1 trillion total parameters and 32 billion activated per token, with a 256K-token context window [2]. Its distinguishing claim is not raw reasoning but endurance: it interleaves chain-of-thought with function calls and, according to the model card, maintains stable tool use across 200 to 300 sequential calls [2]. That matters because agents are chains of tool calls, and reliability tends to degrade as chains grow.

Moonshot reports benchmark results with tools enabled. Its model card marks the GPT-5 and Claude Sonnet 4.5 HLE scores with an asterisk, meaning no public scores were available and Moonshot re-tested them under the same conditions as K2 Thinking [2].

**Kimi K2 Thinking vs. two frontier models, with tools**

Vendor-reported scores on three benchmarks

Unit: %

| Category | Kimi K2 Thinking | GPT-5 (High) | Claude Sonnet 4.5 (Thinking) |
| --- | --- | --- | --- |
| HLE (w/ tools) | 44.9% | 41.7% | 32% |
| BrowseComp (w/ tools) | 60.2% | 54.9% | 24.1% |
| SWE-bench Verified (w/ tools) | 71.3% | 74.9% | 77.2% |

*Scores as published on Moonshot's model card; they are vendor-reported and have not been independently reproduced here. Source: [2], Hugging Face (Moonshot AI)*

The shape of the results is more informative than any single number. K2 Thinking leads on Humanity's Last Exam and BrowseComp, which reward multi-step research and web navigation, and trails both comparison models on SWE-bench Verified, a software engineering task [2].

## The cost side: tokens, not just price

Test-time compute is, by definition, extra computation per question [1]. In practice that shows up as longer outputs, more time to an answer and a larger bill per task, even when the price per token is low. The model card does not publish per-task token counts or latency, so this post does not quote any; measure them yourself on your own workloads.

Open weights add a deployment angle. The model ships with INT4 weight-only quantization applied through quantization-aware training, which Moonshot says gives roughly a 2x generation speedup, and all of its benchmark results are reported at INT4 precision [2]. The weights and code are under a modified MIT license, so read its terms before building a commercial product on it [2].

## What this does not show

- All of the benchmark numbers above come from Moonshot. Independent replication of the tool-use endurance claim is not something the cited sources provide.
- Benchmarks measure fixed tasks. A model that wins on a web research test can lose on your documents.
- Token use, speed and price for the model were not available from the sources we reviewed, so cost comparisons here are directional only.

## How to decide when to pay for thinking

1. **Match effort to difficulty.** Route simple lookups and formatting to a fast model; reserve reasoning models for multi-step analysis, planning and research.
2. **Measure cost per completed task.** Count total tokens and wall-clock time, including retries.
3. **Test long tool chains.** If an agent needs dozens of sequential calls, test that length specifically.
4. **Check the hosting needs** (GPU memory, serving stack) before choosing open weights for a product.

## Practical implication

Reasoning models are a tool for the hard minority of tasks, not a default. Evaluate them on your own workloads, using cost and time per finished task as the metric rather than price per token.

## Sources

1. [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://arxiv.org/abs/2408.03314), arXiv (Snell, Lee, Xu, Kumar), 2024-08-06
2. [moonshotai/Kimi-K2-Thinking model card](https://huggingface.co/moonshotai/Kimi-K2-Thinking), Hugging Face (Moonshot AI), 2025-11-06
3. [Kimi K2 Thinking](https://simonwillison.net/2025/Nov/6/kimi-k2-thinking), Simon Willison, 2025-11-06

## How Sinthetix helps

**Generative AI & LLM**: Integrate powerful AI models to turn unstructured data into insights and power chatbots.

- RAG systems
- LLM-powered chatbots
- Document analysis

[Explore Generative AI & LLM](https://sinthetix.com/services/generative-ai-llm) | [Talk to us](https://sinthetix.com/contact)

Sinthetix is an AI engineering studio. We ship agents, automation, and internal tools for operators who need results, not reports. [About Sinthetix](https://sinthetix.com/about)