Inference economics: why the cost of serving models now shapes chip design

Training a frontier model is a large, one-time bill. Serving it is a bill that grows with every user, every agent loop and every long document. That asymmetry is why the newest accelerators and model architectures are increasingly described in terms of serving cost. This post compares three recent data points: Google's inference-focused TPU, AWS's new Trainium3 systems and DeepSeek's sparse attention.

Why serving dominates the long-run bill

Serving cost scales with usage; training cost does not. Agents make the gap wider, because one task can mean dozens of model calls, each re-reading a long context. Two bottlenecks decide the price: raw compute, and the speed at which memory feeds the chip. Long contexts are especially memory-hungry, which is why high-bandwidth memory (HBM) capacity appears so prominently in new chip announcements.

The cost curve has been steep. The 2025 AI Index reports that the cost to run a system at GPT-3.5 level fell more than 280-fold between November 2022 and October 2024, with hardware costs declining 30% a year and energy efficiency improving 40% a year ^[3]. Unit cost is the figure vendors are now racing to cut further.

Google Ironwood: memory and efficiency

Google announced Ironwood on April 9, 2025, and presents it as the first TPU built specifically for inference ^[1]. Its published specifications:

Source: Google ^[1].

Google's comparison against its previous TPU, Trillium, is the telling part: the emphasis is memory and efficiency, not only compute.

AWS Trainium3: scale and efficiency

At re:Invent in early December, AWS introduced Trainium3 UltraServers, built on its first 3nm AI chip, with up to 144 chips in one system. AWS states they deliver up to 4.4x the compute performance and 4x the energy efficiency of Trainium2 UltraServers ^[2]. It also says customers including Anthropic, Karakuri, Metagenomi, NetoAI, Ricoh and Splash Music report training and inference cost reductions of up to 50% ^[2]. Those savings are customer-reported through AWS and not independently audited, and the announcement gives no pricing.

The pattern is the same as at Google: a hyperscaler building custom silicon to lower the cost of the models it hosts, rather than relying only on merchant GPUs.

DeepSeek: cutting cost in the model

Hardware is not the only lever. DeepSeek's V3.2 report describes DeepSeek Sparse Attention, in which an indexer scores earlier tokens and the model attends only to the top-scoring entries, 2,048 key-value tokens per query in the team's setup ^[4]. The authors report an end-to-end speedup in long-context scenarios, showing token cost against position on H800 GPUs rented at an assumed $2 per GPU-hour ^[4]. The paper presents this as a figure rather than a table, so we do not quote its per-token numbers here. The direction is clear: less work per token on the same hardware.

The incumbent context

NVIDIA remains the volume leader. For its third quarter of fiscal 2026 (ended October 26, 2025) it reported data center revenue of $51.2 billion, up 66% from a year earlier ^[5]. Custom accelerators are a growing alternative, not a replacement, at least on the evidence of those numbers.

What buyers should take from this

• Price per task is falling, but unevenly. Hardware and architecture gains reach customers through API prices only when vendors pass them on.

• Long contexts and agent loops are the expensive workloads. Design for shorter contexts and reuse where you can.

• Ask vendors about serving, not training. Cost per request at your expected volume is the number that hits your budget.

The practical implication: choose models and hosting on cost per completed task at your own volume, and revisit that choice every few months as serving costs keep moving.