How to read an AI announcement critically: a worked example

Every model launch arrives with a pricing line and a benchmark table. Both are accurate as far as they go, and both leave things out. Anthropic's October 7 announcement of Claude Haiku 5.5 is a clean, typical example, so we use it to show five checks. ^[1] This is not a criticism of Anthropic; the same checks apply to every vendor.

1. Separate list price from cost per task

Haiku 5.5 lists at $0.10 per million input tokens, against $1.00 for Haiku 4.5. ^[1] That is a 90% drop in list price. Anthropic's own summary is more modest: on average it costs around 75% less to run. ^[1] The page's footnote explains why: the 75% average accounts for a new tokenizer that uses slightly more tokens per task, and the saving is 90% for requests up to 100k tokens but 50% above that. The gap is the point. Tokens used per task can change between generations, so the number to track is cost per completed task on your workload.

2. Read the price tiers

The quoted $0.10 applies to prompts up to 100k tokens, and a higher rate applies above that. ^[1] Long-document workloads may not see the headline figure at all.

3. Check each benchmark's qualifiers

The announcement's table includes OSWorld 2.1, where Haiku 5.5 scores 72.4% against 15.7% for Haiku 4.5. The row is labeled an "offline subset." ^[1] Humanity's Last Exam appears twice, 45.9% without tools and 57.4% with tools. ^[1] A subset, a tool setting or an effort level changes what the number means, so match them to your use case.

4. Notice who is in the comparison

The table compares Haiku 5.5 with Haiku 4.5, OpenAI's GPT-6 Luna and Anthropic's Sonnet 5.5. ^[1] Beating a previous version looks large (Haiku 4.5 scores 0.0% on Terminal-Bench 4.0), while the larger sibling model scores far higher on the same row. ^[1] Both facts are true. Which one matters depends on whether you need the cheapest tier or the strongest result.

5. Look for independence

The figures above come from the vendor's post. Treat them as a shortlist signal, then run the model on a sample of your own tasks, scoring against your own acceptance criteria.

Practical implication

Before acting on a launch, write down the task you care about, the cost per completed task you can afford, and a small test set. A model that wins the announcement but loses on your test set is not a better model for you.