Pilot to production: a checklist of what changes when an AI system goes live

A pilot proves a system can work for a few people on good days. Production asks whether it keeps working for everyone, on every day, within budget. Use this list as a gate; each item points to a recent source on why it matters.

The checklist

1. A named metric and a baseline. Only 6% of surveyed enterprise leaders report a well-defined ROI framework applied consistently ^[1]. Record the pre-launch baseline and a review date.

2. A model-change plan. Vendors retire and replace models, and replacements can break integrations. Anthropic's Claude Fable 5.1 migration guide, for example, lists API behaviors that now return 400 errors, such as forcing a tool call with any or a named tool, and says thinking blocks are valid only against the prompt and history that preceded them, so code that edits earlier turns can be rejected ^[2]. Pin model versions, test upgrades in staging, and keep a rollback.

3. A tool and connector inventory. List every tool server an agent can call. The Cloud Security Alliance recommends inventorying all MCP servers, including those embedded in developer tools ^[3].

4. Least-privilege credentials and a sandbox. Give each connector its own minimal credentials and run command-executing servers in isolated containers without host credentials ^[3].

5. Data retention confirmed per model. Check retention and processing settings. Anthropic's documentation, for one, states that Fable 5.1 requires 30-day data retention and is not available under zero data retention without express authorization ^[2].

6. A cost-per-task budget. Per-token prices change and hidden usage matters. One write-up of Gemini 3.8 Flash notes that its standard paid rates through December 31, 2026 double on January 1, 2027, and that thinking tokens are billed as output ^[5]. Budget from measured tokens per completed task.

The checklist at a glance

Using it

Assign each item an owner and a date. Any item without an owner means the system is still a pilot, however many users it has.

Practical implication

Treat go-live as a decision with a signed checklist, not a milestone that arrives by default. The items that are cheapest to fix before launch, such as a baseline and an inventory, are the most expensive to reconstruct afterward.