# Pilot to production: a checklist of what changes when an AI system goes live

- **Published:** 2026-09-25
- **Updated:** 2026-10-12
- **Category:** Security & Governance
- **Author:** Sinthetix
- **URL:** https://sinthetix.com/blog/ai-pilot-to-production-checklist

> Six checks to clear before an AI pilot goes live, each drawn from a 2026 source on measurement, model changes, tool security, data retention and cost.

A pilot proves a system can work for a few people on good days. Production asks whether it keeps working for everyone, on every day, within budget. Use this list as a gate; each item points to a recent source on why it matters.

**Key takeaways**

- Six checks cover measurement, change control, access, data retention and cost; each can be answered yes or no.
- Each check below is a yes-or-no question that someone named can answer.

## The checklist

1. **A named metric and a baseline.** Only 6% of surveyed enterprise leaders report a well-defined ROI framework applied consistently [1]. Record the pre-launch baseline and a review date.
2. **A model-change plan.** Vendors retire and replace models, and replacements can break integrations. Anthropic's Claude Fable 5.1 migration guide, for example, lists API behaviors that now return 400 errors, such as forcing a tool call with `any` or a named tool, and says thinking blocks are valid only against the prompt and history that preceded them, so code that edits earlier turns can be rejected [2]. Pin model versions, test upgrades in staging, and keep a rollback.
3. **A tool and connector inventory.** List every tool server an agent can call. The Cloud Security Alliance recommends inventorying all MCP servers, including those embedded in developer tools [3].
4. **Least-privilege credentials and a sandbox.** Give each connector its own minimal credentials and run command-executing servers in isolated containers without host credentials [3].
5. **Data retention confirmed per model.** Check retention and processing settings. Anthropic's documentation, for one, states that Fable 5.1 requires 30-day data retention and is not available under zero data retention without express authorization [2].
6. **A cost-per-task budget.** Per-token prices change and hidden usage matters. One write-up of Gemini 3.8 Flash notes that its standard paid rates through December 31, 2026 double on January 1, 2027, and that thinking tokens are billed as output [5]. Budget from measured tokens per completed task.

## The checklist at a glance

| # | Check | Owner | Review date | Done |
| --- | --- | --- | --- | --- |
| 1 | Named metric and pre-launch baseline |  |  |  |
| 2 | Model-change plan: pinned versions, staging test, rollback |  |  |  |
| 3 | Tool and connector inventory |  |  |  |
| 4 | Least-privilege credentials and sandbox |  |  |  |
| 5 | Data retention and processing settings confirmed per model |  |  |  |
| 6 | Cost-per-task budget from measured tokens |  |  |  |

## Using it

Assign each item an owner and a date. Any item without an owner means the system is still a pilot, however many users it has.

## Practical implication

Treat go-live as a decision with a signed checklist, not a milestone that arrives by default. The items that are cheapest to fix before launch, such as a baseline and an inventory, are the most expensive to reconstruct afterward.

## Sources

1. [Measuring AI ROI: The Next Big Conundrum for Enterprises](https://battery.com/blog/measuring-ai-roi-the-next-big-conundrum-for-enterprises), Battery Ventures, 2026-07-17
2. [Claude Fable 5.1 and Claude Mythos 5.1 migration guide](https://platform.claude.com/docs/en/models/fable-5-1/migration-guide), Anthropic (Claude Platform Docs), 2026-09-01
3. [MCP Security Crisis: Systemic Design Flaws in AI Agent Infrastructure](https://labs.cloudsecurityalliance.org/research/csa-research-note-mcp-security-crisis-20260504-csa-styled/), Cloud Security Alliance AI Safety Initiative, 2026-05-04
5. [Gemini 3.8 Flash pricing: every rate, the hidden cost, and the catch](https://eesel.ai/blog/gemini-3-8-flash-pricing), eesel AI, 2026-09-08

## How Sinthetix helps

An engineering studio for AI agents, automation, and internal tooling. We build the systems that make operators scale.

[Our services](https://sinthetix.com/services) | [Talk to us](https://sinthetix.com/contact)

Sinthetix is an AI engineering studio. We ship agents, automation, and internal tools for operators who need results, not reports. [About Sinthetix](https://sinthetix.com/about)