# The first 30 days of an AI pilot: a practical checklist

- **Published:** 2026-05-29
- **Updated:** 2026-10-12
- **Category:** Enterprise Adoption
- **Author:** Sinthetix
- **URL:** https://sinthetix.com/blog/first-30-days-ai-pilot-checklist

> A week-by-week checklist for the first month of an AI pilot: scope, baseline, simplest build, real-data testing, and a clear go or no-go.

A pilot is useful only if it ends with a decision. This checklist is our suggested structure for the first 30 days, drawing on published engineering guidance and the NIST risk framework. It is a starting template, not a guarantee, and you should adapt it.

**Key takeaways**

- Pick one narrow task and measure how it is done today before building anything.
- Start with the simplest design that could work and add complexity only when it demonstrably helps.
- Test on real examples from your own work, starting with a small set.
- End the month with a written go, fix or stop decision.

## Week 1: scope and baseline

- \[ \] Choose one task with a clear owner, a clear input and a checkable output.
- \[ \] Record how long it takes and how often it goes wrong today. This is your baseline.
- \[ \] Define what "good enough" means in numbers before you start.
- \[ \] List the data the system will touch and who approves its use.

## Week 2: the simplest build

- \[ \] Begin with a single model call plus retrieval of your own documents. Anthropic recommends "the simplest solution possible, and only increasing complexity when needed" [1].
- \[ \] Use a fixed workflow where the steps are known. Reserve self-directed agents for cases where flexibility is genuinely needed, since they trade latency and cost for performance and can compound errors [1].
- \[ \] Decide when the system must hand off to a person, and make that path work.
- \[ \] Turn on logging of every run so you can review what happened.

## Week 3: test on real work

- \[ \] Assemble a test set from real past cases. Anthropic's research team started with about 20 queries that reflected real usage, noting small samples are workable when early effects are large [2].
- \[ \] Score outputs against a written rubric. Anthropic's rubric covered factual accuracy, citation accuracy, completeness, source quality and tool efficiency [2].
- \[ \] Have a person review a sample too; the same team found human testers caught issues that automated evaluations missed, such as agents favoring SEO-optimized content farms over more authoritative sources [2].
- \[ \] Keep every failure as a permanent test case.

## Week 4: risk review and decision

- \[ \] Walk through the risks using the four functions of NIST's AI Risk Management Framework: Govern, Map, Measure and Manage [3]. The framework is voluntary [3].
- \[ \] Compare results to the Week 1 baseline and the "good enough" threshold you set.
- \[ \] Estimate running cost at your expected volume, not the pilot volume.
- \[ \] Write the decision: go, fix and rerun, or stop. Record why.

## Three common traps

1. **No baseline.** Without it, any result sounds like a success.
2. **Testing on invented examples.** Real inputs are messier than anything written for a demo.
3. **Skipping the stop option.** A pilot that cannot end in "no" is a rollout.

**Practical implication:** Treat day 30 as a decision date with written criteria set on day 1, and stop or narrow the pilot if it has not met them.

## Sources

1. [Building effective agents](https://www.anthropic.com/engineering/building-effective-agents), Anthropic, 2024-12-19
2. [How we built our multi-agent research system](https://www.anthropic.com/engineering/multi-agent-research-system), Anthropic Engineering, 2025-06-13
3. [AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework), NIST

## How Sinthetix helps

An engineering studio for AI agents, automation, and internal tooling. We build the systems that make operators scale.

[Our services](https://sinthetix.com/services) | [Talk to us](https://sinthetix.com/contact)

Sinthetix is an AI engineering studio. We ship agents, automation, and internal tools for operators who need results, not reports. [About Sinthetix](https://sinthetix.com/about)