The first 30 days of an AI pilot: a practical checklist

A pilot is useful only if it ends with a decision. This checklist is our suggested structure for the first 30 days, drawing on published engineering guidance and the NIST risk framework. It is a starting template, not a guarantee, and you should adapt it.

Week 1: scope and baseline

• [ ] Choose one task with a clear owner, a clear input and a checkable output.

• [ ] Record how long it takes and how often it goes wrong today. This is your baseline.

• [ ] Define what "good enough" means in numbers before you start.

• [ ] List the data the system will touch and who approves its use.

Week 2: the simplest build

• [ ] Begin with a single model call plus retrieval of your own documents. Anthropic recommends "the simplest solution possible, and only increasing complexity when needed" ^[1].

• [ ] Use a fixed workflow where the steps are known. Reserve self-directed agents for cases where flexibility is genuinely needed, since they trade latency and cost for performance and can compound errors ^[1].

• [ ] Decide when the system must hand off to a person, and make that path work.

• [ ] Turn on logging of every run so you can review what happened.

Week 3: test on real work

• [ ] Assemble a test set from real past cases. Anthropic's research team started with about 20 queries that reflected real usage, noting small samples are workable when early effects are large ^[2].

• [ ] Score outputs against a written rubric. Anthropic's rubric covered factual accuracy, citation accuracy, completeness, source quality and tool efficiency ^[2].

• [ ] Have a person review a sample too; the same team found human testers caught issues that automated evaluations missed, such as agents favoring SEO-optimized content farms over more authoritative sources ^[2].

• [ ] Keep every failure as a permanent test case.

Week 4: risk review and decision

• [ ] Walk through the risks using the four functions of NIST's AI Risk Management Framework: Govern, Map, Measure and Manage ^[3]. The framework is voluntary ^[3].

• [ ] Compare results to the Week 1 baseline and the "good enough" threshold you set.

• [ ] Estimate running cost at your expected volume, not the pilot volume.

• [ ] Write the decision: go, fix and rerun, or stop. Record why.

Three common traps

1. No baseline. Without it, any result sounds like a success.

2. Testing on invented examples. Real inputs are messier than anything written for a demo.

3. Skipping the stop option. A pilot that cannot end in "no" is a rollout.

Practical implication: Treat day 30 as a decision date with written criteria set on day 1, and stop or narrow the pilot if it has not met them.