# Prompting for production: write the spec, the examples and the test first

- **Published:** 2026-05-08
- **Updated:** 2026-10-12
- **Category:** Harnesses & Tooling
- **Author:** Sinthetix
- **URL:** https://sinthetix.com/blog/prompting-for-production-spec-examples-test-first

> A production prompt is a versioned specification with real examples, a schema and an evaluation set. What the vendor docs say to build before you tune wording.

A prompt typed into a chat window and a prompt that runs ten thousand times a week are different artifacts. The first is a conversation you can correct on the next turn. The second is a component: nobody is watching each output, so its behavior has to be specified, tested and versioned in advance. The vendors' own documentation points the same way, and it starts before any wording is written.

## Start with the test, not the wording

Anthropic's prompt engineering guide assumes you already have "a clear definition of the success criteria for your use case" and "some ways to empirically test against those criteria," and tells readers to establish those first if they do not [1]. OpenAI's guidance is similar: build "tests and evaluation suites that measure prompt behavior" so you can track performance as you iterate or change models [4]. An engineering write-up on evals describes the cheapest level as unit tests, which "are assertions" that run fast enough to execute on every change [6]. For prompts, that means a fixed set of real inputs with expected outputs, run before and after every edit.

## What a production prompt contains

**A task spec.** Role, the inputs it will receive, and the exact output you need. OpenAI describes developer messages as the application developer's instructions, ranked ahead of user messages, much like a function definition versus its arguments [4].

**Examples from real cases.** Anthropic says examples are among the most reliable ways to steer format, tone and structure, and should be relevant (mirroring your actual use case) and diverse (covering edge cases) [2]. OpenAI likewise advises showing a diverse range of possible inputs with the desired outputs [4]. Real, messy cases from your own data do this better than invented tidy ones.

**A schema, not a request for JSON.** OpenAI separates two things: JSON mode ensures valid JSON, but "only Structured Outputs ensure schema adherence," and the docs recommend Structured Outputs where possible [3]. Google's Gemini API also lets you supply a JSON Schema, but warns that although output is syntactically correct, you should "always validate values in your application" [5]. A schema guarantees shape, not truth, so the checks still belong in your code.

**An explicit path for missing information.** OpenAI's examples express an absent value as a nullable type, with a description such as "Null otherwise" [3]. If the schema has no way to say "not present," the model is pushed to fill the field anyway. The prompt should also say what to do when a required input is unreadable. OpenAI also notes that a refusal may not follow your schema, so the application needs a branch for it [3].

**A version.** OpenAI recommends pinning production applications to specific model snapshots so behavior stays consistent [4]. The evals write-up suggests recording metrics alongside the versions of your tests and prompts [6]. Treat the prompt, the schema, the example set and the model identifier as one release.

## Weak prompt versus production prompt

| Aspect | Weak prompt | Production prompt |
| --- | --- | --- |
| Instruction | One paragraph of wishes | Role, inputs and output defined as a spec [4] |
| Examples | None, or invented | Relevant, diverse, drawn from real cases [2][4] |
| Output | "Reply in JSON" | Schema enforced by the API, values validated in code [3][5] |
| Missing data | Model guesses | Nullable fields and a stated rule [3] |
| Change control | Edited in place | Versioned with a pinned model [4][6] |
| Quality check | Read a few outputs | Evaluation set run on every change [1][4][6] |

## A worked example

This is an illustrative skeleton for an invoice-field extraction task, not a tested production prompt.

```text
ROLE: You extract fields from supplier invoices for an accounts team.
INPUT: The text of one invoice.
OUTPUT: One object matching the supplied schema. Fields: invoice_number,
invoice_date, currency, total_amount, purchase_order (nullable).
RULES:
- If a field is not present or not legible, return null. Never infer it.
- Copy amounts exactly as printed. Do not recalculate totals.
- If the text is not an invoice, return null for every field.
EXAMPLES: Five real invoices from our archive, including one with no
purchase order, one with a handwritten total, and one credit note.
EVAL: 40 labeled invoices. A change ships only if no field
regresses against the current version.
```

The last line carries the most weight. The wording of the rules matters less than having a labeled set that tells you whether a rewrite helped. The evals write-up adds that you cannot stop looking at real data [6].

**Key takeaways**

- Define success criteria and a test set before tuning prompt wording [1][4].
- Use examples that are relevant and diverse, taken from real cases [2][4].
- Enforce a schema at the API and still validate values in code [3][5].
- Give every optional field a null path so the model never has to guess [3].
- Pin the model and version the prompt, schema and examples together [4][6].

In practice, a prompt change should go through the same review and test gate as a code change, because the evaluation set is what makes it safe to improve the prompt later.

## Sources

1. [Prompt engineering overview](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/overview), Anthropic (Claude API docs)
2. [Prompting best practices](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices), Anthropic (Claude API docs)
3. [Structured Outputs](https://developers.openai.com/api/docs/guides/structured-outputs), OpenAI (API docs)
4. [Prompt engineering](https://developers.openai.com/api/docs/guides/prompt-engineering), OpenAI (API docs)
5. [Structured output](https://ai.google.dev/gemini-api/docs/structured-output), Google (Gemini API docs)
6. [Your AI Product Needs Evals](https://hamel.dev/blog/posts/evals/), Hamel Husain, 2024-03-29

## How Sinthetix helps

**AI Agents & Automation**: Intelligent agents and automated workflows that reduce manual work and streamline operations.

- Multi-agent systems
- Task automation
- AI-driven decision logic

[Explore AI Agents & Automation](https://sinthetix.com/services/ai-agents-automation) | [Talk to us](https://sinthetix.com/contact)

Sinthetix is an AI engineering studio. We ship agents, automation, and internal tools for operators who need results, not reports. [About Sinthetix](https://sinthetix.com/about)