# Twelve questions to ask an AI vendor before you sign

- **Published:** 2026-01-30
- **Updated:** 2026-10-12
- **Category:** Security & Governance
- **Author:** Sinthetix
- **URL:** https://sinthetix.com/blog/twelve-questions-for-an-ai-vendor

> A practical 12-question list for evaluating an AI vendor: data handling, testing, oversight, agent permissions and exit terms.

This week's news gave two reminders of why vendor diligence matters for AI. A popular open-source assistant was reported to have hundreds of instances exposed on the internet, including some with no authentication, and with some user secrets stored in plaintext files [4]. Separately, a new extension lets third-party interface code run inside AI chat tools, and the protocol's own authors caution that it means "running code you didn't write within your MCP host" [5]. Neither is a reason to avoid AI. Both are reasons to ask specific questions first.

**Key takeaways**

- Ask for evidence, such as test results, logs and documents, not assurances.
- Agent products raise new questions about permissions, because they act as well as answer.
- Plan the exit before you commit: data export, model changes and shutdown handling.

## Data

1. **Where does our data go, and who can see it?** Ask for the data flow, including sub-processors and the countries involved.
2. **Is our data used to train or improve models?** Get the answer in written documentation, not the marketing page.
3. **How long is data retained, and can we delete it on request?**

## Quality

4. **How do you test this system, and can we see results on tasks like ours?** Look for a described test set, grading method and repeated trials rather than a single headline score [3].
5. **What happens when the underlying model changes?** Ask whether you get notice, and whether regression tests run before updates [3].
6. **What is the known failure mode?** A credible vendor can name where the product is weak.

## Control

7. **Where does a human approve, review or override?** Reviewers should be able to understand the system's limits, interpret its output, override it and halt it safely.
8. **What can an agent access by default, and can we narrow it?** Ask about file, credential, network and system permissions. The Moltbot reporting describes agents needing access to files, credentials and commands, which is why default scope matters [4].
9. **Is there an audit log of what the system did?** You need a record of actions, not only conversations.

## Risk and governance

10. **Which risk framework do you align to?** NIST's voluntary AI Risk Management Framework, organized around Govern, Map, Measure and Manage, is a common reference point [1].
11. **How do you handle third-party plugins, skills or integrations?** Ask whether they are reviewed, sandboxed and signed. Reporting this week described a skills library that treats all downloaded code as trusted, with no moderation process [4].

## Exit

12. **Can we export our data, prompts and configurations, and what happens if you shut down or change the product?**

## Using the list

Send the questions in writing and compare answers across two or three vendors. Treat vague answers on questions 1, 2, 8 and 11 as a reason to slow down. For most businesses, the practical implication is that a one-page questionnaire answered with evidence will tell you more than a demo.

## Sources

1. [AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework), NIST, 2023-01-26
3. [Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents), Anthropic Engineering, 2026-01-09
4. [Clawdbot sheds skin to become Moltbot, can't slough off security issues](https://www.theregister.com/2026/01/27/clawdbot_moltbot_security_concerns/), The Register, 2026-01-27
5. [Claude can now disgorge interface elements from other apps](https://www.theregister.com/2026/01/26/claude_mcp_apps_arrives), The Register, 2026-01-26

## How Sinthetix helps

An engineering studio for AI agents, automation, and internal tooling. We build the systems that make operators scale.

[Our services](https://sinthetix.com/services) | [Talk to us](https://sinthetix.com/contact)

Sinthetix is an AI engineering studio. We ship agents, automation, and internal tools for operators who need results, not reports. [About Sinthetix](https://sinthetix.com/about)