Twelve questions to ask an AI vendor before you sign

This week's news gave two reminders of why vendor diligence matters for AI. A popular open-source assistant was reported to have hundreds of instances exposed on the internet, including some with no authentication, and with some user secrets stored in plaintext files ^[4]. Separately, a new extension lets third-party interface code run inside AI chat tools, and the protocol's own authors caution that it means "running code you didn't write within your MCP host" ^[5]. Neither is a reason to avoid AI. Both are reasons to ask specific questions first.

Data

1. Where does our data go, and who can see it? Ask for the data flow, including sub-processors and the countries involved.

2. Is our data used to train or improve models? Get the answer in written documentation, not the marketing page.

3. How long is data retained, and can we delete it on request?

Quality

1. How do you test this system, and can we see results on tasks like ours? Look for a described test set, grading method and repeated trials rather than a single headline score ^[3].

2. What happens when the underlying model changes? Ask whether you get notice, and whether regression tests run before updates ^[3].

3. What is the known failure mode? A credible vendor can name where the product is weak.

Control

1. Where does a human approve, review or override? Reviewers should be able to understand the system's limits, interpret its output, override it and halt it safely.

2. What can an agent access by default, and can we narrow it? Ask about file, credential, network and system permissions. The Moltbot reporting describes agents needing access to files, credentials and commands, which is why default scope matters ^[4].

3. Is there an audit log of what the system did? You need a record of actions, not only conversations.

Risk and governance

1. Which risk framework do you align to? NIST's voluntary AI Risk Management Framework, organized around Govern, Map, Measure and Manage, is a common reference point ^[1].

2. How do you handle third-party plugins, skills or integrations? Ask whether they are reviewed, sandboxed and signed. Reporting this week described a skills library that treats all downloaded code as trusted, with no moderation process ^[4].

Exit

1. Can we export our data, prompts and configurations, and what happens if you shut down or change the product?

Using the list

Send the questions in writing and compare answers across two or three vendors. Treat vague answers on questions 1, 2, 8 and 11 as a reason to slow down. For most businesses, the practical implication is that a one-page questionnaire answered with evidence will tell you more than a demo.