How to run a GenAI security control pilot

Pick one or two groups that use AI heavily, such as engineering and customer support, and the AI tools they use most. Include at least one agent with MCP servers if you have them. Agree a short list of data classes to test: customer identifiers, source code, credentials and one category specific to your business.

Write it down before you start. For example: every AI tool in use is discovered; the agreed data classes are caught in prompts and uploads; the chosen response works as configured; false positives on ordinary work stay low enough that people do not complain; and every block or redaction appears in the log with the person and tool.

Paste realistic test records into the main AI tools and check the response. Upload a document with sensitive data. Use an AI feature inside a SaaS app. Run a coding agent through a task that touches a secret and an MCP server. Try a prompt injection from a document. Then turn the service path off, if the vendor allows it, and see whether traffic fails open or closed.

Compare results against your success list, not against the demo. Note what the vendor could not show and ask for it in writing. If two products tie, the deciding factors are usually operational: one console or two, where logs live, and how much work the policy takes to maintain.

Three things come up again and again. The pilot uses the vendor's demo data instead of your own, so the classifiers look better than they will in production. The pilot group is too friendly, so nobody tries the workarounds real users find in the first week. And nobody owns the decision at the end, so the pilot drifts into a rollout without a clear yes. Avoid all three by writing the test plan with your own data classes, adding at least one sceptical team, and naming the person who signs off before the pilot starts. Keep the vendor's answers to your twelve questions next to the results so the decision rests on evidence.