What is AI agent governance?

AI agent governance is how a company stays in control of AI agents: who can build them, what they can see, what they can do, and how you prove afterwards what happened. It matters more for agents than for chatbots, because an agent does not only talk. It reads files, calls APIs and changes records.

This page gives a practical checklist you can apply whatever tools you use.

Why do AI agents need governance?

A chatbot’s worst case is usually a bad answer. An agent’s worst case is a bad action: an e-mail to the wrong customer, a deleted record, or confidential data shown to someone who should not see it.

The OWASP Top 10 for LLM Applications names “excessive agency” as a core risk: an agent with more tools, permissions or autonomy than its job needs. Governance is how you keep agency in proportion.

There is also a people problem. If official tools are slow to arrive, staff use personal AI accounts with company data, which is known as shadow AI. Good governance should make the safe path the easy one, not only add rules.

What does an AI agent governance checklist include?

Area	Question to answer	What good looks like
Identity	Who is using the agent, and who built it?	Single sign-on, named users, roles
Permissions	What can the agent see for this person?	The asker’s own access applied to every search
Tools	What can the agent do?	Only the tools the job needs; read-only where possible
Approvals	Which actions need a person?	Writes and destructive actions approved; conditions for amounts
Data protection	What sensitive data reaches the model?	Masking before the model reads it
Audit	Can you trace every run?	Runs, approvals and settings changes logged with who did it
Model choice	Which model, from which provider, under whose contract?	Approved models, company-owned keys
Cost	Who pays, and what stops a runaway?	Usage visibility per agent, balance or budget checks
Lifecycle	How do changes go live?	Draft, test, publish a version, roll back
Isolation	Can one team’s agent reach another team’s data?	Separate projects with their own knowledge and keys

The sections below walk through each area.

How should identity and permissions work for AI agents?

Identity comes first. Every conversation should be tied to a real, signed-in person, ideally through your single sign-on (SSO) provider so leavers lose access the moment they are disabled. Roles should separate people who only use agents from people who build them.

Permissions decide what an agent can see for a given person. The safest pattern is that an agent never sees more than the person asking could open themselves. That means applying the asker’s access rules inside the search, not filtering results afterwards. Doing this well is harder than it looks: in our benchmark on 100,000 documents with synthetic, team-shaped permissions, a hybrid search filtered afterwards returned nothing for 64% of searches when the asker could see 10% of the documents, and for 99% when they could see 1%. The benchmark post explains why. See permission-aware answers for the idea in more detail.

Public channels, such as a website chat, need a separate rule: anonymous visitors should only reach knowledge meant for the public.

How do you control what AI agents can do?

Tools are where agents create real-world effects. Four controls help:

1. Least privilege. Give each agent only the tools its job needs. A support agent that answers questions does not need a delete tool.
2. Read-only by default. Database and file access should start read-only, with writes turned on deliberately and limited to specific tables or folders.
3. Approvals by risk. Let reads run, ask before writes, and always ask before deletes, payments or access changes. See human in the loop AI.
4. Admin floors. A security team should be able to set an approval rule that individual builders cannot switch off.

Remember that no control removes prompt injection. Assume the agent can be tricked, and make sure a tricked agent cannot do much damage.

How do you protect sensitive data in AI agents?

• Mask values such as card numbers, national ID numbers and API keys before the model reads them, and show the real value only to people who need it.
• Keep data where it belongs. Decide where documents, embeddings and conversation files are stored and for how long.
• Choose providers carefully. Using your own model keys means your contract with the model provider applies to your data.

Be honest about limits: pattern-based masking finds known formats, not every sensitive detail. More in AI data leakage.

How do you audit and monitor AI agents?

Monitoring answers two questions: what happened, and is anything going wrong now?

• Log every run with the tools called, their inputs and results, approvals and who was involved.
• Log settings changes too. Many incidents start with a permission or approval rule quietly loosened.
• Watch usage and cost by agent and model, so a looping agent or a sudden spike is visible.
• Review refused and expired approvals, and failed tool calls. They often show an agent trying to do things outside its job.

The NIST AI Risk Management Framework organises this under four functions (Govern, Map, Measure and Manage), which is a useful way to explain your programme to auditors.

How should AI agents move from draft to production?

Treat an agent like software:

1. Draft the instructions, knowledge and tools.
2. Test in a private chat with realistic and adversarial questions.
3. Publish a named version so people use a known configuration.
4. Roll back quickly if a change behaves badly.
5. Retire agents nobody uses, and remove their tool access.

How promptev handles AI agent governance

• Workspace roles (owner, admin, editor, viewer) and permission-by-permission project roles; viewers only talk to agents, and disabling a member removes access at once. SSO works with any OpenID Connect provider, and SSO groups can become access rules.
• Projects keep their own knowledge, tools, connectors, members, API keys and masking, isolated from each other.
• The asker’s identity and groups are applied inside the search for Google Drive, SharePoint, OneDrive and Dropbox; masking with built-in detectors and your own patterns runs before the model reads documents and tool results.
• Per-tool approvals, with “Ask if” conditions on registered tools and admin rules that agents cannot weaken.
• An audit trail of every run, approval and settings change, with who did it (and the app, for MCP).
• Agents use your own model keys; you test a draft, publish a version and can roll back. A usage dashboard shows model spend by provider, model and agent. See governance.