What mid-market organizations need to know about governing AI agents

A practical guide for teams that adopted agents before they wrote the rules.

Most organizations did not hold a launch review for their AI agents. A team enabled an assistant. A developer wired an agent into a repository. A business unit bought a tool with an API key that never made it into an inventory. The capability arrived before the governance did.

This picture is close to universal. Nearly every organization now reports some unsanctioned AI use, yet only about a third have an AI governance policy in place.[1] Meanwhile, most security leaders name AI as the single biggest driver of change in their field this year.[2]

Agents spread the way most productivity tools spread: because they are immediately useful to the person who turns them on. The governance question is not how to roll that back. It is what you need in place so the usefulness does not quietly become your exposure.

A note on terms. In this article, an agent is any AI system that can take actions using credentials and tools, rather than only producing text for a person to read. That covers coding assistants with repository access, workflow automations that call models, and agent features that arrive switched on inside licensed SaaS products. We use “agent” for all of them.

An agent is not a chatbot

The distinction matters more than it sounds, and it drives everything else in this article.

A chatbot produces text. A person reads the text and decides what to do with it. The human is the actor, and the human is the control.

An agent takes actions. To be useful it needs credentials, network reach, and permission to touch real systems. It reads files, calls APIs, writes to production, sends mail, opens tickets, queries databases, and spawns further agents. The person who asked gets an outcome, not a transcript to review.

That shift moves the risk from “what did the model say” to “what is the agent allowed to do, and who is accountable when it does something nobody intended.” A prompt that reaches an agent with write access to a payment system is a different event from the same prompt reaching a chat window. The model is rarely the sensitive asset. The credentials, the access, and the tools are.

There is a second, quieter shift. Agents act under identities that are not people. Across the average enterprise, non-human identities (service accounts, API keys, workloads) outnumber human users by roughly 45 to 1, and by as much as 144 to 1 in cloud-native environments.[3] Most of those are not AI agents, but agents add to the pile and inherit its weak ownership. In one survey, 92% of organizations said their existing identity tooling cannot manage AI and non-human identity risk, and about half could not name an owner for their agent identities.[4] When an agent acts, the usual question, “whose account was that?”, frequently has no clean answer.

Five foundations

Governance discussions tend to start with policy. That is the wrong end. Policy about systems you cannot enumerate is untestable, and it becomes the document nobody follows. Start with visibility and ownership, because every control you build afterwards depends on them.

1. An inventory, including the agents nobody bought. What exists, where it runs, who enabled it, and what it was for. This includes sanctioned deployments, pilots that never closed, agents inside licensed SaaS that arrived switched on by default, and the developer’s local agent with a personal API key. An inventory you believe is complete and is not is worse than a partial one you know is partial.

At minimum, record these fields for every agent:

Name and the platform it runs on
Named owner (a person, not a team)
Business purpose
Date enabled, and whether it was reviewed
Credentials it holds and when they expire
Systems it can write to
Data classes it can read
Level of autonomy (reads only, proposes and waits, acts on its own)

A spreadsheet is enough to start. The goal is a record you can sort by risk, not a tool purchase.

2. An access map: what each agent can actually reach. Not the intended scope, but the effective permissions: the tokens it holds, the systems it can write to, the data it can read, and whatever else those credentials unlock. The single most consequential finding in most first assessments is the gap between what the agent was designed to touch and what it can touch.

3. Identity and credential hygiene. How many identities do you have, which agents own them, how long do the secrets live, and who can revoke them? Long-lived, broadly scoped credentials are the biggest single multiplier of risk. If an agent’s token never expires and can reach everything, any prompt that reaches the agent effectively inherits that access. Aim for credentials that expire in hours, are scoped to one task, and can be revoked by a named person in minutes.

4. Runtime behavior: what agents actually do, not what they were meant to do. Log agent actions with enough context to separate normal from anomalous: which agent called which tool, on whose behalf, how often, and whether that pattern has occurred before. This is where most organizations are thinnest. It is also the capability that turns every other control from a statement of intent into something you can verify.

5. Accountability: a named owner, decision rights, and a way to stop it. Every agent needs a person who owns it, a defined boundary it may not cross, and a kill switch that has been tested rather than assumed. “The platform team owns the agents” is not accountability. A name attached to a specific agent, with authority to turn it off, is.

The cost of getting this wrong is not hypothetical. IBM’s 2025 breach study found that among organizations that reported a breach of an AI model or application, 97% said they lacked proper AI access controls. One organization in five had already suffered a breach involving shadow AI, at an average of $670,000 more than organizations with little or none.[5]

Governance people will actually follow

Most first attempts at agent governance fail in one of two directions. Either the rules are so tight that teams route around them, with a shared API key, a personal account, or a side project that never appears on the inventory. Or the rules are so vague that nothing changes. Both end in the same place: no visibility and no accountability, with a policy document on the wall. The 37% of organizations that have a policy are not necessarily better off if nobody can verify it is followed.

Four principles are a sound place to start. Each is adapted from long-standing application security practice and from current guidance such as the OWASP work on agentic and LLM applications.

Match privilege to autonomy. An agent that reads and summarizes needs less than an agent that writes. An agent that proposes and waits needs less than one that acts autonomously. Grant access per level of autonomy, not per tool.

Use short-lived, scoped credentials. Tokens that expire and rotate turn a leaked secret from an incident into a nuisance. As a starting rule, anything that can write to a production system should hold credentials that last hours, not months, with 24 hours as an upper bound. Workload identity features in your cloud provider and a secrets manager with automatic rotation will get most teams there. This is one of the few controls where the effort is modest and the risk reduction is large.

Put a human gate in front of irreversible actions. Start with this list: deleting data, moving money, sending external mail, changing production configuration, and creating new identities. Agents should be able to propose these freely and execute them only with approval. The approval can be a ticket, a chat approval, or an in-tool confirmation, but it must be tied to the specific action, not a blanket “allow.” Watch the approval rate. If reviewers approve nearly everything within seconds, the gate has become a rubber stamp. The friction belongs at the point of no return, not everywhere.

Treat everything an agent reads as untrusted. Retrieved documents, web content, ticket text, email, and other agents’ output can all carry instructions. An agent that treats ingested content as instructions has handed its tool access to whoever wrote that content. Three controls make this concrete:

Separate agents that read untrusted content from agents that hold write access.
Restrict where an agent may send data (an allowlist of outbound destinations).
Never give one agent both “reads inbound email” and “sends outbound email.”

This is the same instinct that gave us input validation and least privilege in applications. The surface is new; the principle is not.

A 30/60/90 plan that does not require a new department

Mid-market organizations do not have a dedicated AI governance function, and they should not build one before they have visibility. The work fits into processes that already exist: identity management, change management, incident response, and vendor review.

Here is how the phases map to the five foundations: days 1 to 30 cover foundations 1 and 2, days 31 to 60 cover foundations 3 and 5, and days 61 to 90 cover foundation 4 and the policy.

The thresholds below are starting points. Adjust them to your risk tolerance, but pick numbers so you can tell when a phase is finished.

Days 1 to 30: see it

Lead: IT or identity, with security. Each business unit lead confirms what their team uses.

Build the inventory and access map for the agents you know about, then go looking for the ones you do not. Where to look, and what to do there:

SaaS admin consoles and your identity provider: export the list of OAuth app grants and enabled AI features.
Cloud IAM: list service accounts and their last-used dates; flag any created for AI workloads.
Code repositories: scan for API key patterns and AI provider variable names.
Browser extensions: pull the managed extension inventory and flag AI assistants.
Expense and procurement records: search for AI tool subscriptions and API usage charges.

Expect the second half of this list to take longer than the first.

Done when: every known agent has a record with an owner field filled in, and you have a written list of the places you could not yet check.

Days 31 to 60: contain the worst of it

Lead: security and identity, with each agent’s owner. Legal or compliance joins for anything touching regulated data.

Rank what you found by what an agent could reach, not by how interesting the agent is. Then give every agent one of three dispositions:

Sanction and register: it has a legitimate purpose, an owner, and a scope. Keep it.
Restrict and scope: it is useful but over-privileged. Cut its access to what it needs.
Retire: nobody can say why it exists, or its risk outweighs its value. Revoke its credentials.

Kill the credentials nobody needs. Scope the ones with excessive reach. Assign a named owner to every agent still running, and confirm each has a defined boundary and a tested off switch. A kill switch test means revoking the agent’s credential, confirming the agent actually stops, timing how long that took, and repeating the test every quarter.

Done when: every running agent has a disposition and a named owner, and no agent holds a credential older than your chosen limit with write access to production.

Days 61 to 90: make it repeatable

Lead: security, with change management and procurement.

Write the policy you can now enforce, in short and specific terms. Turn on runtime logging where it is missing. Add agent provisioning and decommissioning to the same review that handles new accounts and new vendors, because an agent added without review is the next shadow deployment. Then decide the uncomfortable part: what your agents must never be able to do, regardless of who asks.

If you need something to show an auditor, insurer, or board, map the policy to a recognized framework. The NIST AI Risk Management Framework, ISO/IEC 42001, and OWASP’s guidance on agentic applications all cover this ground, and referencing one makes the policy easier to defend and easier to start.

Done when: the policy is published, logging covers every agent that can write to a production system, and new agents cannot be provisioned without the review.

If you only do one thing

Do the inventory and the access map.

Everything else (least privilege, approval gates, monitoring, incident response for agent-driven events) depends on knowing what exists and what it can reach. Runtime logging is the control that makes the rest verifiable, but you cannot log an agent you do not know exists. Start with visibility, then add logging as soon as the inventory tells you where it matters most.

Teams that start with policy spend months writing rules they cannot verify. Teams that start with visibility spend the same months finding out that the situation is both more manageable and more surprising than they expected. [Replace with a concrete example from your own assessments, for instance: “In the assessments we run, the typical finding is a handful of agents holding most of the access, and a fix list shorter than the fear suggested.”]

The organizations handling this well are not the ones with the most sophisticated tooling. They are the ones that mapped the problem early, gave every agent an owner, and kept the ability to stop one. That is achievable in a mid-market team. It is mostly a matter of deciding to look.

If you would rather not start from a blank spreadsheet, Illuminate Risk LLC helps mid-market organizations map what their AI agents can reach, govern what they can do, and make the risk visible before it becomes an incident. A 25-minute discovery call is the place to start, with no proposal until we both know it is a fit.

Sources

[1] Cloud Security Alliance, Shadow AI and the Enterprise Visibility Crisis (May 2026): 98% of organizations report unsanctioned AI use; 86% lack visibility into how data flows to and from AI tools; only 37% have an AI governance policy in place. https://labs.cloudsecurityalliance.org/research/shadow-ai-enterprise-visibility-governance-v1-csa-styled/

[2] World Economic Forum, Global Cybersecurity Outlook 2026: 94% of respondents name AI as the most significant driver of change in cybersecurity; 87% identify AI-related vulnerabilities as the fastest-growing cyber risk. https://www.weforum.org/publications/global-cybersecurity-outlook-2026/

[3] Cloud Security Alliance, The Non-Human Identity Governance Vacuum (May 2026), citing Entro Security: non-human identities outnumber humans 45:1 on average, up to 144:1 in cloud-native environments. https://labs.cloudsecurityalliance.org/wp-content/uploads/2026/05/CSA_whitepaper_nonhuman_identity_agentic_ai_governance_v1-csa-styled.pdf

[4] Cloud Security Alliance 2026, AI Agent Identity Crisis: Standards Emerge as Enterprises Lag survey on AI and non-human identity risk: 92% report existing IAM tooling cannot manage AI/NHI risk; about half report no clear ownership of agent identities. https://labs.cloudsecurityalliance.org/wp-content/uploads/2026/03/CSA_research_note_okta_ai_agent_iam_framework_enterprise_gap_20260318-csa-styled.pdf

[5] IBM, Cost of a Data Breach Report 2025: of organizations that reported a breach of an AI model or application, 97% lacked proper AI access controls; 20% of studied organizations suffered a breach involving shadow AI, adding an average of USD 670,000 in breach cost. https://www.ibm.com/think/x-force/2025-cost-of-a-data-breach-navigating-ai