As the use of artificial intelligence (AI) grows in IT management, governance is becoming a key concern. In ITSM.tools’ 2026 survey, respondents stated that “Risk, governance, or compliance concerns” is a top barrier to AI adoption, second only to poor data.
IT organizations may reach for familiar control mechanisms: identity, permissions, auditability, approval, and clearly defined roles and ownership. Those controls remain relevant, but they also rest on the long-standing notion that acceptable behavior can be defined in advance. In deterministic systems, we assign permissions, describe what should happen, and govern the system against that model. Even traditional automation usually fits neatly into this structure: even for complicated workflows, we have paths that are usually designed and known.
With Agentic AI, however, we are deploying something that this governance model does not fit.
Governance of modern agent-to-agent systems
In modern agent-to-agent systems, agents’ paths are not enumerated in advance, and their actions are constructed rather than predetermined. As a result, the behavior we get is not the behavior that we designed. Arguably, it’s one of the key reasons for deploying them, but it leaves the governance question wide open.
How do we keep enough control to trust these systems, while leaving them enough freedom to find better answers than the ones we already had?
Some things absolutely should be deterministic
Of course, complexity is no excuse for fuzzy controls around authority.
I recently wrote a corporate blog about AI guardrails in enterprise service management, describing the need to put behavioral controls around the model itself. No agent should be able to be talked by its human users into assuming approval authority it does not have, or persuaded to change data that ought to be immutable. Role-based access control still decides what an agent can reach and act on, and separate guardrails govern how it behaves within that scope.
However, this approach hits limits when we encounter broader AI solutions in complex, multi-agent environments. We may be confident that a single agent cannot manufacture an approval, but when an agent is progressively deciding how to explore and resolve a complex issue, often in collaboration with other autonomous AI agents, that is another matter.
An incident agent may involve an observability agent because it spots a relationship nobody modeled. The observability agent brings in vulnerability or change context, which pulls in another specialist. A human specialist adds business context and the interpretation shifts again. By that point, the useful behavior sits in no single agent. It is developing between them, each contributing a different view and changing the context available to the others. Each of these contributions was constructed rather than retrieved, so each downstream agent reasons from something nobody enumerated up front.
Complex services already depend on adaptation
Complexity and emergence have been recurring themes in my own work for years. Mark Smalley and I worked on the topic in “ITIL 4: High-velocity IT”, in response to the ever-increasing complexity of enterprise technology, and the resulting issues around prediction and control.
In this and other articles, I’ve argued that the behaviors of complex IT systems can develop through interactions that cannot be predicted in advance, even when they make considerably more sense retrospectively. This principle is well defined in complexity thinking as “emergence,” and my application of it to IT management was not novel. In fact, it drew on the seminal work of some very innovative thinkers.
In his influential 1998 paper “How Complex Systems Fail”, Richard Cook describes systems holding shifting combinations of problems and degraded conditions, with practitioners continuously compensating for what is happening around them. He calls human practitioners the “adaptable element.”
Dave Snowden, with Cynefin, built on Cook’s principles, developing specific sense-making approaches for complicated and complex issues, the latter differentiated from the former by a lack of deterministic cause and effect. Cynefin promoted the idea of executing multiple, parallel experiments when troubleshooting complex issues, amplifying actions that produced positive outcomes and limiting those that did the opposite.
It fascinates me now to see state-of-the-art Agentic AI systems performing their work in a similar way: developing theses, probing and testing, reworking, and repeating, interacting with tools and other agents, and building a path to a resolution that was not determinable at the start of the work.
In a conversation with my colleague Erhan Giral, BMC Helix’s senior data scientist, he described one of AI’s strengths as compressing knowledge. An experienced engineer may know an extraordinary amount. No human expert, though, can hold every service relationship, months of telemetry, previous incidents, vulnerabilities, changes, knowledge, and policy in working memory at once. Nor can instinct for complex path development, honed by their work, easily be disseminated to other humans.
An AI agent, however, does not have this limitation, and that’s key to its power. It can access the full context of the organization’s data and retain what it learns in shared memory.
But the point is not that AI has better memory. The point is what the memory does. An agentic system meets a novel problem. Agents and professionals develop a response nobody prescribed and that response changes knowledge, operating practice, the relationships between agents and perhaps the service itself. The next problem then arrives in a different situation, partly created by how the last one was resolved.
So the governance question is bigger than whether a particular action is acceptable. It becomes what kind of system all these acceptable actions, retained experiences, and developing patterns are helping to produce – and what is the cost of not letting this happen?
Creative freedom must be allowed
A diagram I used in a 2020 personal blog about exploring emergence and IT service management (ITSM) is useful here. It comes from Regina Frei and Giovanna Di Marzo Serugendo’s 2011 paper, Advances in Complexity Engineering (International Journal of Bio-Inspired Computation, 3(4), 2011). Frei and Di Marzo explored the trade-off between tight boundaries and “creative freedom” in systems. The challenge is familiar: how can we constrain behavior enough to keep a system within acceptable limits, but leave it enough freedom to find adaptive solutions?
Desired, Allowed, and Possible areas of system behavior, after Frei and Di Marzo Serugendo (2011).
Their model separates Desired, Allowed, and Possible areas of behavior. “Desired” represents what the designer sets out as the specification for the system. “Allowed” provides a wider acceptable range into which emergent behaviors can take the system, while “Possible” includes behavior outside those limits.
“Allowed” is where evolution has somewhere to happen. Frei and Di Marzo Serugendo state the trade-off plainly: more freedom can improve overall performance, but it reduces the designer’s control over what the system will actually do.
Snowden gets to the same place from a different direction. Constraints in the complex domain are enabling rather than prescriptive. They create the conditions for something useful to emerge without specifying what it should be. The design question becomes how much room to leave.
Hand that space to AI agents and it gets much richer. An incident agent may find that involving a specialist earlier produces better outcomes. Several agents may work out a productive way to handle a class of problems that nobody thought to encode. As long as the hard boundaries hold, novelty is not a governance failure. It may be a sign the agentic model is earning its place.
Governance may itself become agentic
I already argued that we need to be brave enough to relax control selectively. Hard action-level guardrails need to stay hard, but applying this thinking to every acceptable decision and interaction would quietly collapse the Allowed space back into Desired. We would have a bigger specification. It would still be a specification.
The alternative is to make governance active rather than prescriptive. Instead of defining every good path beforehand, governance would watch what is developing and respond: dampening damaging patterns, amplifying useful ones, and intervening when the direction of travel moves outside what the organization is prepared to accept. This is Snowden’s amplify and dampen, pointed at behavior nobody designed rather than at experiments set up in the flow of solving an issue. It’s compatible with Frei and Di Marzo’s governance principles, because we can limit the boundaries of experimentation.
However, when AI agents are doing the work, scale is unbounded by human work capacity, and the activity happens at the speed of machines, not people.
The inherent unpredictability of LLMs aggravates that issue. We can give the same agent the same input twice, in the same conditions, at the same time, and it can respond differently. An AI model samples from a distribution rather than looking up an answer, so there is no single behavior to observe. A conventional test observes an input, checks the output, and enables a confident inference about what will happen next time. That inference is not available with LLMs.
None of this is settled, and it is worth asking what a governance method has to do before it can work with Frei’s model at all. It needs to tell Allowed from outside the limits. It also needs to recognize behavior inside Allowed that is better than what was specified, without holding a specification of what better looks like.
Most IT organizations will reach first for the methods they already own, and those largely fail the second test.
Judging behavior without prescribing the path
At least one technique is worth considering. LLM-as-judge uses a second LLM to watch the behavior of the tested one. The “judge” observes how the conversation progresses, evaluating whether it followed an acceptable path to its conclusion rather than a predefined one.
Watching an AI agent handle a case well tells you it can, not that it will. But this is not a defect. The sampling is part of what lets an AI agent construct a response rather than retrieve one, so removing it would remove the capability we deployed it for. However, as a result, a single acceptable observation proves very little, and assessment has to be continuous and repeated rather than point-in-time. What it produces is a picture of how the system behaves over many runs, not a verdict on a case.
Applied to Frei’s model, an AI-based judge could help assess where behavior sits, at the speed required. It can assess if a response is holding inside Desired, working usefully in Allowed, or creeping beyond the edge of what the organization will accept. Non-determinism is accounted for by the nature of testing. The same input may produce a different route each time, but running the same scenarios repeatedly produces a distribution rather than a verdict, and that distribution can be plotted onto the same three regions.
The result is closer to a heat map than a pass mark: this is where this AI agent, or this group of AI agents, actually lands across many runs. The result is statistical, showing how often an AI agent works in a certain way. Repeatable runs let you assess performance before or after a prompt change or tuning action. Hence, AI agent optimization comes from data, not hunches. The more runs, the more likely it is that an occasional, potentially disastrous deviation into an undesired state will be detected, rather than missed because no single test encountered this low-probability anomaly. The “judge” LLM can also sit completely independent of the tested system, leaving the latter unaware of the observation and reducing the risk that evaluation influences its behavior.
Governing what develops
IT has spent decades getting better at governing technology by specifying what it should and should not do. We should keep that discipline wherever the boundary genuinely can be specified, particularly around authority, security, and consequential actions.
We now need something both different and more scalable. Complex services will keep producing circumstances we did not anticipate, and AI will do so far faster than before. AI agents with broad context and real autonomy may develop novel responses, and their interactions with experienced professionals may produce useful patterns nobody designed. Accurate, always-deployable memory will make those patterns reach much further than they used to. What emerges today becomes part of the conditions from which tomorrow’s behavior develops.
The autonomy that makes Agentic AI hard to govern is the same autonomy that makes it worth doing. If we reduce governance to keeping everything inside the behavior we already designed, we may end up governing away much of the reason for using AI agents in the first place.
The challenge is to become good enough at governance to safely allow more of what we did not design.
AI Governance FAQs
Controls like identity, permissions, and approval workflows assume acceptable behavior can be defined in advance, which fits deterministic systems and even most traditional automation. Agentic AI systems construct their paths rather than follow predetermined ones, so the behavior that results isn’t the behavior anyone designed, which is part of why deploying AI agents is worth doing in the first place.
The model, drawn from Regina Frei and Giovanna Di Marzo Serugendo’s 2011 research, separates what a system’s designer specifies (Desired), a wider acceptable range where emergent behavior can develop (Allowed), and everything else outside those limits (Possible). Leaving more room in Allowed can improve performance, but it costs the designer some control over what the system actually does.
It’s a technique where a second LLM observes whether a tested AI agent followed an acceptable path to its conclusion, rather than checking against one predefined route. Because a single run proves little on its own, assessment has to run repeatedly across many cases, producing a picture of how a system behaves overall rather than a pass or fail on one case.
A conventional test gives an input, checks the output, and lets you infer what will happen next time. LLMs sample from a distribution instead of retrieving a fixed answer, so the same AI agent can respond differently to the same input under the same conditions, and that inference isn’t available.
Multi-agent systems produce useful behavior that develops between AI agents rather than living in any single one, drawing on ideas from Richard Cook’s work on complex systems and Dave Snowden’s Cynefin framework. Constraints work best here as enabling boundaries rather than a prescribed path, creating the conditions for something useful to develop without specifying what it has to be.
Authority, security, and consequential actions. Role-based access control still governs what an AI agent can reach and act on, and hard guardrails around approval authority and immutable data need to stay hard, no matter how much freedom the rest of the system is given.
Jon Stevens-Hall
Jon Stevens-Hall is an IT service management practitioner, product lead, author, and industry contributor who has worked in ITSM since 1997. He has contributed to industry thinking across ITSM, DevOps, and AI, and was an author of ITIL 4. He currently works at BMC Helix as a ServiceOps Evangelist and Principal Solutions Marketing Manager.
