Compositional Opacity
Why Observability is Hard in Agent Systems
For years, a central critique of artificial intelligence has been opacity.
Large neural networks, especially large language models (LLMs), are described as black boxes: we can see the inputs and the outputs, but why the model arrives at a particular decision is difficult to inspect. The internal reasoning is inaccessible, non-intuitive, or simply too complex to follow. For smaller neural networks, tools like Shap can help, but for larger models, interpretability diminishes.
The black box critique is typically aimed at the model itself.
But we don’t just ship models. We ship systems. And the systems have a different, and sometimes more dangerous, kind of opacity, one that does not live inside any single component.
I call it compositional opacity.
The shift from models to systems
The critique inherits a simple picture: a user sends a prompt, a model returns an answer, and the hard question is what happened inside the model.
User → LLM → Output
Agent systems sometimes are more complex than that. They often look more like this:
User → Planner → Memory → Retrieval → Tool/API → Executor → Critic → Output
Or:
Supervisor → Research agent → Code agent → Validation agent → Approval layer → Deployment pipeline
Each part contributes to every action the system takes.
The brain of the system is still a model. But the thing we experience as an assistant or an agent is the sum of many parts, and each part contributes to every action the system takes. This changes how we need to think about risk and observability.
Three layers of opacity
It helps to separate three things we tend to lump together as “opacity.”
Layer 1: model opacity. A single network produces outputs we cannot fully explain. The classic black box. What the model knows lives in billions of weights rather than in rules anyone wrote down, and asking the model to explain itself yields a plausible narrative rather than a faithful trace of the computation. Of the three layers, this one has the most tooling aimed at it: mechanistic interpretability (Anthropic’s circuit-tracing work, for example) is steadily mapping what individual features and circuits do, helped by the fact that a model is a fixed artifact you can hold still and probe.
Layer 2: pipeline opacity. A fixed chain of prompts, retrieval steps, and tools makes it unclear where an error originated. This is a familiar problem in retrieval-augmented-generation (RAG) systems. A wrong answer might come from a bad retrieval, a mangled chunk, a prompt that buried the instruction, or the model itself, and nothing in the output tells you which. But the control flow is static: the chain runs the same steps in the same order every time, so the ordinary remedies apply. Log every hop, replay the failing input, and bisect until you find the step that went wrong.
Layer 3: compositional opacity. A dynamic system of agents, memory, tools, and feedback loops, in which the control flow itself is decided at runtime by the components, and produces behavior that cannot be reliably explained at the system level. There is no fixed chain to replay, because the chain was improvised: which components ran, in what order, reading and writing what state, was itself an output of the run. A log can record the path taken, but the same input on the next run can take a different path, so what you reconstruct afterward explains one trajectory rather than the system.
What separates Layer 2 from Layer 3 is autonomy of control flow, not complexity. A pipeline runs a path you laid down in advance. A complex agentic system chooses its own path, revises it, loops, and spawns sub-agents, so the path is not knowable beforehand and often not reconstructable afterward.
Defining it
Compositional opacity: a property of AI systems in which the interaction of multiple components (models, tools, agents, or memory) produces behavior whose causal pathways cannot be reliably inferred, even when each component is individually interpretable.
Composition makes traceability difficult, even when every part is easy to reason about, so just making the parts more interpretable doesn’t solve the problem. You can inspect a complete log and see what the LLM output was for a given input, but it’s harder to know if that output would have been different if some other component three jumps away had behaved differently.
Manufacturing has a name for what composition does to per-step quality: rolled throughput yield, the yield of a whole process, computed by multiplying the yields of its steps. A line where every step runs at 95 percent sounds healthy. Ten steps in series deliver about 60 percent of units defect-free, and twenty steps deliver roughly a third. Reliability engineering runs the same multiplication for components in series.
Explainability compounds the same way. Suppose interpretability can account for 95 percent of what a single component does, a generous figure for a modern model. Compose twenty components and the share of runs in which every step is explained falls to about a third, with the unexplained step landing somewhere different on each run. So even before composition destroys anything structurally, the arithmetic alone is against you: per-component interpretability gains do not accumulate into system explainability, because composition taxes them multiplicatively.
The phenomenon has cousins. Security has the confused-deputy problem and its chained variants, distributed systems have emergent misbehavior, and safety engineering has interaction failures. Each names a symptom in one domain. Compositional opacity names the general property underneath them: legibility lost to composition itself.
What this is, and what it is not
It is tempting to call this emergence and reach for something mystical. Resist that. Very little of it is strong emergence in the philosophical sense of genuinely irreducible system-level properties. The mechanics are mundane, and more tractable:
combinatorial state: the number of reachable system states compounds very quickly,
hidden external state: agents write files, move money, and change databases, so understanding the system now requires understanding the world it touched,
non-reproducibility: every layer makes a probabilistic choice, so the same system fails differently on different runs.
That is distributed-systems complexity on a probabilistic substrate. If we wave at emergence, we excuse ourselves from engineering it.
Why safe parts do not make a safe whole
You cannot certify the parts and infer the whole.
The most dangerous assumption in agentic engineering is this: If every component is safe, the system is safe.
It is false, and not as a matter of bad luck. It is false for three structural reasons.
First, actions compose into effects no component authorized. Read a record, then send an email: each step is permitted, the composition is exfiltration. A dozen trades, each under the limit, compose into a position no one approved. Authority laundering needs no broken component.
Second, one component’s trusted output is the next component’s unverified input. Safety properties that held at a boundary do not survive being carried across it. A planner’s “suggestion” becomes an executor’s instruction.
Third, shared state and feedback loops result in conditions no single component created. A critic reinforces a planner’s flawed premise, memory surfaces it again next turn, and the loop converges on something none of them would have chosen alone.
Safety is not a compositional property. You cannot certify the parts and infer the whole.
Composition defeats control, not just understanding
My own first pass at compositional opacity framed it as an explainability problem. That is only half of it.
The other half is a security problem. Composition does not only make a system hard to understand after the fact. It makes the system hard to constrain in the first place.
A guard that inspects one action at a time cannot catch a harm that exists only across actions. Researchers have already turned this into an attack. Jones, Dragan, and Steinhardt (2024) showed that an adversary can combine two individually safe models, sending the hard-but-benign subtasks to an aligned frontier model and the easy-but-disallowed ones to a weaker, misaligned model, and produce results neither model would generate alone. Vulnerable-code generation rose from under 3 percent for either model by itself to 43 percent for the combination. No single model was broken. The harm lived in the composition.
The same logic powers authority stacking, structuring, and trajectory attacks: assemble a forbidden outcome out of individually permitted parts. A control plane that reasons one action at a time won’t see the assembly, which is the attack.
So compositional opacity has two faces. It defeats inspection, because you cannot see what the composed system did, and it defeats policy enforcement, because you cannot stop a chain of actions if each action is approved. The two reinforce each other. The opacity is exactly what lets the laundering go unnoticed.
The four risks
Attribution failure. When something goes wrong, you cannot say which component caused it. A component that fails loudly is the easy case. The hard case is a plausible output, subtly wrong several handoffs earlier, laundered into confident downstream work. By the time the damage surfaces, every component behaved reasonably given its input, so the blame has nowhere to land.
Compositional alignment failure. Individually aligned components interact into a globally misaligned outcome. Alignment does not compose any more than safety does. Each component optimizes a local objective: the planner wants a plan that survives the critic, the critic wants objections it can defend, retrieval wants relevance. The composition optimizes whatever those pressures add up to, which nobody specified and nobody checks. The decomposition attack in the previous section is this risk done deliberately; the accidental version needs no adversary at all.
Debugging collapse. Traditional debugging assumes linear, reproducible causality. A dynamic, recursive, probabilistic system offers neither. Reproduce, bisect, fix: each standard move assumes the failure will happen the same way when you run it again. Here the rerun can succeed on the same input, because a different sample sent the run down a different path. Programmers call a bug that disappears when you try to observe it a Heisenbug; in a compositional system that is the default case rather than the exception.
Governance failure, which is the one worth pushing on. The reflex is to ask who is responsible when an autonomous system acts unpredictably (the model vendor, the orchestration framework, the developer, the tool provider, or the deployer) and to conclude there may be no clear answer.
That conclusion is a choice, and the wrong one. Responsibility diffuses only when we decline to assign it. The defensible rule is that accountability follows design-time control: whoever composed the system and put it into operation owns its behavior, whether or not the causal trace can be reconstructed afterward. Opacity is not a liability shield. If anything, deciding to deploy a system you cannot fully explain is itself the accountable act.
The wrong place to stop
The analysis so far invites a bleak stopping point: the system is unknowable, so feel dread.
That’s not a good place to stop. Agent systems can be built safely. It just requires intention and some good engineering practice.
You do not have to fully understand a composed system and all its possible permutations to keep it safe. You just have to bound it.
Consider a data center I once worked in that used a man-trap at the entrance. You badge in, the inner door seals behind you, and a scale weighs you. There is no restroom inside. If you weigh more leaving than you did entering (so you must be carrying something out that you did not carry in), the outer door does not open. The system makes no attempt to understand what you did in there, which racks you touched, or what you carried where. It does not care. It enforces one invariant at the boundary: you leave as you came. The opacity of your visit is total, and completely irrelevant.
That is the shape of the answer. Stop trying to win the shell game of tracking the object through every shuffle. Watch the door.
For agentic systems that means three commitments, none of which require interpreting the agent.
Enforce in the substrate, not in the agent’s reasoning. A deterministic gate that the agent cannot argue its way past decides what actually executes. The agent proposes, deterministic code disposes. The model’s reasoning can inform that gate as an advisory signal, but it can never be the thing that authorizes the action.
Guard the trajectory, not only the isolated action. Because harm composes across steps, the boundary has to carry a little state: cumulative budgets for blast radius, a contamination mark that spreads down any path that has touched protected data and clears only at an explicit sanitization step, and preconditions that a later action re-checks against live state, so that “disable the backups, then delete the data” fails at the delete. Compose the guards to match the composed threats.
Accept the limits honestly. A conservation invariant like the scale works for things that are conserved: money, inventory, physical state. It does nothing for information, which can be copied without changing the weight in the room. For data you fall back to tainting the path and sanitizing at the boundary. The right constraint is re-derived from what you are actually protecting, not assumed once and reused everywhere.
What this does not buy you is forensics. Bounding the system keeps it safe without your understanding it, but when you do need to reconstruct what happened, you need a record built for it: an external, tamper-evident log of what each action was and what authorized it, captured outside the reasoning layer. The boundary keeps you safe. The log is how you later learn why.
Observability that survives composition
The boundary answers the safety question and leaves the observability question standing. A man-trap prevents the theft; it names no thief. Attribution failure and debugging collapse are still waiting when you come back to ask what happened, and the yield math from earlier says why the intuitive response, explain each step a little better, cannot work: explanation is lossy, and lossy quantities compound away.
Recording does not share that fate. A trace ID either propagates across a hop or it does not. A logged call either captured its input or it did not. Per-step yield on deterministic capture can sit at effectively 100 percent, and anything at 100 percent survives multiplication. That is the design rule: put observability in the deterministic substrate, where it composes, rather than in per-step explanation, where it decays. Most of the machinery already exists, because distributed systems hit this problem first.
Trace every hop. Distributed tracing (Google’s Dapper paper started the lineage; OpenTelemetry is the standard today) propagates a single trace ID through every component a request touches, so a run can be reconstructed as a tree of spans. OpenTelemetry now defines semantic conventions for generative AI, covering spans for model calls, tool invocations, and agent operations, and tools such as LangSmith, Langfuse, and Arize Phoenix apply the pattern to agents: every model call, tool call, and memory access becomes a span with a parent. This is the direct attack on attribution failure. You may never know why the planner chose the path. You will know which components touched the outcome, in what order, holding what data.
Record the random choices. Non-reproducibility comes from probabilistic sampling, so capture each sample at the moment it happens. Event sourcing treats an append-only log of events as the system of record; applied here, that means logging every model call’s full input, output, and sampling parameters. The system will never run the same way twice, but a recorded trajectory can be replayed deterministically: step through it, inspect intermediate state, test whether a fix would have changed the outcome. Record-and-replay debugging of nondeterministic programs is established practice (Mozilla’s rr debugger is the classic example). This is the attack on debugging collapse.
Make handoffs machine-checkable. One component’s trusted output becomes the next component’s unverified input most easily when the handoff is free prose. Structured, schema-validated messages between components, a plan as a typed object rather than a paragraph, can be checked at the boundary when they cross and diffed later when you are reconstructing. Design by contract is the old name for this discipline, and it transfers.
Chain the decision log. The tamper-evident record from the previous section has established forms: hash-chained append-only logs of the kind Certificate Transparency runs in public, and the W3C PROV vocabulary for recording which agent derived which artifact from which source. This is what makes the governance position enforceable. Accountability can follow design-time control only if the record of what the deployed system did survives the deployer’s incentive to lose it.
Evaluate whole trajectories. Harms that exist only across steps can only be caught by evaluation that reads across steps. The traces above are the input: run evaluators and anomaly detectors over completed trajectories, asking whether the sequence assembles a forbidden outcome out of permitted parts and whether contaminated data reached an exit, rather than only scoring single responses. Composed threats need composed detection, on the evaluation side as much as at the boundary.
None of this restores explainability in the interpretability sense. The model’s reasoning stays dark, and Layer 3 stays Layer 3. What it sustains is legibility of behavior: who did what, with what data, authorized by what, in what order. And that is what incident response, debugging, and accountability actually run on.
The stance
Everything above, in an essay that keeps telling you to prepare for the forensic investigation, could read as pessimism about agents. It is worth saying plainly that I am not pessimistic about AI. Agent systems are powerful, they will solve real problems for us, and I want them built. Like anything else powerful, they have to be designed well, and designing well starts with stating the design conditions the way a structural engineer states a load assumption. The engineer designs for loads beyond the estimate, and nobody calls that pessimism about buildings. The considerations an engineer makes in design are the reasons the building does not fall over.
So state the conditions and design under them:
Assume agents will be opaque.
Assume their reasoning will be unreliable.
Assume composition will defeat interpretability.
Assume alignment will not compose.
Then build systems that stay safe anyway. Constrain the substrate. Watch the boundary. Let understanding come later, if it comes at all.
Design things so they do not hurt people, and use them as designed
Powerful technology has earned its rules this way before. Decades of boiler explosions preceded the first ASME Boiler and Pressure Vessel Code in 1915, and we now have shelves of standards and whole agencies whose working premise fits in a sentence: design things so they do not hurt people, and use them as designed. AI is at the beginning of that arc. It is still possible to put an unsafely designed agent system into production, which says something about the age of the field rather than about the technology. The norms will come. Whether they arrive as real controls or as checkboxes depends on what exists before the mandate does, because rules written in a hurry after a public loss tend to require whatever is easiest to check. Naming compositional opacity is one early entry in the vocabulary those rules will need.
The argument for good design, even in the absence of generally imposed rules of design, is clear and we have a pretty good example from history. In March 1928, the tugs T.J. Hooper and Montrose were towing coal barges up the Atlantic coast when a gale off the New Jersey shore sank the barges and their cargo. Radio receivers carrying weather forecasts existed and were cheap; the tugs sailed without them, and so did much of the industry. In The T.J. Hooper (1932), the judge held the tugs unseaworthy anyway: “a whole calling may have unduly lagged in the adoption of new and available devices,” and “there are precautions so imperative that even their universal disregard will not excuse their omission.” The boats floated and could pull the load. That did not make their owners ready to be doing business in weather they had chosen not to hear about.
Every agent system in production today sails in the weather this essay describes. The radio is the observability of the earlier section: the trace, the recorded trajectory, the chained decision log. It exists, and it is cheap next to the cargo. The fact that much of the industry sails without it will not excuse anyone, and it should not.
We spent a decade trying to see inside the model. The next decade is about accepting that we often cannot see inside the system, and engineering so that it does not have to matter.
A system whose exact outputs we know we cannot reliably predict at least keeps us careful. The dangerous ones are the systems we believe we understand, just because we understand the parts. Agents will be wrong sometimes. When our systems are designed so that it’s safe for the agent to be wrong, the overall system gets a lot safer for us to use.
Images generated with Google Gemini