Ninety-five per cent is a failing grade
An agent is a multiplication of failure probabilities. Why per-step brilliance doesn't survive a loop, and how reliability actually gets engineered.
Here's a piece of arithmetic that explains most of the gap between AI demos and AI products.
Suppose you build a system that performs a task correctly ninety-five per cent of the time. In a demo, this feels superb. It is superb; five years ago it was impossible. Now put that system in a loop and ask it to complete a job that takes ten such steps in a row. The probability the whole job succeeds is 0.95 to the tenth power: about sixty per cent. Twenty steps: thirty-six per cent. Your brilliant system now fails most of the time, and nothing about it got worse. You just asked it to do what agents do, which is chain.
This is the entire agentic reliability problem in one line of maths. An agent, stripped of the mystique, is a model in a loop: it looks at a goal, picks an action, uses a tool, observes what happened, and goes again. The loop is what makes agents powerful, because it lets them do multi-step work rather than answer single questions. And the loop is what makes them treacherous, because it changes the algebra of failure. Errors in a single-shot system are isolated events. Errors in a loop compound, multiplicatively, and a demo only ever shows you the numerator.
It's actually slightly worse than the arithmetic suggests, because the steps aren't independent. An agent's outputs become its own future inputs. When step four contains a small hallucination, step five doesn't encounter it as a suspicious claim from an unreliable stranger; it encounters it as established context, ground truth, part of the record. The error doesn't just persist, it gets built upon, laundered into the foundation of everything after it. One confabulated detail early in a long task can quietly bend the whole trajectory, and by the end nobody, including the agent, can say where it went wrong. Anyone who has watched an error propagate through a hospital record, each subsequent note faithfully copying the mistake forward with growing confidence, will recognise the mechanism exactly. Documents trust documents. So do models.
The seductive answer is that this is temporary: models improve, per-step reliability rises, problem dissolves. Better models genuinely help, and I want them. But look at the shape of the curve. Raise per-step success from ninety-five to ninety-nine per cent, a heroic improvement, and a twenty-step task still fails roughly one time in five. The exponent is the villain, and you don't beat an exponent by nudging the base. You beat it by changing the structure of the game. Which is to say: reliability is not a model property. It's a design property. It lives in the architecture around the model, and it gets built with a toolkit that is, at heart, the same one every safety-critical field converged on long before software existed.
The toolkit looks like this, at field level.
Make steps verifiable. The single biggest upgrade available to any agent is the ability to know whether a step actually worked. Structured outputs that can be checked against a schema. Tool calls with contracts. Results that get validated rather than assumed. A failure that's detected is a retry; a failure that isn't detected is a poisoned context, and everything downstream inherits the poison. Loud failures are a gift. Silence is the enemy.
Shrink the blast radius. Before an agent gets an action, ask the question a hospital asks before granting any clinical privilege: what is the worst this can do, and is it reversible? Reading is cheap. Drafting is cheap. Sending, deleting, prescribing, paying: expensive, and often one-way. Good agentic design sorts actions by reversibility and puts friction, confirmation or a human exactly at the one-way doors, and nowhere else. Friction everywhere is a different failure called uselessness.
Design for the retry, not the happy path. The demo mindset builds the sequence that works. The production mindset assumes every step will eventually fail and asks what happens next: can the agent notice, back up, try differently, or escalate? An agent that can recover from its own errors is worth ten that are marginally less likely to make them, for the same reason a junior doctor who says "I'm not sure, I'll check" is worth ten who never do.
Log everything. You cannot fix what you cannot replay. When a twenty-step run goes wrong, the difference between a mystery and a lesson is whether you can reconstruct exactly what the agent saw, chose, and got back at every step. Observability isn't overhead on the real work. Past a certain level of stakes, it is the real work.
Grant autonomy in slices. This is the one with the highest stakes in my field, and the one the discourse gets most wrong. Autonomy gets talked about as a capability, something a system has or hasn't. In practice, autonomy is a privilege, and it's earned the way privileges are earned everywhere serious: narrowly, with evidence, one evaluated scope at a time. You don't build a system and then decide to trust it. You define one tightly bounded slice of work, evaluate the hell out of that slice, prove the failure modes are understood and the residual risk acceptable, grant autonomy for that slice and nothing else, and then widen, slowly, with the evals leading every step. This is precisely how medicine grants privileges to humans, and it's how the first AI system a US state has let into the prescribing loop, inside a regulatory sandbox, is earning that trust, one supervised decision at a time: not by being impressive in general, but by being proven in particular. It is slow and unglamorous, and it's the only path that deserves to work.
I understand the impatience with all of this. Everyone can feel what agents are about to become, and everyone wants to build the thing from the films: the tireless assistant that just handles it. I want it too; it's a large part of why I do this work. But the ceiling on agents right now is not intelligence. The models are, per step, already astonishing. The ceiling is trust, and trust is not a mood that arrives once the demos get good enough. It's an engineering artefact, assembled from verification, blast-radius control, recovery paths, logs and evals, by people who take the exponent seriously.
Everyone wants to build the robot butler. Fine. Build the seatbelt first. The butler is coming either way; the seatbelt decides whether we get to keep him.
Prefer the full experience? Read this essay in the house. Machine-readable: markdown source.