Plausible is not safe
Why fluency is the most dangerous property of language models, what an evaluation suite actually is, and how a system earns the right to be trusted with medicine.
The most dangerous property of a large language model is not that it makes mistakes. Everything makes mistakes. The dangerous property is the form the mistakes take.
In normal software, failure looks like failure. The page won't load, the number is obviously absurd, the stack trace fills the screen. The system announces its own brokenness, and everyone downstream gets a chance to react. Language models broke this contract. When a language model fails, it fails in perfect grammar, with a confident tone, in the same fluent register as its successes. The wrong answer and the right answer are visually identical. Failure, in these systems, looks like success with better vocabulary.
Most of the time this is a nuisance. In medicine it's a hazard with a specific mechanism, and it's worth spelling the mechanism out. A wrong answer that looks wrong gets checked. A wrong answer that looks right gets acted on, and worse, it closes the question: the person stops looking. I've written on the clinical side of this site about how a normal test result narrows a search without ending it. A plausible AI answer does something more insidious, because it doesn't just narrow the search, it declares the search finished. A model that is confidently wrong is worse than no model at all, because no model at all leaves the uncertainty honestly visible.
So the central question of my working life is not "how do we make the model smarter". It's "how do we know". How do you demonstrate, with evidence rather than adjectives, that a system is safe to let near a decision with real stakes? The answer has a name, it's unglamorous, and almost everything important about medical AI lives inside it: evaluation.
First, clear away the thing people mistake for it. You will have seen headlines about models acing medical licensing exams. This is true, repeatable, and close to meaningless as a safety claim. A benchmark like a licensing exam measures knowledge under exam conditions: clean vignettes, one correct answer, all the relevant information supplied up front, no consequences. Real medicine offers none of those courtesies. Real people present with three overlapping problems and a fourth they're embarrassed to mention. They say "dizzy" and mean any of five different things. They omit the crucial detail not out of malice but because nobody told them it was crucial. Passing the exam tells you the model is good at exams. Patients are not exam questions.
An evaluation suite is the opposite of a benchmark. A benchmark asks "how does this model compare with other models on a standard test". An eval asks "how does my system behave on my task under my conditions, including the ugly ones". It's the difference between a medical student's exam results and a consultant's audit of their actual decisions. You want to hire on the second.
What does a serious eval suite for a clinical system actually contain? At the level of the field, the anatomy is fairly consistent. There's a body of cases with known-good answers, built and argued over by clinicians, because someone has to define what "correct" means before you can measure distance from it. There's a taxonomy of failure, because "wrong" is not one thing: an omitted question, a hallucinated fact, a missed red flag, an unsafe reassurance and an overcautious referral are five different failures with five different harm profiles, and a suite that scores them all as generic "errors" is throwing away exactly the information the verdict rests on. There are graders: humans with rubrics, and increasingly models grading other models, which works far better than it has any right to, but only if you evaluate the grader with the same suspicion you apply to everything else. Turtles all the way down, and every turtle gets audited. There's an adversarial layer: the weird inputs, the 3am messages, the questions phrased in ways no test-writer would choose, because production traffic is an adversary even when no human means it to be. And binding it all together, there's regression discipline: every failure ever found becomes a permanent test, so the system can never quietly reacquire an old flaw. The suite only ratchets in one direction.
Now the part that surprised me, coming into this from clinical work, and the part I suspect surprises everyone eventually. The hardest thing in evaluation is not building the machinery. It's the definitional work the machinery forces on you. To grade an answer about, say, when a headache needs urgent attention, someone has to write down, precisely, what a good answer contains, what a dangerous answer contains, and where the boundary sits. Medicine has spent a century letting that boundary live inside the judgement of individual clinicians: fuzzy, personal, mostly unexamined, and genuinely contested at the margins by reasonable doctors. An eval suite refuses the fuzz. It makes you commit, in writing, in a form precise enough to fail a build. Which means an eval suite is not really a piece of testing infrastructure. It is a statement about what good medicine is, expressed in a language that can be executed. Writing that statement is the most clinically interesting work I have ever done, and I'm including the work I do with a stethoscope.
One more thing the field borrows from medicine, or should. Not all errors are equal, and safety lives in the asymmetries. A screening test is deliberately tuned to catch nearly everything at the cost of false alarms, because the two mistakes carry wildly different prices. Clinical AI needs the same explicit weighting: a missed red flag and an unnecessary referral are both "wrong", one of them is categorically worse, and the system's thresholds should encode that the way triage does. A suite that optimises raw accuracy without weighting by harm is optimising the wrong number with great precision.
I'll end with the question people actually ask me, at dinners, with a slight edge: would you trust it? The honest answer is that I've stopped relating to trust as a feeling. Trust in an engineered system is not a vibe you develop after watching enough good demos. It's a number you can regenerate on demand: this suite, these thousands of cases, this failure taxonomy, these results, rerun after every change, forever. When the evidence is there, trust is just reading. When it isn't, no amount of fluency should buy it, and fluency is the one thing these systems will always have in surplus.
The demo is the exciting part of medical AI. The eval suite is the important part. Nobody screenshots it, and it's where the whole field will be won or lost.
Prefer the full experience? Read this essay in the house. Machine-readable: markdown source.