Evaluations are not tests
A test tells you whether the code did what you wrote. An evaluation tells you whether the system is still good enough. Confusing the two produces suites that pass while the product gets worse.
Engineering teams arrive at AI systems fluent in testing, and reasonably assume evaluation is testing with fuzzier assertions. The vocabulary encourages it: both run in CI, both go green or red, both are described as coverage. The assumption is wrong in a way that takes about six months to become obvious, usually when a suite that has been green throughout ships a regression everyone can see in the product.
The difference that matters
A test asks a closed question with a knowable answer: given this input, does the function return this output? It is deterministic, it is cheap, and a passing test is a proof. When a test fails, something is broken.
An evaluation asks an open question with no fixed answer: across this distribution of inputs, is the system’s behaviour good enough for what we are using it for? It is statistical, it costs real money, and a passing eval is evidence, not proof. When an eval score drops, something may be broken, or the sample may have shifted, or the judge may have drifted, or the threshold may have been wrong.
Three consequences follow, and each one contradicts a testing habit that is otherwise a virtue.
A single run is a sample, not a result. One green run means little. What you need is a distribution and a trend — the same suite, run repeatedly, tracked over releases. Teams that treat the first green as pass/fail throw away the only signal that actually predicts regressions.
100% is a warning sign. In testing, a suite that always passes is a suite doing its job. In evaluation, it means the cases are too easy, and the suite has stopped being able to detect the thing you built it for. Healthy suites sit somewhere uncomfortable and move.
The suite is a living asset, not a ratchet. Test suites accumulate — you rarely delete a test. Eval sets need active curation: retiring cases the system has fully absorbed, and continuously adding the failures found in production. An eval set that has not changed in two quarters is measuring last year’s system.
“A suite that always passes has stopped being a measurement and become a ritual.”
What earns its place in a suite
The useful mental model is three layers, and they are not interchangeable.
Assertions. The genuinely deterministic properties: valid JSON, schema conformance, no PII in the output, required disclaimer present, tool called with well-formed arguments. These are tests, they are cheap, and they belong in CI on every commit. Most teams under-invest here and over-invest in the layer above, because assertions feel too simple to be the answer to an AI problem. A surprising share of production incidents are assertion failures.
Graded cases. A curated set with known-good answers, scored by a rubric — sometimes by a model, sometimes by an exact or fuzzy match. This is the layer that tracks quality over time. It needs enough cases to be statistically meaningful, and it needs to be built out of real traffic rather than invented examples, because invented examples encode your assumptions about what users do.
Human review. A sampled slice looked at by someone who knows the domain. Expensive, slow, and the only layer that can find a failure mode you did not think to encode. It is also the layer that calibrates the one above it: if your model-as-judge disagrees with your domain expert, the judge is wrong until proven otherwise.
The common mistake is trying to push everything down into the graded layer because it is the automatable one. Assertions catch the cheap failures for almost nothing. Human review catches the novel ones. The middle layer is where you watch the trend, not where you find surprises.
Judges need evaluating too
Model-as-judge is what makes graded evaluation affordable at scale, and it introduces a measurement instrument that can itself drift. Treated carelessly it converts an unknown quality problem into an unknown measurement problem, which is worse because it looks like progress.
Two disciplines make it trustworthy. Hold out a set of human-labelled cases and periodically measure agreement between your judge and your humans — if that agreement number is not tracked, your quality numbers mean nothing in particular. And pin the judge’s model version and prompt, treating a change to either as a change to the instrument: re-baseline, and never compare scores across the boundary. A silent judge upgrade will move every number in your suite and look exactly like a quality change.
Where the suite should live
The organisational placement matters as much as the design. Evals built inside the delivery team and never seen by anyone else are useful for iteration and worthless for assurance — they are marking their own homework, which second line will spot immediately and correctly.
The pattern that holds up: the delivery team owns the eval suite and runs it on every change; the standard for what a suite must contain at each risk tier is owned centrally; and the results land somewhere risk and audit can read without asking. That last part is what converts a development tool into a control, and it is almost entirely a plumbing problem — a durable location, a stable format, and results that are not manually curated before anyone sees them.
Get that right and the eval suite stops being the thing you run before a release. It becomes the continuous evidence that the system is still operating inside the envelope you were approved for — which is the only form of assurance that survives a system that changes weekly.