
An eval defines what correct means somewhere outside the thing being measured, and checks it mechanically every time something changes. Without one, changing a prompt is a guess, and I learn the result weeks later from a support ticket. With one, I learn it from a number.
The idea first came up as a side remark in the middle of an unrelated feature spec, the kind I describe in Voice-first specs: verification should be automatic, and the user should not have to say “verify” all the time. It is the rule of Agent self-reports are not evidence, one level up.
A fixed set of cases, rerun on every change
Evals are automated tests for an agent, as opposed to a feature. I build thirty to fifty cases, each with an expected outcome, run the agent against all of them, track the pass rate, and rerun the whole set whenever a prompt changes.
Most of the work in building an agent is prompt refinement, edge cases and evals. The code is the smaller part. People skip evals because they feel like overhead standing between them and shipping.
The smallest eval I run is one check on every stop. A hook reads the last user prompt, the last agent message and the list of files that were actually modified, and gives all three to a small model with one question: is this done. If the answer is no, it adds tasks to the list or refuses to let the agent stop. It costs almost nothing, because the checker is the cheapest model available and not the one that did the work. That is the split from Actor and verifier loops at its cheapest.
Correct is what the source says today
The tracker that holds every model’s price across every provider is a tool that measures other things. If it is wrong, nothing built on it can be trusted. So correct was defined as an exact match with each provider’s current documentation. What the code believed did not count, and neither did what a model remembered.
Getting there took roughly a dozen rounds of two auditors working in parallel. Neither was allowed to rely on the other’s answer, and each had to return a flat yes or no. What they caught was specific and unglamorous. One provider’s compact model-listing endpoint was missing a competitor’s real cache-read tier. An aggregator’s copy of one model lacked a long-input pricing tier that the direct API had. A third model had no long-context tier at all, though the tracker assumed one. The same check ran after every fix until both auditors came back clean.
What is the number measured against?
A check is only as good as its reference. An automated sync once diffed the translation catalog against what it believed was the main branch, and filed a task claiming thousands of missing keys per locale. That backlog did not exist. The branch it compared against shared no history with the working tree, so every difference it found was noise. A later run recomputed parity against the real tree, used the merge-base command to prove the two branches had no common ancestor, and closed the phantom task with the proof attached.
A gate can also check the wrong thing. An internal package was once released with 407 TypeScript type errors, an improvement on the previous release’s 458. The release gate checked that the build succeeded. It did not check that the type checker was clean. Shipping with a failing typecheck had become the norm for that package, and it took a separate audit to notice the gap between the stated rule and what the gate enforced. A green check from a gate like that tells me as little as a self-report does.
An eval is a claim about what the number was measured against, and that claim needs checking as much as the result does.
The same holds for causes: reproduce one before naming it. After a large rename, seventeen tests came back red. The easy reading was that the rename had broken them. Instead each failure was reproduced on its own terms. A stack overflow still happened with the rename’s changes stashed. A batch of Python failures traced to a pinned dependency with no build for the current platform. A batch of sandbox failures traced to a missing kernel permission, shown by running the underlying system call by hand and watching it fail with the same error. Two of the seventeen were regressions from the rename. Fifteen predated it, and each has a reproduced cause attached.
The technique can do damage of its own. While testing whether some failures predated a change, the usual move of checking out the old state and rerunning reverted a conflict-marker fix that had just been made, and the broken version was committed on top of the good one. It was caught because the same session then searched for leftover conflict markers and got a count above zero. The tool used to rule out a regression had caused one. The check needs a check as well.
Some checks only count in a browser
A spinner that never clears, messages that arrive out of order, a reply that takes a full minute to a plain hello: none of these show up in a unit test. So browser testing in my fleet runs as eight agents. Each is locked to one section of the app, all work from the same shared plan, and each must drive the page, not read the source and guess. At larger scale a tool lists every link, button, form and input on a page as its own scenario and runs the batch across sandboxes.
A test suite that never opens a browser has already decided, by leaving it out, that nothing a user clicks on needs checking.


