Essays/Agents

Actor and verifier loops

One thing produces, a different thing checks it against evidence the producer did not create, and the check can say no. That is how unwatched work gets delegated.

Nothing in my fleet of agents marks its own work done. One thing produces a result, a different thing checks it against evidence the producer did not create, and the check can say no in a way the rest of the system respects. It is how I delegate work I cannot watch, because, as Agent self-reports are not evidence says, an agent’s report is not evidence.

I use this shape for code, for message delivery and once for a contract. The smallest version is a gate in code.

In our orchestration framework a cheap model produces: it writes the code, runs the test, fills the form. A stronger model reads that output the way a senior reviewer reads a pull request. It has exactly one way to approve, an explicit tool call that is structured and logged. Prose that says “looks good” does not count. Without the call, the result is not accepted.

Two details make it work. The verifier is a different model from the one it checks, because a model reviewing its own output tends to agree with itself, for the same reasons it produced that output. And the two run at different volumes. The producer fires on every file read and every test run. The verifier fires once per meaningful unit of work. Judgment is the expensive part, so I buy it rarely.

What does a verifier look at?

Never the actor’s own summary of what it just did. Beyond that, the evidence depends on the domain.

During a sprint with eight agents on one product, deploys started failing because two agents were renaming exports without committing the files that used them. A coordinator agent froze refactors and renames. It also caught something no self-report would have surfaced. One worker had been retrying the same 429 and 500 errors for over thirty minutes, producing nothing and reporting nothing wrong. The coordinator marked the task blocked and moved the worker to work that needed no running server. It could, because it read the worker’s actual output and had the authority to interrupt.

A tool’s own success signal gets the same treatment. A batch of work once went out to fourteen repositories at once, each in its own terminal session, and the delivery tool reported fourteen successful sends. A separate read-only verifier got the delivery logs. Its job was to check on the machines whether the tasks had been created, whether the sessions were alive, and whether any send had misfired despite returning success. It redid none of the work. It checked the work with different evidence from the tool that made the claim.

The shape holds outside software. For a contract waiting to be signed, I used an agent as an independent second reader, not as a replacement for a lawyer. Its brief was to give a verdict of sign or do not sign as is, list the highest-risk blockers and the exact changes it would request, and say what an earlier human review note had missed or overstated. What made the review independent was the order. It read the contract before the existing note, so its view formed before it saw anyone else’s conclusions.

One question per verifier

Some checks are a chain. A contract for filtering webhooks and routing tasks across three packages went through five stages in under forty-five minutes. Three reviewers read the design in parallel: one for API and usability, one for rollout safety and runaway agents, one for edge cases in the filter semantics. After implementation, three more read the diffs, one per repository, each looking for holes in the test evidence it was given. A seventh judged whether the combined evidence was enough to call the work validated. An eighth reviewed an end-to-end run from a clean directory, checking that replay did not duplicate loops and that private events stayed private. A ninth confirmed that the commits that landed matched the intended scope and nothing extra. None of the nine could edit a file. Each answered one question.

A verifier with a broad scope becomes a second actor making its own unverified claims. Narrow scope is also what lets the pattern grow. For one architecture decision I ran it with twelve reviewers and a rebuttal round, which The twelve-reviewer board describes.

Two ways I have set it wrong

Read-only is a setting, and I have set it wrong. I once asked for sixteen sub-agents, each hunting for bugs and bad logic in a different part of an app and reporting them as tasks without fixing anything. They did exactly that. They were thorough and careful, and no help with the next step: sixteen reports now needed sixteen fixes, from agents that had been told not to touch anything. I stopped them and relaunched them with full access. Read-only is right when the point is an independent check. It is wrong when the bottleneck is getting the fix written.

The other failure is a verifier that fails quietly. One production symptom, the assistant seeming to stop replying, kept returning over five weeks with a different cause each time: a scheduling bug, a rate-limited provider confused by a bare hello, and finally the guardrail layer itself. The guardrail got its own audit. It looked for approvals that get stuck, expired approvals that are accepted anyway, scoped policies that are quietly ignored, and chat state that wedges without an error. A guardrail that blocks silently is worse than one that blocks loudly. So the verifier gets the same suspicion it applies to everything else.

What stays with me

A verifier moves judgment. It does not remove it. Someone still decides what it may check, how narrow its question is, and what it does with something it cannot classify. Deciding what “correct” means for a mechanical check is its own subject, covered in What evals are for. The loops took over the checks that could be specified and handed off. What counts as an acceptable risk in a contract, and whether a question deserves twelve reviewers or two, are still mine.

Back to the essays

Field Notes

A letter from Andrei Hasna on building with AI agents

Subscribe by email

By clicking “Subscribe”, I confirm that I want to get email delivered to my address and agree to Andrei Hasna’s Privacy Policy and Terms.

In each letter

  • New essays on AI agents and the systems behind them
  • What Hasna is building, and why
  • Costs, routing and results from real work
  • What did not work
  • News of the book and the podcast