Evals before agents

01The demo is not evidence

Every troubled agent project we have reviewed shares an origin scene: the demo went well. The agent booked the refund, found the bug, drafted the reply, and the room concluded the remaining work was polish. What the room saw was one sample from a distribution, taken by the people most motivated to see it succeed.

Production does not run a task once. It runs the same class of task hundreds of times a day, and repetition is where agents come apart. tau-bench, Sierra's benchmark of tool-using agents operating under realistic retail and airline policies, measures this directly with pass^k: the probability that all k independent attempts at a task succeed, rather than at least one. GPT-4o with function calling passed 61.2% of retail tasks on a single attempt, and 35.2% of airline tasks. Held to eight consecutive successes on retail, it dropped below 25%.^[1] An agent that clears three of five tasks once clears fewer than one in four when asked to be consistent.

Capability growth is not closing that gap. METR measured the length of software task, in human time, that frontier agents complete at 50% success, and found it has doubled roughly every seven months since 2019.^[2] The buried lede sits in the same data: raise the required success rate to 80% and the horizon shrinks by roughly a factor of five.^[2] Claude Opus 4.5 measured a 50% horizon of about 4 hours 49 minutes (95% CI, 1 hr 49 min to 20 hr 25 min). Its 80% horizon was 27 minutes.^[3]

The demo lives in the optimistic column. Production pays for the other one. A ship decision made on single-run evidence is a decision made on the wrong column.

02You cannot unit-test a model

A unit test is a contract with a deterministic function: fixed input, asserted output, binary verdict, and the verdict means the same thing tomorrow. A language model breaks every clause. The same prompt yields different outputs across runs and across model versions, failures concentrate in corners of the input distribution nobody thought to try, and a passing run says almost nothing about the run you did not take. For these systems the question "does it work" has no answer. The questions "how often, on which distribution, at what confidence" do.

That reframing has consequences. An eval is an experiment. The task set is a sample from a larger population of tasks, the pass rate is an estimate, and the difference between two estimates may be noise. Anthropic's statistical treatment of evals works this out in full: standard errors on eval scores, paired comparisons between models, and power analysis to decide whether a task set is even large enough to detect the effect you care about.^[4] Teams routinely celebrate two-point improvements measured on eighty tasks. Without error bars, that is reading tea leaves.

For agent workflows the honest headline metric is pass^k rather than pass@k. pass@k, at least one success in k attempts, rewards lottery tickets. pass^k, all k attempts succeed, measures whether a user can rely on the system, and it is the number that collapses in public measurements.^[1]

"Refuse to ship without numbers" therefore means something specific. A defined task distribution. A success metric scored the same way every run. Repeated trials with a confidence interval. A threshold an accountable owner signed off before the run, enforced by a machine. It does not mean certainty, and it does not mean proof. It means that when the number moves you notice, and when the number is under the line the release stops.

03Public numbers will not save you

The tempting shortcut is to let someone else do the measuring: pick the model at the top of a public leaderboard and skip the harness. The public record on that strategy is bad in four distinct ways.

Contamination. When Scale AI rebuilt GSM8k from scratch with human writers and retested models on the clone, accuracy dropped by as much as 8% in some model families, and the gap correlated with a model's probability of regenerating original GSM8k items.^[5] Frontier models showed minimal overfitting, so the honest lesson is narrow: distrust small deltas on public benchmarks, not every score.^[5]

Decay. SWE-bench Verified exists because the original benchmark was broken. OpenAI and 93 professional developers audited 1,699 samples and filtered out 68.3% for underspecified problems or unit tests that rejected valid solutions; on the cleaned set GPT-4o scored 33.2% against 16% on the original.^[6] Eighteen months later OpenAI stopped evaluating on Verified itself, reporting that frontier models reproduced benchmark solutions verbatim and that gains on the benchmark no longer reflected real-world coding ability.^[7] The benchmark of record for coding agents had a shelf life under two years.

Gaming. The Leaderboard Illusion documented how Chatbot Arena rankings were structurally gameable: Meta tested 27 private Llama-4 variants and disclosed the best; Google and OpenAI received an estimated 19.2% and 20.4% of all Arena data while 83 open-weight models combined got 29.7%; and access to Arena data yielded relative gains of up to 112% on the Arena distribution, which is learning to win the leaderboard, not the task.^[8]

Judges. Much of the modern leaderboard stack rests on LLM judges, and judges are biased instruments. GPT-4 agreed with human preferences 85% of the time in MT-Bench's pairwise setup, close to the 81% rate at which humans agreed with each other, but that figure is specific to one chat-quality setup and is routinely overgeneralized.^[9] Under answer reordering with default prompts, Claude-v1 stayed consistent only 23.8% of the time and favored the first position in 75% of cases; padding an answer with repetitive rephrasings fooled GPT-3.5 and Claude-v1 as judges in 91.3% of cases.^[9] Reordering alone was enough to make Vicuna-13B beat ChatGPT on 66 of 80 test queries in front of a ChatGPT judge.^[10]

Add cost blindness: accuracy-only leaderboards select for needlessly complex and expensive agents, while simple baselines such as repeated calls to a plain model sit on the Pareto frontier at a fraction of the cost.^[11] Procurement by leaderboard is procurement by vibes with extra steps. Public benchmarks tell you about the field. Only your harness tells you about your task.

04The harness, step by step

Here is the procedure we run at the start of an agent engagement, before the agent exists as more than a spike. None of it requires buying a platform. Most of it is a few hundred lines of code and some discipline.

1. Write the success criteria first. One sentence per task class, stated as an observable end state: the refund exists in the ledger, the build is green, the answer cites the policy document. If you cannot state the end state, you are not ready to build the agent, let alone evaluate it.
2. Collect 20 to 50 tasks from real failures. Support tickets, escalations, transcripts of the workflow the agent will replace. Anthropic's guidance matches our experience: teams delay evals because they imagine needing hundreds of tasks, and 20 to 50 simple tasks drawn from real failures is a strong start. Start now, because evals get harder to build the longer you wait; early on, product requirements translate directly into test cases.^[12]
3. Grade end states with code wherever possible. Database rows, API calls made, files changed, tests passing. Code graders are fast, objective, and free of judge bias. Reserve model-based grading for qualities code cannot check, and human grading for calibrating the judges.^[12]
4. Run every task k times and report pass^k. A k of 4 to 8 is enough to expose consistency collapse.^[1] Report the interval, not the point estimate.^[4]
5. Gate releases in CI. This scales. GitHub runs more than 4,000 offline tests before shipping Copilot model changes, including roughly 100 containerized repositories deliberately broken to test whether the model can restore a green build, and over 1,000 judge-graded technical questions with the judge audited against human reviewers.^[13] Your version can be two orders of magnitude smaller and still catch regressions the demo never will.
6. Put cost and latency on the same dashboard as accuracy. Optimizing accuracy alone selects for expensive agents; select on the Pareto frontier instead.^[11]
7. Feed production back. Every incident becomes a task. Every model upgrade reruns the full set. The harness is an accumulating asset; the agent code around it is comparatively disposable.

The threshold itself is a product decision, not an engineering constant. An internal drafting tool can ship at a pass rate that would be negligent for an agent answering policy questions to customers. What is not negotiable is that the threshold exists, is written down, and is checked by a machine.

05Practice notes on graders

The harness is only as good as its graders, and graders fail in known ways. This is how we assign them.

For LLM judges the debiasing playbook is short, and it is not optional. Present candidate pairs in both orders and average the verdicts, since ordering alone can reverse a ranking.^[10] Give the judge a reference answer when one exists: reference-guided grading cut GPT-4's failure rate on math grading from 14 of 20 to 3 of 20.^[9] Use a different model family for judging than for generating, since the MT-Bench authors also flag possible self-enhancement bias in judges evaluating their own outputs, while noting their study could not settle it.^[9] Then calibrate against human labels before trusting the judge at scale, while remembering what the gold standard is worth: humans agreed with each other 81% of the time in that same setup, so a judge calibrated to humans approximates one noisy rater pool, not truth.^[9]

Two more notes from practice. Grade end states rather than turn-by-turn trajectories; agents find legitimate alternate paths to a goal, and trajectory grading punishes them for it.^[12] And read the transcripts. Anthropic's phrasing matches what we see: you will not know whether your graders work until you read the transcripts and grades from many trials.^[12] Every grader audit we have run has surfaced at least one rubric clause the judge was quietly ignoring.

06The ledger

The cost of skipping measurement rarely presents as an outage. It presents as a customer-facing sentence nobody measured, because nobody was measuring sentences.

In Moffatt v. Air Canada, the airline's website chatbot told a passenger he could apply for a bereavement fare retroactively, contradicting the policy page the bot itself linked to. Before the tribunal, Air Canada argued the chatbot was a separate legal entity responsible for its own actions. The tribunal called that a remarkable submission, held the company responsible for all information on its website, chatbot included, and awarded CA$812.02.^[14] Two honest cautions. The record does not establish what technology the chatbot used, so this is not evidence about language models. And the award was small. Neither caution helps. The precedent is that your bot's words are your words, and the standard of care is taking reasonable care that representations are accurate.^[14] That standard is hard to meet for a system you never measured. It also cuts the other way: no eval score is a legal shield, because liability attaches to the representation, not to your test coverage. Measurement lowers the incident rate; governance, meaning policy grounding, escalation to humans, and disclaimers that actually bind, caps the blast radius when an incident happens anyway.

Measurement also tells you which governance artifacts are load-bearing. In tau-bench, GPT-4o's pass^1 falls from 61.2% on retail to 35.2% on the airline domain with its more demanding policy, and the authors report that function-calling agents are poor at following the rules in the policy documents.^[1] Without a harness, that policy document is a paragraph someone pasted into a prompt. With one, its failure cost shows up as a number.

07Position and limits

Our defaults, applied unless a specific argument moves us off them:

• The harness precedes the agent. Twenty tasks and a code grader on day one beat a platform evaluation in month three.^[12]
• pass^k with error bars is the headline metric for anything customer-facing; single-run pass rates are for spikes.^[1][4]
• Ship thresholds are set per risk tier by an accountable owner, enforced in CI, and never waived verbally.
• Cost sits on the same axis as accuracy.^[11]
• Production traces feed the task set continuously. A frozen harness is a decorative one.

And the limits, because the harness is not a talisman. Static task sets rot: the most carefully cleaned coding benchmark in the field went from reference standard to abandoned in under two years.^[7] Any number that gates releases will be optimized, and some of that optimization will drift away from the task; the Arena story is what that looks like at field scale.^[8] Your gold standard is a noisy rater pool, not ground truth.^[9] So we treat eval integrity the way we treat security posture: an adversarial property that needs maintenance, with the task set refreshed from production and the graders re-audited whenever the models change.

You cannot unit-test a model. You can refuse to ship without numbers. In our engagements the harness comes first and the agent second, and the harness usually outlives the first agent it was built to test. The agent is the part everyone wants to build. The harness is the part that makes the agent worth having.