Agent Evals: the Statistics of pass@k and pass^k

Eight for eight in the meeting

I watched an agent demo go eight for eight. A multi-step research agent — browser, tools, the works — ran its showcase workflow in front of the room and landed it every time. The room was sold. Three weeks into the pilot, the first honest number came back: users were running the same kind of workflow once each, and for about a third of them it fell over.

Nobody faked anything. The meeting saw the agent’s best task under its best conditions — warm caches, pinned dependency versions, a scenario that had survived a week of rehearsal in which the attempts that failed quietly shaped which task got demoed. Attempts are cheap, and you schedule the meeting after a good rehearsal. Production users, meanwhile, ran an average task exactly once, cold. The demo was evidence about whether the agent can ever do it; production measured whether it does it every time. Those are two legitimate statistics — capability and reliability — with a formula-sized gap between them, and agent evals have names for both.

One probability, two exponents

Start from the per-task truth. Every task t has a true per-attempt success probability p_t: the chance a single fresh attempt succeeds, marginalizing over everything that wobbles between attempts — sampling temperature, tool nondeterminism, the harness’s mood. Everything in this post is an aggregate of p_t over your task suite, and the whole family differs only in what you do to it before averaging:

1. pass@1 = E[p_t]. Average single-attempt success. This is the production number when users try once.
2. pass@k = E[1 − (1 − p_t)^k]. The probability that at least one of k attempts succeeds. This is capability, and it is monotonically flattering: it can only grow with k, and any task with nonzero p_t eventually “passes.”
3. pass^k = E[p_t^k]. The probability that all k of k attempts succeed. This is reliability, and it is monotonically brutal: a task the agent gets right 90% of the time is a 43% task at k = 8.

pass@k is the old Codex-paper metric for code generation, where you really could sample k candidates and keep any that passed the tests. pass^k is newer and came from the opposite pressure: Sierra’s τ-bench introduced it because customer-facing agents don’t get k tries — every user gets one draw, and the business feels the consistency, not the best-of. Their headline finding was exactly the divergence: agents that looked respectable at k = 1 fell off a cliff by pass^8, and the follow-up τ²-bench kept the metric front and center. Anthropic’s agent-evals guidance now treats reporting the pair as table stakes. One caveat to keep in your pocket: p_t^k treats the k attempts as independent draws from a stationary task — if your attempts share state (a cache, a polluted sandbox), you’re measuring something else.

Here is the part I want to insist on, because it’s the reason to compute both rather than picking a favorite: the divergence between pass@k and pass^k is itself the measurement. A suite scoring pass@8 = 90% and pass^8 = 16% is not “an agent that works” or “an agent that doesn’t” — it’s an agent that is capable but erratic, and that is a different engineering problem than incapable. Capable-but-erratic points at verification, output selection, retry policy, tighter harnesses, lower temperature — ways to convert capability you already have into reliability. Low pass@k points at capability work: better models, better scaffolds, better tools. One subtraction tells you which roadmap you’re on. The first tab of the explorer below lets you feel how the two curves pull apart as k grows, and how the task mix — not the model — controls the shape.

Estimating pass@k without fooling yourself

You never observe p_t. You run n attempts on task t, count c successes, and estimate. Two tempting estimators, one wasteful and one biased:

The grouping estimator. Chop your n runs into n/k disjoint groups, score each group 1 if it contains a success, average. Unbiased — each group is an honest Bernoulli draw of the pass@k event — but it throws away data: it treats your n runs as n/k coarse observations, ignoring that there are C(n, k) ways to choose k runs from n, not just the n/k you happened to draw as groups.

The plug-in estimator. Set p̂ = c/n and compute 1 − (1 − p̂)^k. This one is biased, and the reason is two sentences of Jensen’s inequality: (1 − p)^k is convex in p, so feeding it a noisy p̂ inflates the failure term on average, which means the plug-in pass@k is systematically too low — and the shortfall grows exactly where you can least afford it, at small n and large k. The mirror argument runs through p^k, also convex, so the plug-in pass^k comes out systematically too high: optimistic, in the one metric whose entire job is pessimism.

The fix has been in the literature since the Codex paper (Chen et al., 2021, appendix). Run n ≥ k attempts, count c successes, and per task compute

pass@k̂ = 1 − C(n−c, k) / C(n, k)

then average over tasks. The reading is mechanical: of the C(n, k) ways to pick k runs out of your n, C(n−c, k) pick only failures — so this is exactly the fraction of k-subsets of your actual runs that contain at least one success. Averaging over all subsets instead of plugging in a point estimate is what kills the Jensen bias; formally it’s a U-statistic, and Han Lee’s derivation is the cleanest walkthrough of why the expectation comes out exact. The mirror estimator falls out of the same logic and deserves to be better known:

pass^k̂ = C(c, k) / C(n, k)

— the fraction of k-subsets that are all successes — is unbiased for p_t^k. One implementation note: compute those binomial coefficients as differences of log-gamma (log-factorials), never as raw factorials — C(200, 10) already ruins a double. The widget’s source does it that way, formulas in the comments.

The error bar nobody computes

Everyone who writes about pass@k explains the estimator. Nobody puts an error bar on it — so let’s. The aggregate pass@k is a sample mean twice over, because there are two nested sources of randomness: which tasks ended up in your suite (between-task), and which runs you happened to draw on each task (within-task — decoding noise, harness noise). An honest standard error must respect both. If you’ve read part 3, you already own the machinery: this is clustering, applied vertically. Each task is a cluster; your n runs are its items; treating the n·T runs as independent Bernoullis is the same fiction as pretending 120 questions from 12 documents were 120 independent draws.

The recipe, in full:

1. Per task, compute the unbiased estimate 1 − C(n−c, k)/C(n, k) from your n runs.
2. Aggregate: the suite pass@k is the mean of the per-task estimates over your T tasks.
3. Bootstrap over tasks: resample T task IDs with replacement, recompute the mean of their per-task estimates, repeat ~1,000 times; the standard deviation of those means is your SE.

Notice what you did not have to do: model the within-task noise explicitly. It rides along inside the per-task estimates — each one is already a noisy measurement of that task’s true pass@k, so resampling tasks propagates both layers at once. That’s the same reason part 3 resampled clusters rather than items.

The recipe raises the budgeting question every agent team actually faces: with a fixed budget of B total runs, do you spend them on more tasks or more runs per task? The variance arithmetic gives a crisp answer. The within-task variance of the unbiased estimator falls steeply as n grows past k — for typical p_t it drops to a third or less of its n = k value by n = 2k, keeps improving usefully to about n = 3k, and then flattens; beyond that you are polishing a number whose uncertainty lives elsewhere. The between-task variance, meanwhile, shrinks only like 1/T and does not care how many times you reran each task. And at realistic suite sizes — the 20–50 tasks that Anthropic’s guidance says agent evals actually have — between-task variance dominates the total almost immediately.

The allocation heuristic: run each task n ≈ 2k–3k times, and spend every remaining run on new tasks. Go past 3k runs per task only when there are no more tasks left to buy.

The second tab of the explorer makes this concrete. At the defaults — B = 1,200 runs, k = 3, a 200-task pool — the SE of the aggregate bottoms out at n = 6, exactly 2k, where the budget saturates the pool. Push runs-per-task to 24 (50 tasks) and the error bar grows by roughly two thirds on the same budget. Drop to n = 3 and you strand half the budget against the pool cap while roughly tripling the within-task noise. The curve’s two bad directions fail differently, but they fail.

The environment is part of the instrument

Everything above quietly assumed p_t holds still while you measure it. Agents earn an extra failure mode here, because an agent task is defined partly by a live environment — and the environment is part of your instrument, whether you enrolled it or not. The mild version is harness flakiness: timeouts, sandbox hiccups, rate limits. That inflates within-task variance — your per-task estimates wobble for reasons that have nothing to do with the model — and if the flakiness correlates with task type (long-horizon tasks time out more), it stops being noise and starts being bias.

The severe version is drift, and it’s sneakier than it sounds. A pattern from a document-research agent eval I worked near: the facts the agent had to find were perfectly stationary — a company’s fiscal-2024 revenue does not change — but the terrain the agent had to traverse to find them was not. Websites restructure, APIs version, data sources restate figures on their own schedule, search results reshuffle. Over months, a frozen question set silently degraded from “hard retrieval and reconciliation” to “trivial lookup”: the score climbed while the agent sat still. That’s the worst kind of eval rot, because it looks exactly like progress. The task your benchmark measures today is not the task it measured at design time, and no re-run of the agent will tell you.

The countermeasures are unglamorous and they work. Version ground truth as (question, answer, environment-snapshot) triples — the snapshot is part of the truth, not an implementation detail. Keep a frozen lane (record-and-replay API responses, cached pages, pinned snapshots) for regression comparisons, and a live canary lane against the real environment; when the two lanes diverge, that divergence is your refresh signal. And measure your own eval’s drift the way you’d measure anything else: freeze the agent, re-run the same set weeks apart against live sources, and put paired differences with CIs on the deltas — part 3’s machinery, pointed at your own instrument. Re-audit on a schedule, not on suspicion.

One last environment trap, because it corrupts the statistics of this post specifically: retry policies change k without telling you. A harness that silently retries timeouts is not measuring pass@1 — it’s measuring pass@(1+retries) with a timeout-shaped censoring rule, and your carefully unbiased estimator is now unbiased for the wrong quantity. Retry semantics are part of the metric’s definition. Pin them, and report them next to k.

Works in the demo, works every time

Reliability engineering has known the difference between “works on the bench” and “works every time” for a century — it’s why MTBF exists and why nobody certifies an aircraft part with a highlight reel. Agent evals didn’t discover the distinction; they compressed it into two exponents on the same probability, which is the kind of gift you should accept. If you take one habit from this post: never report a lone “pass rate” for an agent again. Say which statistic, at which k, from how many runs, with what error bar, against which environment snapshot. That sentence is longer than a number — and it’s the shortest sentence that means anything.

Next, the finale: taking every instrument this series has built — error bars, power, raters, judges, rankings, pass@k — and assembling them into one product eval, end to end.

What this means for your eval

• Report the pair: pass@k for capability, pass^k for reliability, at a declared k. The gap between them is a diagnostic, not an embarrassment — it picks your roadmap.
• Estimate per task with the unbiased combinatorial estimator (n ≥ k runs): never plug c/n into (1−p)^k or p^k — Jensen makes the first pessimistic and the second optimistic.
• Put an error bar on the aggregate by bootstrapping over tasks, not runs. Per-task estimates carry the within-task noise for you; tasks are the sampling unit.
• Budget rule: n ≈ 2k–3k runs per task, everything else into more tasks. Past 3k, buy tasks, not reruns — between-task variance dominates at 20–50 tasks.
• Version ground truth as (question, answer, environment-snapshot) triples; run a frozen regression lane beside a live canary lane, and treat lane divergence as the refresh signal.
• Measure your own eval’s drift on a schedule: agent frozen, same set re-run weeks apart, paired differences with CIs.
• Declare the harness retry policy inside the metric definition. Silent retries change k; a changed k changes everything above.

References & further reading

1. Mark Chen et al., Evaluating Large Language Models Trained on Code (2021) — pass@k and the unbiased estimator (appendix), born with Codex.
2. Han Lee, pass@k as a U-statistic (2025) — the clean derivation of why the combinatorial estimator is unbiased and the plug-in isn’t.
3. Shunyu Yao et al., τ-bench (2024) and τ²-bench (2025) — the pass^k reliability framing for customer-facing agents.
4. Anthropic, Demystifying Evals for AI Agents (2026) — pass@k vs pass^k as standard reporting; the 20–50-task reality.
5. Evan Miller, Adding Error Bars to Evals (2024) — resampling within questions (“epochs”) and the law of total variance behind the two-layer SE.
6. Part 3 of this series — clustered standard errors and the block bootstrap; the horizontal version of this post’s vertical clustering.

Beyond Error Bars · Part 8 of 9 · Previous: Rankings Under Uncertainty · Next: Building a Product Eval, Instrumented

Views mine.