Error Bars, Applied

Five models tied for first

I once watched a leaderboard for coding agents come together — a hard one, about 120 tasks, frontier models scoring in the 20–30% range, with respectable-looking gaps of two or three points between neighbors. Then someone computed honest 95% confidence intervals. At those sample sizes and score levels, the intervals came out around ±8 points. Overlaid on the scores, they collapsed the podium: five models, statistically tied for first place.

The statistics were not the interesting part. The interesting part was the room. Nobody disputed the math; the discussion immediately became “we can’t publish five number-ones” — and it was a reasonable discussion, because a leaderboard that shrugs is genuinely less useful to its readers than one that ranks. The proposals on the table (rank by raw score and show the intervals? report each model’s range of plausible ranks?) weren’t statistical disagreements. They were disagreements about what the leaderboard was for — which, if you read part 2, is exactly where every eval argument ends up.

Hold that story; we’ll resolve it at the end. First, the part you can now look up elsewhere.

The part you can now look up

As of a couple of years ago, the following is (finally) mainstream: an eval score is a sample mean. Your eval set is a draw from some universe of possible questions, so the score has a standard error — for binary pass/fail scoring, SE = √(p(1−p)/n) — and it deserves a confidence interval, not a bare number. When you compare two models, comparing their separate intervals is needlessly weak: run both on the same questions and analyze the paired differences, which typically shrinks the comparison’s uncertainty dramatically (part 4 quantifies just how much). None of this requires anything beyond intro statistics; it just requires remembering that it applies to evals.

I’m compressing this because it’s well covered now: Evan Miller’s Adding Error Bars to Evals is the paper that started the wave, Cameron Wolfe wrote the long tutorial with code, and statsforevals.com is the running reference. Go there for derivations.

One tooling note before we go further: the plumbing for this already exists. Hugging Face’s evaluate library and lighteval will hand you bootstrap confidence intervals today — the feature is shipped; the docs just don’t insist you use it. If your eval harness reports a bare mean, that’s a configuration choice, not a limitation.

That’s the solved part. The rest of this post is where real evals leave the textbook.

Which variance are you even measuring?

“The score moved” has at least three different causes, and they need different instruments:

1. Sampling variance. Which questions ended up in your eval set. This is what the confidence interval from the previous section measures — how much the score would move if you’d drawn a different set of questions from the same universe.
2. Decoding variance. Same questions, same model, different runs. Nonzero temperature, nondeterministic serving, tie-breaking in sampling — the model disagrees with itself.
3. Harness variance. For agent evals: sandbox flakiness, timeouts, network hiccups, a dependency that resolves differently on Tuesdays. The environment is part of the instrument, and it wobbles (part 8 is about this).

1. Sampling”which questions?”measured by sampling CI
2. Decoding”same question, different runs”measured by rerun σ
3. Harness”the sandbox wobbles”measured by retry logs

Three different wobbles, three different instruments — a stability gate is not a confidence interval.

Here’s why the distinction is more than pedantry. A pattern I’ve seen in serious enterprise eval work — sophisticated work, by people who care — is an improvised stability gate: run the whole eval three times, and only trust metrics whose run-to-run standard deviation is small (say, under three points). It feels rigorous, and it does catch something real: it’s a decoding/harness variance gate. But it says nothing about sampling variance. A metric can be perfectly stable across reruns — three runs, identical 84.3% — and still be ±8 points from the population value, because all three runs used the same 120 questions. The runs aren’t independent evidence about the universe; they’re independent evidence about the decoder.

Self-consistency is not validity. A ruler that gives the same wrong length every time is precise, reliable — and wrong.

The fix isn’t choosing one: report run-to-run σ and a sampling CI. They answer different questions (“will this number reproduce tomorrow?” vs. “is this number close to the truth?”), and a decision-maker usually needs both answers.

Clustered items: the honest interval is wider than you think

The textbook CI assumes your items are independent draws. Real eval sets almost never are. Questions generated from the same source document. Fifty variants expanded from one seed prompt. Tasks harvested from the same repo. Turns within the same conversation. Items labeled by the same annotator. In every case, items share a parent, and items that share a parent are correlated — if the model struggles with that document’s domain, it struggles on all ten questions born from it, together.

Correlated items carry less information than independent ones. The intuition: 120 questions drawn from 12 documents is a lot closer to n = 12 than n = 120 whenever documents differ from each other more than questions-within-a-document do. Your effective sample size shrinks by the design effect — roughly 1 + (k−1)ρ for k items per cluster and within-cluster correlation ρ — while the naive formula happily divides by the full n. Miller’s paper found naive standard errors understated the truth by up to 3× on clustered benchmarks. In blog-land this has gone approximately nowhere, so let’s make it visible.

Your eval set’s family tree shrinks its effective size — 120 items born from 12 parents is a lot closer to n = 12.

Three practical notes. First, you usually know your clusters — you built the dataset. Source-document IDs, seed-prompt IDs, repo names, annotator IDs, conversation IDs: the metadata is sitting in your generation pipeline, and “aggregation assumed exchangeability” from part 2’s anatomy is the line item this fixes. Second, the repair is cheap: compute cluster-robust standard errors (treat cluster means as your units) or run a block bootstrap (resample clusters, not items). Third, know where it bites hardest: synthetic data (one prompt template → fifty variants), RAG evals (one corpus document → many questions), and multi-turn evals (turns within a conversation are practically siblings).

Small n: where the CLT quietly gives up

All the intervals above lean on the central limit theorem, and the CLT needs n to be big enough. How big? For binary scores, a useful recent result — “Don’t use the CLT in LLM evals with fewer than a few hundred datapoints” — puts it bluntly in the title. Below a few hundred items, normal-approximation intervals develop real pathologies, and they’re worst exactly where hard benchmarks live: scores near the extremes (10–30%, or 90%+), where the interval can poke past 0 or 100% or, worse, miss the truth far more often than advertised.

Now recall where agent evals actually live: Anthropic’s own agent-evals guidance suggests starting with 20–50 tasks drawn from real failures. That is not a criticism — 30 real tasks beat 300 synthetic ones — but it means the default ±1.96·SE recipe is unreliable precisely in the regime the field is moving toward.

typical agent eval (20–50) bootstrap / Wilson territory (n < ~200)

typical benchmark (500+) CLT is fine (n ≥ 300)

20501002003005001000+

eval set size n (log scale)

Agent evals live exactly where the textbook interval fails — small n, extreme scores, amber zone.

The decision tree is short. Single proportion, small n → use a Wilson or Clopper–Pearson interval (closed-form, better coverage, in every stats library). Anything more complicated — clustered data, weird metrics, ratios → bootstrap, with the honest caveat that at n=20 the bootstrap isn’t magic either; it’s just less wrong than the normal approximation. And the third branch deserves respect: collect more tasks is a legitimate statistical method — part 4 is about knowing how many you need before you start.

So who’s #1?

Back to the five-way tie. Given scores with honest intervals, there are three defensible ways to publish a ranking, and the simulator’s third tab lets you toggle between them:

1. Rank by point estimate, show the intervals. Reads decisively; quietly implies precision that isn’t there. The reader who ignores whiskers — most readers — takes away a false podium.
2. Confidence-interval ties. Models whose intervals overlap the leader share rank 1. Statistically honest; editorially it reads as a shrug. Five number-ones is an accurate description of the evidence and a terrible headline.
3. Rank ranges. Report each model’s plausible ranks: “1st–5th,” “1st–2nd,” “4th–7th.” This is how LMArena reports, and Scale’s public SEAL leaderboards rank by the upper bound of the plausible range — a model’s rank is the best position it can’t be statistically distinguished from. More to read, much harder to misread.

Which is right? Part 2’s question decides it: what decision does the leaderboard feed? A leaderboard that feeds purchase decisions owes its readers honesty about ties — options 2 or 3 — because “these five are indistinguishable on this benchmark; choose on price and fit” is genuinely useful advice. A leaderboard that feeds marketing wants option 1, and that tension — not statistical ignorance — is why most public leaderboards look the way they do. The people in my cold-open story understood the statistics perfectly; they were negotiating what the instrument was for.

(How rankings should be computed in the first place — Bradley–Terry models, pairwise data, why arena rankings flip — is part 7’s territory. Today was only about what intervals do to a podium.)

The through-line

Error bars aren’t a garnish on the score; they’re the resolution limit of your instrument — part 2’s spec sheet, rows one and two. A leaderboard whose intervals span eight points is a scale that reads to the nearest eight points: fine for weighing elephants, useless for weighing letters. The next post is about matching the resolution to the decision before you build: how many examples you need to detect the difference you care about, why paired designs are the biggest free lunch in eval statistics — and when you’re allowed to stop early.

What this means for your eval

• Report n next to every score. No exceptions — it’s the minimum viable error bar.
• Name your three variances: sampling CI, run-to-run σ, harness flakiness. Report the ones that matter for the decision; never let a stability gate impersonate a sampling CI.
• List your clustering keys (source doc, seed prompt, repo, annotator, conversation). If items share a parent, use cluster-robust SEs or a block bootstrap — the naive interval is fiction.
• Below ~200 binary items, drop ±1.96·SE: Wilson or Clopper–Pearson for proportions, bootstrap for everything else.
• Publishing a ranking? Decide what a tie looks like before you see the scores — rank ranges beat false podiums.
• If two models’ intervals overlap, the honest sentence is “this eval cannot tell them apart” — say it, then decide on other grounds.

References & further reading

1. Evan Miller, Adding Error Bars to Evals (2024) — the canonical treatment: CLT framing, clustered SEs, paired analysis, power.
2. Cameron R. Wolfe, Applying Statistics to LLM Evaluations (2026) — the long-form tutorial with code.
3. Ian Arawjo et al., statsforevals.com — living reference for test selection and intervals.
4. Don’t Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapoints (2025) — small-n interval pathologies and alternatives.
5. Anthropic, Demystifying Evals for AI Agents (2026) — the 20–50-task reality of agent evals.
6. Scale AI, SEAL leaderboard methodology; LMArena rank ranges — rank-under-uncertainty reporting in the wild.

---

Beyond Error Bars · Part 3 of 9 · Previous: An Eval Is a Decision Instrument · Next: How Many Examples — and When Can You Stop?

Views mine.