How Many Examples — and When Can You Stop?

Enough for what?

The statistics question I get asked more than any other — by engineers, by PMs, once by two different teams in the same week — is some version of “is 50 examples enough?” It’s asked the way you’d show a doctor a mole: hoping for a quick “that’s fine.” And it has no answer, because as posed it’s missing half its clauses. Enough to detect what size difference, at what base rate, feeding what decision?

The framing from the last two posts does the heavy lifting here. An eval is a decision instrument (part 2), and its error bars are its resolution limit (part 3). So “is n enough” really means: does the instrument’s resolution match the difference the decision cares about? At a base accuracy near 70%, a 50-example score carries roughly a ±13-point confidence interval. If your deploy gate exists to catch a 15-point collapse, that’s a working instrument. If it exists to confirm the new prompt is 2 points better, it’s a bathroom scale for weighing envelopes.

The pleasant news: converting “enough for what” into a number takes one formula. The unpleasant news is what the formula says. And the genuinely dangerous news is the second question in the title — when you’re allowed to stop — where the standard practice in eval loops quietly manufactures false positives at six times the advertised rate. In order.

The formula that answers “how many”

When you compare two models, there are two ways to be wrong. You can see a difference that isn’t there — a false alarm, capped at 5% when you test at 95% confidence. And you can miss a difference that is there; the probability of catching it is called power, conventionally set at 80%. Fix both knobs, decide the smallest true difference Δ you need to detect, and the sample size falls out. For two models scored pass/fail on separate question sets at base accuracy around p, the per-model requirement is approximately:

n per model ≈ 2 · (z_α/2 + z_β)² · p(1−p) / Δ²  ≈  15.7 · p(1−p) / Δ²

with z_α/2 = 1.96 and z_β = 0.84, whose sum squared gives the constant. Plugging in a realistic base accuracy of p = 0.7:

Difference to detect	n per model
Δ = 2 points	≈ 8,200
Δ = 5 points	≈ 1,300
Δ = 10 points	≈ 330

Two things to notice before the table finishes sinking in. The Δ² in the denominator is the sting: halving the difference you care about quadruples the bill, which is why “we’ll just eyeball whether the 2-point gain is real” fails on a thousand-example benchmark. And p(1−p) peaks at 50% accuracy, so benchmarks that hover near coin-flip — which is where hard benchmarks are deliberately built to sit — need the most items. The derivations are in Evan Miller’s paper (recommendation 5) and worked through with code in Cameron Wolfe’s tutorial; I won’t repeat them.

How many examples to detect a difference of Δ?

n per model, unpaired, at base accuracy p = 70%

Δ = 10 pts

n ≈ 330per model

4× the examples

Δ = 5 pts

n ≈ 1,300per model

6× more again

Δ = 2 pts

n ≈ 8,200per model

Halving the difference you need to detect quadruples the bill — the Δ² in the denominator is the entire economics of eval sizing.

What should Δ be? Not “any statistically detectable difference” — the smallest difference that would change the decision. If you’d only switch model providers for 5 points or more, Δ is 5 points, and detecting 1-point differences is a luxury you shouldn’t buy. This is part 2’s question wearing a sample-size costume.

The lens is kinder to folk heuristics than you might expect. Anthropic’s agent-evals guidance suggests starting with 20–50 real tasks; the community rule of thumb says review ~100 traces. Run those through the formula and they’re power analysis in folk form: at n = 50 and p = 0.7, an unpaired comparison reliably detects differences of about 26 points. That’s a smoke detector — an instrument calibrated for “did I break something badly?” — and for that decision it’s honestly sized. The mistake isn’t small n. It’s small n with a 2-point claim stapled to it.

The paired free lunch

The table above has an assumption baked in: the two models answered separate question sets — the “unpaired” design, priced at that 2·p(1−p) variance term. But nobody evaluates that way voluntarily. You own the eval set; you run both models on the same questions. If you keep the per-question results instead of just two toplines, the comparison changes character entirely. Questions both models get right tell you nothing about which is better. Questions both get wrong: also nothing. Those concordant pairs cancel out of the comparison. The entire signal lives in the discordant pairs: count b, where A succeeds and B fails, against c, where B succeeds and A fails. McNemar’s test is just:

z = (b − c) / √(b + c)

Two counts and a square root. The required sample size now scales with the discordant rate d = (b+c)/n — the fraction of questions where the models disagree — instead of with 2·p(1−p): roughly n ≈ 7.85·d/Δ². And frontier models agree a lot: similar training data, the same easy questions, the same hard ones, so d is often 5–20%. At p = 70% the unpaired variance term is 0.42; a paired eval with d = 10% needs roughly a quarter of the examples, and the Δ = 5 row of the table drops from ~1,300 to ~310. Across realistic settings the saving runs 2–8×.

detecting the same Δ = 5 pts at base accuracy p = 70%

Unpaired: two separate question sets

model A’s set

+

model B’s set

every question answered by one model only

n ≈ 1,300

questions per model

Paired: same questions, both models

one question, scored by both models

only the disagreements carry signal

n ≈ 310

questions · d = 10% → ~4× fewer

The cheapest rigor upgrade in eval statistics: keep the pairing and test only where the models disagree — the same detection job drops from ~1,300 questions per model to ~310.

The cheapest rigor upgrade in eval statistics: keep the pairing and test only where the models disagree — the same detection job drops from ~1,300 questions per model to ~310.

Here’s my running complaint: paired analysis is the cheapest rigor upgrade in eval statistics, and almost nobody takes it. The data is already paired — you ran both models on the same benchmark, this afternoon. Then the dashboard throws the pairing away, displays two aggregate accuracies with separate error bars, and invites everyone to squint at whether they overlap. Keep the per-question grid. Test the discordant counts. It’s free money. (Miller makes the same point as his recommendation 4; the widget below lets you feel the size of the lunch — and its second tab holds the trap this post is really about.)

The peeking problem

Now the trap. A team is A/B testing a prompt change. The eval runs in batches — nightly, or per CI push — and the dashboard shows a running significance test on the accumulating data. Tuesday the difference crosses p < 0.05; they stop the test and ship. It feels like efficiency: why burn examples to reach n = 500 when the answer was already legible at 180? This is the single most reliable way I know to manufacture a false positive.

Check often enough and noise will eventually look like a discovery — this run is two identical models, and stopping at the first crossing is how a 5% error rate becomes ~30%.

Here’s why. When there is no real difference, the z-statistic doesn’t sit at zero — it wanders. It’s a random walk. Test once at a pre-committed n and there’s a 5% chance the walk happens to be beyond ±1.96 at that instant; that’s the advertised error rate, and it’s honest. But check after every batch and the walk gets a fresh chance at every look. The looks are correlated — but not perfectly, so each look adds real additional risk, and “stop the moment it’s significant” means you keep the maximum of the walk, compared against a threshold calibrated for a single glance. With a handful of planned looks the nominal 5% inflates to ~14%; checking after every one of 25 batches pushes it to 25–30%; peek even more often and 40% is reachable. The second tab of the simulator runs the cleanest version of this: an A/A test, two identical models, true Δ = 0. The fixed-n counter sits near the advertised 5%. The peeking counter — same data, same test, different stopping rule — sits around 30%. One eval in three “finds” a difference that does not exist.

A running significance test on accumulating data is not a stricter check. It is a false-positive machine with a progress bar.

None of this is new — Evan Miller (yes, the same one) wrote “How Not to Run an A/B Test” in 2010 about exactly this failure in web experimentation, and it remains the cleanest explanation in print. What’s striking is watching the eval world rebuild the same trap with better fonts: model-regression gates that re-test on every push, prompt bake-offs watched batch by batch, leaderboard deltas checked daily as new tasks trickle in. Every one of those is a sequential experiment wearing a fixed-n test’s statistics.

The legitimate menu

Wanting to stop early isn’t a sin — examples cost money, and shipping a real improvement three days sooner is worth something. Statistics has three honest ways to buy that option.

(a) Fix n in advance and don’t peek. Use the power formula to choose n, run to n, look once. This is the right default for release gates and one-shot model comparisons, and its only cost is discipline. Boring is a feature: most eval decisions do not actually need to be made mid-run.

(b) A few planned looks, with an alpha budget. If you genuinely need the option to stop early — the clinical-trials situation, where continuing a bad arm has real costs — you can split your 5% error budget across a small number of pre-scheduled looks, paying a stricter threshold at each. The two classic spending schedules differ in temperament: Pocock spreads the budget evenly, so every look pays the same moderately raised bar (roughly z = 2.4 instead of 1.96 for five looks); O’Brien–Fleming spends almost nothing early — the first looks demand overwhelming evidence, so only monster effects end the experiment ahead of schedule — and in exchange the final look costs nearly the full 1.96. Medicine settled this machinery in the 1970s; an eval program that re-tests at three planned checkpoints can borrow it wholesale.

(c) Continuous monitoring, licensed: mSPRT and e-values. If people are going to watch the dashboard every day — and they are — use a statistic that’s designed to be watched. Instead of a p-value, which is calibrated for exactly one look, you maintain a running likelihood ratio: an evidence meter that tracks how much better “the models differ” explains the data so far than “they’re identical.” The betting intuition: under the null hypothesis this meter is your wealth in a fair casino, and a fair casino doesn’t let your wealth multiply 20-fold with more than 1-in-20 probability — ever, at any time, under any stopping rule. So “declare a difference when the meter hits 20” keeps the 5% guarantee no matter how often you look or why you stop. The price is real: by n = 500 in the simulator, the meter demands the equivalent of z ≈ 3.5 where a fixed-n test asks 1.96 — always-valid inference buys the anytime license with power. Commercial A/B platforms adopted this a decade ago (Optimizely’s engine is an mSPRT, from Johari, Pekelis & Walsh); eval tooling, as far as I can tell, has not caught up. The third counter in the simulator is a working one.

Choosing between them is part 2’s question again. A gate that runs once per release: fix n. A comparison where early stopping has genuine value and you can pre-schedule looks: alpha spending. A living dashboard that the whole team will glance at daily no matter what you tell them: always-valid statistics — because if the tooling shows a plain p-value over accumulating data, people will peek, and the fix is to design for the humans you have rather than the discipline you wish they had.

The sensor inside the instrument

Everything in this post assumed the score itself is trustworthy: an answer is right or wrong, a test passes or fails, and the only noise is which questions you sampled and when you stopped. Power analysis tells you how many readings to take. It says nothing about whether the sensor taking them is lying. Most product evals put a judgment call between the model’s output and the number — a human rater, or an LLM playing one — and that sensor has its own noise, its own biases, and its own drift. The next post is about treating humans as what they are: the measurement instrument inside the measurement instrument, with agreement statistics, a paradox that makes high agreement meaningless, and the case for rubric surgery.

What this means for your eval

• Before collecting anything, write down Δ — the smallest difference that would change the decision. Then size the eval: n ≈ 15.7·p(1−p)/Δ² unpaired, n ≈ 7.85·d/Δ² paired.
• Run both models on the same questions and keep the per-question grid. Test the discordant counts (McNemar) — typically 3–10× cheaper than comparing two toplines.
• Know your discordant rate d. It’s the exchange rate of the paired design, and it’s measurable from any past pair of runs.
• Commit to a stopping rule before the first batch: fixed n, planned looks with an alpha budget, or an always-valid method. “Stop when it goes significant” is not on the menu.
• If a dashboard displays running significance over accumulating data, treat its positives as ~30%-false until proven otherwise — or switch the dashboard to an e-value/mSPRT meter.
• A 20–50 task agent eval detects ~20+ point differences. That’s a legitimate smoke detector — label it as one, and don’t staple 2-point claims to it.

References & further reading

1. Evan Miller (Anthropic), Adding Error Bars to Evals (2024) — recommendation 4 (paired differences) and 5 (power analysis) are this post’s starting points.
2. Cameron R. Wolfe, Applying Statistics to LLM Evaluations (2026) — full derivation of the power formula, with code.
3. Ian Arawjo et al., statsforevals.com — living reference for test and interval selection.
4. Evan Miller, How Not to Run an A/B Test (2010) — the classic on peeking; substitute “eval batch” for “visitor” and it reads as written yesterday.
5. Johari, Pekelis & Walsh, Always Valid Inference: Bringing Sequential Analysis to A/B Testing (2015) — the mSPRT machinery behind commercial A/B platforms.
6. Anthropic, Demystifying Evals for AI Agents (2026) — the 20–50-task reality this post sizes honestly.

---

Beyond Error Bars · Part 4 of 9 · Previous: Error Bars, Applied · Next: Humans Are Instruments

Views mine.