Your Design Picks Your Estimator

Estimand before estimator

What The estimand is the quantity you intend to measure. The estimator is the arithmetic you use to guess it.

Why A leaderboard is a claim about a population — of tasks, of prompts, of users. If you can’t write that population down, the ranking isn’t wrong, it’s undefined. “Model X is better” always needs the silent second half: at what, for whom.

Trap Most methodology arguments that feel unresolvable are two people optimising different estimands. Settle the estimand first and the estimator argument usually evaporates.

A point estimate without an interval isn’t a measurement

Why Every leaderboard number is computed from a finite sample — of tasks, of votes. Reported bare, it invites readers to treat noise as signal, and rank changes as events.

Trap Publishing the interval is a separate decision from computing it, and the second one lags. Arena was bootstrapping intervals from December 2023; the 95% CI column didn’t reach the table people actually read until well into the following year.

Ranks are not scores

What A rank is argsort applied to a vector of noisy estimates — a non-smooth function of every model’s score at once.

Why Your own estimate can sit perfectly still while three neighbours drift past it. Scores move continuously; ranks can only jump.

Trap The delta method needs differentiability, so closed-form standard errors don’t give you rank intervals. For ranks you must resample — even when the coefficients themselves have a perfectly good closed form. Knowing which of the two you need is where a lot of compute gets wasted.

A tie is a result

Why “These two are not distinguishable with the evidence we have” is a complete, reportable finding. Forcing an order onto it manufactures a claim the data doesn’t support.

Trap Product surfaces want a strict ordering, so ties get quietly broken by the point estimate. If you publish hard ranks, publish the intervals beside them so a reader can see which neighbours overlap.

Decompose the variance before you measure it

What A per-item score is y₀ = μ₀ + ε₀: the model’s expected score on that item, plus run noise. So a benchmark mean carries two sources of variation, σ²task + σ²noise.

Why Which one you’re estimating decides your whole design. Re-running a fixed item set only ever touches the second term. But the items themselves are a sample from the domain you claim to measure, and that’s usually the bigger term.

Worked Set temperature to zero. Your three runs now agree exactly, the run-to-run spread is zero, and a scoreboard built on it reports that you know the model’s capability precisely. You don’t. You have 82 items, and 82 items can’t pin a score tighter than roughly ±8 points however many times you re-run them.

Trap Reporting run-to-run variance as if it were measurement uncertainty. It is the smaller number and it is the wrong one, so it flatters you twice. And don’t reach for low temperature as a variance-reduction trick — Miller’s eval-statistics paper shows it can shunt variance into the conditional means instead, or buy a smaller spread with outright bias.

Watch the degrees of freedom

What An estimated standard error is itself an estimate, with its own relative error of roughly 1/√(2·df).

Worked Three repeat runs give you two degrees of freedom, so that SD is about 50% uncertain. Twenty jackknife buckets give 19 df → 16%. Eighty-two items give 81 df → 8%. And a correct 95% interval from 2 df needs a t-multiplier of 4.30, not 1.96 — a ±1 SD bar drawn from three runs covers the truth roughly a third to half the time, against the 68% readers assume.

Trap Small-k repeat designs feel empirical — you measured the variability rather than assuming a formula. But at k=3 you measured it so badly that a formula would have done better.

Jackknife, and when it collapses into something you already know

What Recompute the statistic leaving out one item at a time; the spread of those leave-one-out values gives the standard error.

Worked For a plain mean, the per-item jackknife is algebraically identical to s/√n. Four lines of code over per-item scores you were already logging, no resampling loop, no seed.

Why Then why run the machinery? Because the same code handles the statistics where no closed form exists — Elo, a macro-average across categories, pass@k — and handles clustered items by switching what you leave out. One estimator, every metric.

Trap This generalises: the general resampling machine collapses to a formula you already recognise whenever the statistic is simple. Use the formula, and save the machine for when it’s earning its cost.

Cluster on the real dependency

What When items arrive in correlated groups — five questions from one passage, several tasks from one repository — resample whole groups rather than individual items.

Worked Effective sample size is neff = n / (1 + (m−1)ρ) for clusters of size m with intra-cluster correlation ρ. Twenty clusters of five at ρ=0.5 gives you the power of about 33 independent items, not 100.

Trap Ignoring clustering is the most common way evaluation intervals come out too narrow — Miller reports naive standard errors understating by up to 3× on real reading-comprehension evals. And the buckets must be the actual dependency structure. Random bucketing is unbiased but wasteful: 20 buckets throw away df for nothing.

Compare models on paired differences

What To compare A and B, work with the per-item difference d₀ = y₀ᴬ − y₀ᴮ rather than differencing two independent means.

Why Item difficulty is shared, so it cancels. The variance you’re left with is only the part where the models actually disagree.

Worked With between-model correlation r, the difference SD is σd = σ√(2(1−r)). Typical r of 0.5–0.7 puts σd at 0.8–1.0 σ — and at r=0.5 the paired test needs roughly half the items of the unpaired one for the same power.

Trap Pairing is free power that most eval reports leave on the table, because scores get published as separate means and the comparison happens by eyeballing two error bars. Overlapping intervals do not mean two models are indistinguishable when the comparison is paired.

Fix the detectable effect before you fix n

What Decide the smallest gap that would change a decision, then solve for the item count that can see it.

Worked For a paired comparison at 80% power and α=0.05, n = 7.85·σd²/Δ². With per-item SD of 0.35 on a 0–100 scale: a 10-point gap needs ~97 items, 5 points needs ~385, 3 points needs ~1,069. Invert it and 82 items can’t resolve anything under about 11 points — which means most adjacent entries on a small benchmark are statistical ties and should be shown as such.

Trap Running the calculation backwards. Given a fixed n, people compute what’s detectable and then quietly redefine that as the gap worth caring about. Also: at equal compute, buying items beats buying repeat runs and it isn’t close — tripling the item count narrows the interval by 42%, where three reruns narrow it by 9–29% depending on noise share, because reruns only shrink the smaller term.

Bradley–Terry, and why not Elo

What Bradley–Terry (1952) fits a latent strength per competitor by maximum likelihood over the full history of pairwise outcomes. Mechanically it’s a logistic regression on an extremely sparse design matrix: each battle touches exactly two parameters.

Why Elo is the streaming approximation — it updates after each match, which is useful when you can’t refit, and costs you order-dependence: the same votes arriving in a different sequence give a different answer.

Worked Two models, 1,000 battles, A wins 550. δ̂ = logit(0.55) = 0.201, SE = 1/√(n·p(1−p)) = 0.064, so the 95% interval is [0.076, 0.325]. On the Elo-like display scale (×400/ln10): A is 35 points ahead, 95% CI [13, 57].

Trap Arena still labels the output “Elo” for familiarity, but has been fitting Bradley–Terry since December 2023. If someone tells you Elo suits streaming settings better, note that the most streaming setting in existence dropped it precisely because sequential updating was the liability.

Strength of schedule is why raw win rate breaks

What An aggregate win rate is only interpretable when opponents are of comparable difficulty. On an unbalanced graph it silently encodes who you happened to face.

Worked Under a full round robin the top model beats everyone below it and lands near 90%. Under a budget-conscious schedule it mostly plays the second-place model — close to a coin flip — and lands near 55%. A mid-tier model that drew a couple of weak opponents lands at 70%. Raw win rate ranks the mid-tier model first.

Trap The bias has a direction: the harder you optimise the matchup budget, the more you suppress strong models. This is not a tail risk, it is the systematic consequence, and it’s the single most important thing Bradley–Terry is buying you.

Control the confounders you can’t filter

What Some biases are present in every observation, so removing the affected rows isn’t available. You add them to the model as covariates and let the fit estimate their pull.

Worked Style control puts response length and markdown formatting into the Bradley–Terry fit as their own variables. Arena has since added an indicator for direct-chat battles (position bias toward the first option) and one for same-organisation matchups.

Trap A control is only a control if what it removes isn’t what you’re measuring. Arena runs style control on chat and turns it off for search, where how an answer is laid out is part of the product. Same knob, opposite setting, because the estimand changed.

Independence is the assumption every filter defends

What Bradley–Terry treats each battle as an independent draw: one person, one uninformed look at two answers.

Worked Arena’s published filters, read as a list of ways that fails: repeated high-frequency prompts (about 10% of votes) — two hundred asks of one question are not two hundred draws; users who extract the model’s identity before voting (under 4%) — not a judgement about the answer; accounts with statistically anomalous voting patterns; and models matched against themselves, which carry no information at all.

Trap The filter list is the list of ways your independence assumption has already failed. If you run a preference board and have no such list, that isn’t evidence the problem is absent.

Reweighting trades precision for fairness

What Models accumulate wildly unequal matchup counts. Weighting battles by the inverse of how often that pairing appears corrects the resulting bias.

Trap It widens the intervals, and Arena says so plainly: more model pairs then overlap, which means more entries become ties. Don’t attribute that widening to a co-shipped change in how intervals are computed — swapping bootstrap for a closed form is asymptotically equivalent and doesn’t move the width. Two changes, one release note.

Spend comparisons where they change the ranking

What Choose the next matchup to maximise reduction in ranking uncertainty, rather than sampling pairs uniformly.

Worked Ranking is a sorting problem: it needs O(K) informative comparisons, not O(K²). For 12 models a full round robin is 66 pairs; verifying near-neighbours takes roughly 2K−3 ≈ 21. The Arena paper’s formal version reports reaching a given precision on the win matrix with 4,400 samples against 6,800 for random sampling.

Trap Doing this obligates you to Bradley–Terry. See the previous section, and the crossover below — this is the most common way a team saves budget and pays it back in bias without noticing.

You start scheduling adaptively

Move Complete → incomplete.

Skipping the matchups that are already clearly separated is the right call on budget. But it makes the schedule deliberately unbalanced, and an aggregate win rate is no longer a valid global ordering. Choosing an adaptive schedule is what obligates you to latent-strength estimation. Budget saved in comparisons comes back as bias in the ordering unless the estimator moves with the design.

New models join and old ones aren’t re-run

Move Complete → incomplete, quietly, over time.

A board that started as a clean Regime A drifts out of it the first time you add a model without backfilling, or retire items. Means stop being comparable because they’re no longer means over the same population. This is the cell where IRT lives, and it arrives by calendar rather than by decision.

You start reporting a macro-average

Move Linear → non-linear statistic.

Averaging per-category scores rather than per-item scores is usually the right reporting choice — it stops a large category dominating. But the headline number is no longer a simple mean, so s/√n doesn’t describe it and the closed form quietly stops applying. Resample at the level the dependency actually lives.

Your tie rate spikes

Move Not a move at all — and that’s the point.

The instinct is to collect more data. Sometimes that’s right, and it’s the most expensive of the three possibilities, so check it last. A high tie rate is a symptom, not an answer. Work down in this order:

One Do raters disagree with each other? Then it’s rater noise, and more samples buy you more expensive noise. Fix calibration, guidelines, or rater qualification.

Two Do they agree, and agree it’s a tie? Then the models really are equivalent on these items, and the fix is a harder or more discriminative item set — not a larger one. Cohere's prompt-prioritisation work found that selecting on model-disagreement cut indecisive outcomes by 54% in the top quintile.

Three Neither? Now it’s a sample-size question, and now you may spend. And size it up front from a detectable effect — collecting until the tie breaks is optional stopping, and your false-positive rate ends up nowhere near the 5% you think you’re running at.