Rankings Under Uncertainty: What Arena Numbers Mean

From #4 to #7 without shipping anything

A model I help evaluate spent a comfortable month at #4 on a public arena. Then, over about ten days, it slid to #7. Nothing had shipped — same weights, same endpoint, same system prompt; the release calendar was empty and the on-call channel was quiet. Leadership asked what happened, in the way leadership asks when a number with your name on it moves in public. The expected deliverable was a root cause. The honest answer — which I recommend delivering with more confidence than it seems to deserve — was: possibly nothing.

That answer sounds evasive, but it’s a precise statistical claim. A rank is not a measurement of your model. It is a nonlinear function of every model’s noisy strength estimate at once: “we fell three places” is an event involving at least four estimates, three of which belong to other people. Your strength estimate can sit perfectly still while three neighbors’ estimates wobble upward past it. And where strength estimates drift smoothly as votes accumulate, ranks are integers — they can only jump. Rank is arguably the most volatile statistic the field publishes, and it is also the one that goes in the headline. To defend “possibly nothing” in a meeting, though, you need to know what an arena number actually estimates. That’s this post.

From scores to preferences

Everything earlier in this series had a score per item: the model passed the task or it didn’t, and the benchmark number was a mean with an error bar (part 3). Arena data is a different animal. A person types a prompt, two anonymous models answer, and the person clicks the better one. There is no ground truth per item and no absolute scale — nobody ever decides whether an answer was good, only which of two was preferred, once. The dataset is a tournament, not a gradebook, and every datapoint is purely relative evidence. This design is not a defect; it’s the point. Fresh prompts from real users are hard to contaminate, and “which would you rather receive?” is often closer to the product question than any static benchmark. But it changes the statistics from the ground up.

Benchmark scores: absolute

a score means something on its own

• Model A · 78
• Model B · 71
• Model C · 64

0 —— 25 —— 50 —— 75 —— 100  ·  a real zero

Arena votes: relative

only pairwise preferences

A A wins 65% → B A wins 68% → C B wins 55% → C

no ruler, no zero — only gaps

Pairwise data has no zero — only distances between players. A benchmark score means something on its own; an arena rating only means something relative to the other ratings on the board.

The Bradley–Terry model, and why “Elo” is a nickname

The standard machine for turning tournaments into rankings is the Bradley–Terry model — 1952, invented for actual tournaments. Every model i gets a positive latent strength π_i, and

P(A beats B) = π_A / (π_A + π_B)

Take logs — β = ln π — and the same model becomes

P = 1 / (1 + exp(−(β_A − β_B)))

a logistic regression in strength differences, where the only features are model identities. Two consequences fall straight out. First, only differences are identified: add a constant to every β and no prediction changes, so the fit pins the mean (or a reference model) at zero — a rating means nothing alone; only gaps do. Second, gaps convert to win probabilities: a β-gap of 0.62 is a 65–35 matchup, and on the Elo-style display scaling arenas use, about 100 points is roughly a 64% win rate. That’s the exchange rate between leaderboard points and reality, and it’s worth doing in your head before reacting to any gap.

Where does Elo fit in? Elo is Bradley–Terry fitted incrementally: after each game, nudge both ratings by a learning rate times the surprise (observed result minus predicted win probability). That is exactly the right tool for chess, where players genuinely improve and decay and the rating must track a moving target. It is a suboptimal tool for model snapshots, which are static: the learning rate never fully averages out, and the estimate depends on the order the votes arrived. Chatbot Arena worked through this in public and switched, in December 2023, from online Elo updates to fitting Bradley–Terry by batch maximum likelihood over the whole vote log (LMSYS blog; the methodology paper is Chiang et al., 2403.04132). The numbers are still called “Elo scores” everywhere. They are batch BT estimates wearing Elo’s scaling.

The error bar on a rank

Fitting gives you a β per model, and β deserves the part-3 treatment: a confidence interval. You can get one analytically — invert the Fisher information at the maximum, or use a sandwich estimator if you distrust the model (you should; see the next section) — or, simplest to get right, bootstrap over votes: resample the vote log with replacement, refit, repeat a few hundred times, take percentiles. Strength CIs behave like every CI in this series: they shrink like one over the square root of the number of votes, matchup by matchup.

But nobody publishes strengths. They publish ranks, and ranks inherit more uncertainty than strengths. Rank is a step function of all the estimates simultaneously — rank_i = 1 + #{j : β_j > β_i} — so it inherits noise from every model, not just yours. In a tight pack — and the top of every arena is a tight pack, because frontier models cluster — a model can have a perfectly respectable strength CI and a plausible-rank range of #2–#8, because five neighbors sit within one wobble of each other. The more models crowd the board, the worse this gets: every new entrant near your strength adds another coin flip to your rank. This is part 3’s error-bar problem, one level up — ranks inherit more uncertainty than strengths — and the honest reporting conventions are the same ones: rank ranges (LMArena’s convention) or rank-by-upper-bound — the convention on Scale’s public SEAL leaderboards, where your rank is the best position you cannot be statistically distinguished from.

Which was, in the end, the answer that went upstairs: our model’s rank range had been #3–#8 all quarter. #4 and #7 were both inside it the whole time. The arena had never measured us to single-rank resolution — the leaderboard just displayed it that way.

What the model assumes — and how arenas break it

Bradley–Terry is a model, and its error bars are only as honest as its assumptions. Four of them fail on real arenas, each in its own way.

One dimension — hence transitivity. BT places every model on a single strength line, which implies stochastic transitivity: if A usually beats B and B usually beats C, then A must usually beat C. Real preferences don’t owe us that. Models have skill profiles — one is the best coder, another the best writer, a third the fastest and tersest — and an arena’s prompt stream mixes domains, so aggregate preferences can cycle: A over B over C over A. BT cannot represent a cycle. It does not crash or warn; it converges cheerfully to middling strengths that misdescribe every single matchup, and the fit diagnostics look normal. That’s the sandbox’s second world: the bars settle, the CIs tighten, and only the win-rate matrix shows what the ranking is hiding. Practitioner translation: when models specialize differently, their BT gap is mostly a fact about the prompt mix, not the models — read category leaderboards and pairwise matrices before believing a scalar order.

What the votes say

preferences can cycle

What Bradley–Terry reports

forced onto one line, the cycle cancels

Aβ 0.00

Bβ 0.00

Cβ 0.00

BT says: all equal

A one-dimensional scale cannot represent a cycle — it averages it away. The fit converges, the CIs tighten, and every single matchup is misdescribed.

Stationarity. The fit assumes each β is constant across the vote window. But vendors update models behind stable names: silent snapshot swaps, serving and quantization changes, deprecations. A name on an arena is a pointer, and votes from March pool with votes from June into the rating of a blend that may never have existed at any single moment. Practitioner translation: before reacting to a rating, check that it’s stable across time slices — a rating attached to a name is an average over everything that name has pointed to.

The vote-generating population. The estimand is “the preferences of people who visit this arena, on prompts they choose to type, judged by whatever criteria they feel like using.” That last clause bites hardest: votes reward style as well as substance. Longer, better-formatted, more confidently phrased answers collect extra wins independent of correctness — and when LMArena added style controls, regressing length and markdown effects out of the BT fit, rankings moved, in some cases substantially. Practitioner translation: an arena rank measures a population you don’t serve, on a task mix you didn’t choose, with tastes you may not share; if your users or your rubric differ, expect your ranking to differ too.

Selection effects. “The Leaderboard Illusion” documented the strategic layer above the statistics: private variant testing (submit several variants anonymously, publish only the winner — the maximum of k noisy draws is biased upward, plain order statistics), selective score disclosure, and unequal sampling and deprecation across providers. Note carefully what this is not: it is not sampling error. Every confidence interval on the board can be computed flawlessly and the ranking still be distorted, because the bias entered through which data was allowed to exist. Practitioner translation: statistical rigor in the fit says nothing about the strategy above it — before trusting a leaderboard, ask what the entrants got to choose.

When to trust an arena number

Trust it when the evidence is strong in exactly the ways this post makes checkable: rank ranges that don’t overlap (a gap, not a photo finish); ratings stable across time slices; a prompt distribution that resembles your task, or a category leaderboard that does. Arenas are genuinely good at the shortlist question — which five models are worth evaluating at all — and at detecting large capability gaps, which no amount of styling or selection fully manufactures. Distrust it for adjacent-rank distinctions, narrow domains, and style-sensitive tasks — which is to say, for most decisions people actually bring to a leaderboard. For those, the answer is parts 3 and 4: your prompts, your rubric, a paired design, and a sample size you chose on purpose.

When to believe a leaderboard, in one tree. Arenas are good at the shortlist and at large gaps; adjacent ranks and narrow domains are yours to measure.

One endorsement to balance the skepticism. For internal model comparisons — a prompt bake-off, a fine-tune against its base — pairwise preference plus Bradley–Terry is often better instrumentation than absolute scoring. Raters are far more reliable at “which of these is better?” than at holding a 1–7 rubric anchored across weeks (part 5’s drift problem), and BT turns those comparisons into strengths with real CIs. Two design rules make it work. First, keep the comparison graph connected: BT only identifies differences along paths of comparisons, so if your models split into two pods that never face each other, cross-pod gaps aren’t just noisy — they are undefined; the likelihood has no unique maximum. Random pairing keeps you connected; so does a common baseline every candidate fights. Second, decide what ties mean: real votes include “both fine” and “both bad,” and tie-aware extensions (Rao–Kupper, Davidson, and the systematic treatment in Ameli et al.’s leaderbot) use that information instead of discarding it.

The unit changes next

Every post so far, this one included, has assumed the atom of evidence is a single attempt: one item, one vote, one pass/fail. Agents break the atom. When a model gets k tries at a task, when the sandbox itself is flaky, when “can it ever do this?” and “can it do this every time?” have different answers — a single attempt isn’t even the right unit of analysis. Capability and reliability come apart, and pass@k and pass^k need their own error bars. That’s part 8.

What this means for your eval

• Never quote an arena rank without its rank range. A rank move inside overlapping ranges is noise until proven otherwise — “possibly nothing” is a complete answer.
• Treat “Elo” numbers as Bradley–Terry log-strengths: only gaps mean anything, and convert gaps to win probabilities (≈100 points → 64%) before reacting to one.
• Before citing a leaderboard, run the three checks: stable across time slices? prompt mix matches your domain? does the ranking survive style control?
• Look at the pairwise win matrix before believing the scalar order — if preferences cycle, the prompt mix, not the models, is choosing the ranking.
• Remember the CIs can all be right while the ranking is gamed: selection effects (private variants, selective disclosure) live above the statistics.
• For internal comparisons, prefer pairwise votes + BT over absolute rubric scores — and keep the comparison graph connected by design, with a policy for ties.

References & further reading

1. LMSYS Org, Chatbot Arena leaderboard update (Dec 2023) — the switch from online Elo updates to batch maximum-likelihood Bradley–Terry.
2. Chiang et al., Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (2024) — the arena methodology paper: BT fit, sampling, intervals.
3. David R. Hunter, MM Algorithms for Generalized Bradley–Terry Models, Annals of Statistics 32(1) (2004) — the fitting algorithm the sandbox implements.
4. LMSYS Org, Does Style Matter? (2024) — style control: separating substance from length and markdown in arena votes.
5. Singh et al., The Leaderboard Illusion (2025) — private variant testing, selective disclosure, unequal sampling; see also Simon Willison’s digest.
6. Ameli et al., A Statistical Framework for Ranking LLM-Based Chatbots (leaderbot, 2024) — tie-aware Bradley–Terry extensions with covariance modeling.
7. Scale AI, SEAL leaderboard methodology — rank-by-upper-bound reporting; LMArena — rank ranges in the wild.

---

Beyond Error Bars · Part 7 of 9 · Previous: Your LLM Judge Is a Biased, Noisy Instrument. Debias It. · Next: Agent Evals: the Statistics of pass@k and pass^k

Views mine.