How Do You Rank an AI?

The idea behind the most-watched AI leaderboard is simple enough to explain at dinner. You type a question. Two AI models answer it, side by side, and you aren’t told which is which. You pick the better answer. Do that a few million times, and you have a ranking.

That’s it. That’s the whole premise.

What almost nobody reads is the maintenance log. Arena publishes a changelog — every change to how the leaderboard works, dated, in public. Most of its entries are housekeeping: a new model was added. But scattered among them are a couple of dozen entries that quietly rewrite the rules, and read in order they tell a story that isn’t really about leaderboards at all. It’s a story about how fast the thing being measured kept moving, and what had to be invented each time it moved.

I spend my working life on this kind of measurement, so I read these entries the way other people read release notes for a game they play. Here is what three years of them say.

• Dec 2023 · the math

  Counting wins gave the wrong answer

  Opponents aren’t equally hard. Switched to a 1952 model that works strength out from the whole pattern of results — the only time the ranking math has changed since.

• Aug 2024 · the voters

  Style was beating substance

  Bullet points and length sway a vote. Measured how much, then subtracted it out.

• Jul–Oct 2025 · the ballots

  Not every vote was a vote

  Repeated prompts (~10% of votes), users unmasking the model first (<4%), odd voting patterns, models matched against themselves — all filtered out.

• Jul 2025 · the bill

  Knowing how sure you are got expensive

  Replaced a thousand simulations with one formula — same margin of error, a fraction of the compute.

• Nov 2025 · the test

  The questions got too easy

  Models outgrew everyday prompts. Started ranking on the hardest 5.5% of what real users actually send.

• Jun 2026 · the task

  “Answer” stopped being the right word

  Models now do multi-step work. The new scoreboard grades whether the job got done — files produced, retries needed, pushback from the user.

Every entry below is dated and public. Only the first one changed how the ranking is calculated; everything after it changed what gets counted.

1. You can’t just count wins

The first serious change came in December 2023, and it fixed something anyone who follows sports already understands intuitively: a win only means something relative to who you played.

A tennis player with ten victories over beginners is not obviously better than one with five victories over top seeds. Counting wins gets you the wrong answer whenever the schedule isn’t even — and in an arena where matchups are handed out semi-randomly, it never is.

The fix was to stop counting and start inferring. The original leaderboard used Elo — the chess rating system, which nudges your score up or down after each match. Elo is simple and it works, but it has an awkward property: because it updates match by match, the answer depends on the order the votes happened to arrive.

So Arena moved to Bradley–Terry, a model published in 1952 for exactly this problem. Instead of updating as it goes, it looks at the entire history of who beat whom at once and works backwards to a single strength number per model — automatically discounting easy opponents and crediting hard ones. Shuffle the votes into a different order and you get the same answer. (The leaderboard still labels the result “Elo,” because the number was familiar. Under the hood it has been Bradley–Terry since December 2023.)

The leaderboard · December 2023

Model Arena Elo rating MT-bench (score) MMLU License

The leaderboard · August 2024

Rank* (UB) Delta Model Arena Score 95% CI Votes Organization

The same change brought something else that’s easy to miss: for the first time, each rating was published with an error bar. The technique is called bootstrapping — resample the pile of votes at random, refit the model, do it a thousand times, and see how far the score wanders. The spread is your margin of error, and it’s an admission that a score built from a finite number of votes is known only approximately. The interesting part is the lag. The uncertainty was being calculated from late 2023, but it took months longer to reach the table people actually read. Knowing how sure you are and telling anyone turn out to be two separate decisions.

Worth noticing: that was the last time the ranking math changed. Three years on, the algorithm is settled. Everything that follows is about something else entirely.

2. People vote for the wrong reasons

Blind voting removes brand loyalty. It does not remove taste.

Give a person two answers with identical content, but format one as a tidy bulleted list and leave the other as a paragraph, and the list tends to win. Longer answers do better than short ones. Confident phrasing does better than hedged phrasing. None of that is about whether the model is right.

It’s the same problem as grading essays and accidentally rewarding neat handwriting — except at the scale of millions of votes, a small bias doesn’t average out. It just sits there, quietly built into the ranking.

So Arena added what it calls style control. Mechanically it’s the same move an economist makes when controlling for a confounder: response length and markdown formatting get added to the Bradley–Terry model as variables of their own, so the fit estimates how much each one is worth in votes — and the leftover strength score is what’s closer to substance.

And then came my favourite entry in the whole log. In August 2025, Arena launched a leaderboard for search-style answers — and turned style control off for it. The reasoning is a small masterpiece of thinking clearly: in ordinary chat, how an answer is formatted is a distraction from what you’re measuring. In search, how an answer is laid out is part of the product.

A correction is only a correction if what it removes isn’t the thing you were trying to measure.

3. Not every vote is a vote

The next stretch of entries is less elegant and more familiar: people started messing with it.

In July 2025 Arena began filtering repeated prompts — the same handful of questions asked over and over, which had accounted for roughly 10% of all votes. Those aren’t ten independent opinions; they’re one opinion, ten times, and the math had been treating them as ten.

The same update added detection for a nicer trick: users who coaxed the model into revealing which one it was, then voted for their preferred brand. Just under 4% of votes. Two months later came a filter for accounts whose voting patterns were statistically strange, and another for the mildly absurd case of a model being matched against itself.

Each of these is a retroactive admission that an assumption had quietly failed. Bradley–Terry treats every battle as an independent draw — one person, one honest look at two answers. A user asking the same question two hundred times isn’t two hundred draws. Someone who already knows which model is which isn’t voting on the answer. Read together, these entries mark a threshold. A leaderboard starts life assuming everyone is playing in good faith, because at first everyone is. It stops assuming that the moment the number starts to matter to somebody.

4. The questions got too easy

Then a stranger problem: the test stopped working, because the students got too good.

When models were worse, ordinary questions separated them cleanly. As they improved, most everyday prompts stopped discriminating at all — every leading model handles them, and the votes come back close to a coin flip. You can collect a million more of them and learn nothing new.

In November 2025 Arena’s response was to carve out the hardest prompts real users had actually sent, and rank on those. The detail that stays with me is the size of that set: 5.5% of all prompts.

94.5% — can no longer tell the leading models apart 5.5% — still can

Turn that around and it says something striking about where AI is. Roughly 94.5% of what people actually ask these systems can no longer tell the leading models apart. The differences haven’t vanished — they’ve just retreated into a narrow band of genuinely hard work.

5. “Answer” stopped being the right word

All of the above assumes a shape: a question goes in, an answer comes out, a person compares two of them. For a couple of years that was what these systems did.

It isn’t any more. A model now goes off and works — writes and runs code, browses, edits files, uses tools, takes minutes rather than seconds, and comes back with something done rather than something said. You aren’t grading an essay. You’re grading an intern’s whole afternoon.

In June 2026 Arena launched a leaderboard for exactly this, and what it measures is worth reading slowly: not which answer you preferred, but whether files got produced, how often the model had to retry, how many times the user pushed back, and how well it took direction.

That’s a different kind of number, and it needs different statistics. “Which of these two is better” assumes there are two comparable things. “Did it get the job done” is a success rate — and success rates are slippery, because the same task can go well one time and badly the next for no visible reason. So you stop asking is it good and start asking two sharper questions: how often does this work at all — run it a few times, count the wins, which the field calls pass@k — and how often does it work every single time, which is pass^k. The first number climbs as you allow more attempts; the second falls. Which one matters depends entirely on whether you can afford a retry.

6. Knowing how sure you are got expensive

One last entry, from July 2025, that looks like pure plumbing and isn’t.

Every leaderboard number should come with a margin of error — the honest admission that with a finite number of votes, you know the ranking only approximately. The standard way to work that out is brute force: shuffle your data, recompute, repeat a thousand times, and see how much the answer moves. It’s simple, it’s reliable, and it costs a thousand times one calculation.

That technique is the bootstrap from the beginning of this post. In July 2025 Arena replaced it with a closed-form interval — the Central Limit Theorem applied to what statisticians call an M‑estimator, which Bradley–Terry is. Same intervals, one pass instead of a thousand.

Nothing statistical changed; the intervals mean what they meant before. What changed is the budget. When a single test was one question and one short answer, running anything a thousand times was free. When a single test is an AI working for twenty minutes, nothing is free — and the cost of knowing how uncertain you are becomes a line item like any other.

The same update carried a second change that cut the other way. Models don’t all get the same number of matchups, and the fix — reweighting each battle by how often that pairing shows up, applied the same day — buys fairness at the cost of precision, so the margins of error got wider. More pairs of models now overlap, which is to say the leaderboard went from calling them ranked to calling them tied. On the day it got cheaper to measure uncertainty, it also admitted there was more of it.

The pattern

Line the entries up and they’re all the same move, made over and over.

The ranking math was settled in year one and hasn’t been touched since. Everything after that was a response to the ground shifting: people voted on style, so style got measured and removed. Some votes weren’t sincere, so sincerity got tested. Ordinary questions stopped separating good from great, so the questions got harder. Models stopped answering and started working, so the scoreboard stopped grading answers.

Measurement is never finished, because the thing you’re measuring keeps becoming a different thing. What I find genuinely unusual about Arena is not that this happened — it happens to every measurement system eventually — but that they wrote it all down in public, dated, as it went.

The words, if you want them

Everything above has a name in the literature. None of it was invented for AI — the ranking model predates the transistor era, and the rest is standard applied statistics pointed at a new problem.

If you just want to read a leaderboard

Three habits, and you’ll be ahead of most people quoting these numbers in meetings:

• Read the gap, not the rank. Rank is the twitchiest number on the page — a model can drop three places while its own score sits perfectly still, simply because others moved. If two models’ ranges overlap, the honest reading is that nobody knows which is better yet.
• Ask what was on the test. Every leaderboard ranks models on some particular mix of questions. If that mix doesn’t look like your work, the ranking isn’t about you.
• Check what it measures now. “Best model” meant a preferred answer in 2024 and a completed job in 2026. Those are different claims, and a leaderboard from one era doesn’t settle a question from the next.

Sources

1. LMArena, Leaderboard Changelog — the primary source for everything above; every dated change is there.
2. LMSYS Org, Chatbot Arena leaderboard update (December 2023) — the move from win-counting to the 1952 model.
3. LMSYS Org, Does Style Matter? (2024) — how formatting and length were measured and subtracted.
4. Chiang et al., Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (2024) — the methodology paper behind the arena.
5. LMArena, arena-rank — the ranking code, open sourced.

---

Views mine.