Every Eval Era Created the Next Measurement Problem

Every few years, the field quietly decides that its way of measuring language models is broken, invents a new one, and moves on. It’s usually right both times — the old way was broken, and the new way is better. What rarely gets said out loud is that each new way was built to fix the specific fracture in the last one, and each brought a fresh measurement problem of its own. If you line the eras up, LLM evaluation stops looking like a pile of benchmarks and starts looking like one long engineering project: patch, crack, patch.

I find this the most useful way to orient anyone entering the field, and I owe the framing’s cleanest statement to Percy Liang’s Stanford evaluation lecture: the core challenge of evaluation is turning an abstract construct (“is this model good?”) into a concrete metric. Every era below is one attempt at that translation. Every crack is a place where the metric and the construct came apart.

This post is the story. The rest of the series — nine parts, all linked at the bottom — is the toolbox the story ends by demanding. I work as a data scientist at an eval vendor, which means I’ve spent the last couple of years watching these cracks up close, on leaderboards and inside enterprise eval programs; that vantage point shapes everything here.

Six eras, one pattern

Perplexity–2020 Exams2020–23 Preferences2023– LLM judges2023– Agents2024– Environments2025–

Measured: Perplexity — the exponentiated average surprise on held-out text: (1/P(D))^(1/|D|). One number, smooth, differentiable.

Why it won: Cheap, universal, and theoretically seductive: perfect perplexity = the true distribution = every task solvable. Scaling laws are drawn in this currency.

How it cracked: It spends attention indiscriminately — every token penalized equally, relevant or not — and models with similar perplexity behave differently on tasks people care about. The construct drifted away from the metric.

Measured: Accuracy on curated question sets with unambiguous answers — MMLU’s 57 subjects, then harder: MMLU-Pro, GPQA, HLE.

Why it won: Control over topic and difficulty, trivial grading, cross-model comparability. The lingua franca of model releases for three years.

How it cracked: Saturation forced a difficulty arms race; internet-scale training made contamination the default assumption; and nobody asks their assistant multiple-choice questions — realism was never in the design.

Measured: Human pairwise preferences on real prompts, aggregated into rankings (Elo, then Bradley–Terry) — Chatbot Arena.

Why it won: Real users, real prompts, and a construct (what people prefer) that exam scores never touched. Free data at scale.

How it cracked: Style beats substance distressingly often; the voter population is uncontrolled; it’s too slow for a weekly dev loop — and rank is the most volatile statistic on the page (part 7).

Measured: Model outputs graded by another model against a rubric or reference — pass rates, scores out of ten, win rates.

Why it won: Evaluation at 1% of the cost and 100× the speed. Every team could suddenly afford to measure everything.

How it cracked: The grader is an unvalidated instrument: position bias, verbosity bias, self-preference — and length-hacked leaderboards forced the question ‘how do we evaluate the metric?’ The answer is part 6.

Measured: Task outcomes in tool-using, multi-step settings — repos patched (SWE-bench), terminals driven, tickets closed.

Why it won: Closest yet to economic value: measuring what models do, not what they say.

How it cracked: You’re grading model + scaffold as one unit; environments flake in ways that impersonate model failure; and one attempt is the wrong unit — pass@k vs pass^k (part 8) is the demo/product gap.

Measured: Performance inside living worlds: interactive environments, live websites, occupation-realistic tasks graded by professionals (GDPval, ARC-AGI-3).

Why it won: The realism endgame — tasks with the texture, ambiguity, and stakes of actual work.

How it cracked: The world won’t hold still: sites redesign, APIs version, data rots. The instrument decays in your hands — eval maintenance becomes part of the eval (parts 8–9).

Eras overlap and none ever fully dies — perplexity still steers pretraining, exams still gate releases. What changes is which instrument the field trusts for its headline claims.

Perplexity was evaluation’s classical period: a language model is a probability distribution, so score it by the probability it assigns to held-out text. It was clean, cheap, differentiable, and it produced the smooth curves that scaling laws are made of — it still does. The theory was even seductive: perfect perplexity means you’ve matched the true distribution, which means you can solve any task expressible as text. But perplexity spends its attention indiscriminately — it penalizes surprise on every token, whether or not the token matters — and two models with similar perplexity can behave very differently on the tasks people care about. The construct (“useful model”) and the metric (“average surprise”) drifted apart, and GPT-era models made the drift undeniable.

Exams patched that: test the model like a student. MMLU’s 57 subjects, unambiguous answers, easy grading — control over difficulty and topic that perplexity never offered. The crack came from two directions at once. First, saturation: models caught up, so the field escalated — MMLU-Pro pruned the noise and scores dropped by double digits; GPQA hired PhDs to write questions Google can’t answer; Humanity’s Last Exam paid a prize pool for questions at the frontier of human knowledge. An arms race of difficulty is not a sign of healthy measurement; it’s a sign the instrument keeps getting outgrown. Second, contamination: when your training set is the internet, the test set is probably in it. And beneath both cracks sat a validity problem no difficulty could fix: nobody asks their assistant multiple-choice questions. The construct had moved on.

Human preferences patched the realism gap: put two anonymous models in front of real users with real prompts and ask which answer they prefer. Chatbot Arena turned pairwise votes into rankings, and preference data became the currency of the RLHF era. The cracks: preference conflates style with correctness (a confident wrong answer beats a hedged right one distressingly often), the voting population is whoever shows up, and the whole apparatus is slow and expensive relative to a development loop that ships weekly. Preference is a real construct — it’s just not the only one, and it’s gameable by charm.

LLM judges patched the cost problem: if humans are slow and expensive, let a model grade the outputs. Suddenly every team could afford evaluation at scale. The crack is the one this series spends an entire flagship post on: the judge is itself an unvalidated instrument — position-biased, verbosity-biased, partial to its own family’s outputs — and the first leaderboard gaming scandals (length-hacking AlpacaEval) forced the question that measurement science starts with: how do you evaluate the metric? The judge era smuggled a second model’s error into every number, and most teams still report those numbers as if the judge were ground truth.

Agents patched the “models talk but don’t do” gap: give the model tools and a goal — patch this repo, complete this ticket, run this terminal — and score outcomes. SWE-bench and its cousins moved evaluation dramatically closer to economic reality. The cracks multiplied accordingly: you’re now measuring a model plus a scaffold and the scaffold matters as much as the model; environments are flaky in ways that masquerade as model failure; and a single attempt stopped being the right unit of measurement — the gap between “can do it ever” and “does it every time” (pass@k versus pass^k) became the difference between a demo and a product.

Environments — where we are now — patch the last stretch of realism: interactive worlds, live websites, multi-step professional tasks judged by working professionals, benchmarks like GDPval built from actual occupations. The crack is already visible, and it’s the strangest one yet: the world refuses to hold still. Websites redesign, APIs version, data rots — the instrument decays while you’re using it, and an eval you don’t re-audit is quietly measuring the drift of the world rather than the ability of the model.

The pattern, stated plainly

Read the chain again and a pattern falls out: each era solved the previous era’s construct problem and introduced a new measurement problem. Exams fixed perplexity’s relevance gap and created contamination and saturation. Preferences fixed the realism gap and created rater-population and style-confound problems. Judges fixed the cost problem and created the unvalidated-instrument problem. Agents fixed the “doing” gap and created scaffold conflation and run-to-run stochasticity. Environments fixed the last of the realism gap and created instrument decay.

1. Perplexityrelevance gapP2
2. Examscontamination + saturationP3
3. Preferencesstyle confound, who votes?P7
4. Judgesunvalidated graderP6
5. Agentsscaffold + flakinessP8
6. Environmentsinstrument decayP9

The cracks don’t close when the field moves on — they accumulate. The badge under each crack names the series part that treats it: this series is the repair manual.

The benchmarks keep changing. The measurement problems just accumulate — every era’s crack is still open, still waiting for statistical treatment.

One thread runs through all six eras and deserves its own sentence: validity. Contamination is not one problem but a spectrum of severities — benign overlap of task universes at one end; exact prompt-and-solution leakage in the middle; and, at the nasty end, ground-truth answers leaking into training signals, where a model never sees the test but learns to game its distribution anyway. The field’s defenses (private holdout splits, fresh-item pipelines, reporting norms, contamination inference) are all instrument-maintenance practices — and the fact that we now need severity taxonomies for test leakage tells you evaluation has become adversarial infrastructure, not a courtesy report.

benignnasty

1. shared task universesame domain, no overlap
2. similar tasksparaphrases & near-twins
3. exact prompts leakedtest items in pretraining
4. answers leaked into training signalgamed without being seen

Not all contamination is equal — locate yours on the severity spectrum before panicking (or relaxing).

The wave that stopped at the shore

Something encouraging happened in late 2024: the field noticed the missing error bars. Evan Miller’s Adding Error Bars to Evals said the quiet part with formulas: an eval score is a sample statistic, and reporting it without uncertainty is malpractice we’d accept in no other quantitative field. The wave that followed — Cameron Wolfe’s long tutorial, statsforevals.com, CIs appearing on serious leaderboards — genuinely moved the field. If you only absorb one idea from that wave, it’s enough to change how you read every leaderboard you’ll see this year.

But the wave stopped at the shore. Confidence intervals on a mean are the first page of measurement science, and the cracks the six eras left open need the rest of the book: the statistics of noisy graders (human and machine), of rankings, of repeated stochastic runs, of calibrated scores, of sequential decisions. Nearly all of it exists — in epidemiology, psychometrics, survey methodology, and a recent run of excellent papers that almost nobody has translated for practitioners. That translation is this series.

The series

1. Every Eval Era Created the Next Measurement Problem — you are here.
2. An Eval Is a Decision Instrument — the anatomy: five components, five silent assumptions, and the decision the number feeds.
3. Error Bars, Applied — clustered items, small n, and why five models tied for first is an honest leaderboard’s natural condition.
4. How Many Examples — and When Can You Stop? — power analysis, the paired-design free lunch, and the peeking problem.
5. Humans Are Instruments — the prevalence paradox, disagreement as diagnostic, and rater ops as engineering.
6. Your LLM Judge Is a Biased, Noisy Instrument. Debias It. — the flagship: judge validation, Rogan–Gladen, and prediction-powered inference.
7. Rankings Under Uncertainty — Bradley–Terry, arena critiques, and what rank ranges actually mean.
8. Agent Evals: the Statistics of pass@k and pass^k — capability vs reliability, unbiased estimators, and run budgets.
9. Building a Product Eval, Instrumented — the end-to-end build, calibration ≠ discrimination, and eval governance.

Every statistical post ships an interactive demo — you can watch a naive confidence interval fail its coverage promise, watch a leaderboard podium dissolve into a statistical tie, watch a biased judge get debiased in real time. Statistics you can poke at sticks better than statistics you read.

What this means for your eval

• Name your era: which instrument does your team’s headline number come from — exam, preference, judge, agent, environment? Its era’s crack is your eval’s standing risk.
• If your eval strategy is “a benchmark we adopted in 2023,” check it for the two exam-era diseases: saturation and contamination.
• If a model grades your outputs, ask what validates the grader — and read part 6 before trusting its aggregate numbers.
• If your agent eval runs live environments, schedule instrument re-audits: measure the eval’s drift, not just the agent’s.
• Put error bars on the next eval number you report. It’s the first page — the series is the rest.

References & further reading

1. Percy Liang, CS336 Lecture 12: Evaluation — the construct→metric framing and the era survey this post builds on; the best single hour on eval evolution.
2. Evan Miller, Adding Error Bars to Evals (2024) — the paper that started the statistics wave.
3. Hendrycks et al., MMLU (2021); Wang et al., MMLU-Pro (2024); Rein et al., GPQA (2023); Phan et al., Humanity’s Last Exam (2025) — the exam-era difficulty escalation, in four acts.
4. Chiang et al., Chatbot Arena (2024) and Singh et al., The Leaderboard Illusion (2025) — the preference era and its critique.
5. Zheng et al., Judging LLM-as-a-Judge (2023) — the judge era’s founding audit.
6. Jimenez et al., SWE-bench (2023) and OpenAI, SWE-bench Verified (2024) — the agent era and its first instrument-maintenance overhaul.
7. Zhang et al., GSM1k (2024) — contamination measured directly: up to 13-point drops on a fresh twin benchmark.
8. OpenAI, GDPval (2025) — the environment/realism era’s flagship.

---

Beyond Error Bars · Part 1 of 9 · Next: An Eval Is a Decision Instrument

Written by a data scientist who builds and lives with evals for a living. Views mine.