An Eval Is a Decision Instrument

The same model, three verdicts

A few months ago I watched a model get three different report cards in the same week.

On the team’s internal benchmark, the new checkpoint scored 84%, four points above the incumbent. On a public leaderboard, it ranked behind two models that our benchmark said it beat. And in a small pilot with actual users, people quietly preferred the old version — the new one’s answers were technically better and somehow more annoying.

Nobody involved was incompetent and no eval was “wrong.” The three evals were measuring three different things: performance on a curated task distribution, performance on someone else’s task distribution, and whether real humans in a real workflow wanted to keep using the thing. The mistake — and I’ve made it myself — is talking about “how good the model is” as if that’s one number that these three instruments estimate with different amounts of noise. It isn’t. They measure different constructs, for different decisions.

Percy Liang opens his Stanford evaluation lecture with the cleanest statement of this problem I’ve seen: evaluation is the challenge of turning an abstract construct into a concrete metric. That’s not an engineering observation. That’s the founding problem of measurement science — psychometrics has been chewing on it for a century, and survey methodology, clinical trials, and econometrics all grew their machinery around it. Data scientists inherit that toolbox. This series is about applying it to LLM evals.

But before any statistics, you have to answer a pre-statistical question: what is this eval for?

What is this number going to decide?

Liang’s lecture lists four purposes an evaluation can serve, and they’re worth keeping on a sticky note:

1. Adoption decisions. Someone is choosing model A or model B for a use case. The eval is a purchase recommendation.
2. Capability measurement. Researchers want to know what models can do — the “science of AI progress” purpose.
3. Benefit and harm assessment. Policy, safety, and business risk — what damage could this system do, to whom?
4. Development feedback. The tightest loop: is yesterday’s change an improvement or a regression?

Working at an eval vendor, I’d add a fifth from lived experience:

5. Acceptance. Two organizations agree that work is complete, or a dataset is good enough, or a system is ready to ship, when the eval says so. The eval is part of a contract — sometimes literally, with agreement thresholds and audit rights written into the deal. When quality has a price attached, the eval stops being a dashboard and becomes the referee.

1. Adoptionthe number decides:buy A or B?
2. Capabilitythe number decides:what can models do?
3. Harmsthe number decides:what could go wrong?
4. Dev loopthe number decides:did we improve?
5. Acceptancethe number decides:is the work done?

Same eval, five masters — it cannot serve them all well, and unstated purpose is where eval arguments are born.

The reason to enumerate purposes isn’t taxonomy for its own sake. It’s that the same eval cannot serve all of them well, and unstated purpose is the root cause of most eval arguments I’ve sat through. A benchmark tuned for capability measurement (hard, discriminative, adversarially refreshed) makes a terrible regression suite (you want stable, sensitive, cheap). A preference arena is a fine adoption signal and a useless development loop — by the time your Elo moves, you’ve shipped six versions. When two people disagree about whether an eval is “good,” ask them what decision it feeds. Half the time they’ll discover they’re not disagreeing at all; they’re building two different instruments and calling them by one name.

The anatomy: five components, five silent assumptions

Every eval — MMLU, your RAG regression suite, a human preference study — decomposes into the same five components. Each one ships with a default, and each default smuggles in an assumption that nobody wrote down. Hover or tap each component below.

Datasetthe sample Taskthe rules Scorerthe sensor Aggregationthe mean Decision rulethe gate

Silent assumption: these examples represent the distribution the decision is about.

Failure mode: the eval set was built from the tickets that were easiest to export — sampling bias with a friendly face.

Silent assumption: everyone reports against the same rules of the game.

Failure mode: an agent benchmark where the scaffold — not the model — is doing the heavy lifting, compared against a bare model run. Methods vs. models: say which you’re evaluating.

Silent assumption: the scorer itself is accurate — the sensor inside the instrument is calibrated.

Failure mode: an LLM judge that quietly rewards verbosity, or unit tests that cover half the behavior. The scorer is an instrument inside the instrument (parts 5–6).

Silent assumption: examples are exchangeable and equally important — so a plain mean is meaningful.

Failure mode: 40% of items come from one template or document. Clustered items lie about your effective sample size (part 3), and a mean overweights easy traffic.

Silent assumptions (two): a difference big enough to act on exceeds the noise — and the score means what it says numerically.

Failure mode: thresholding a score with great AUROC but compressed probabilities. Discrimination ≠ calibration (part 9). If the rule uses the value, check calibration.

The arrow into the decision — four questions to answer in writing before building anything: What decision does this number feed? · What’s the baseline? · What’s the cost of each error direction? · What movement would change someone’s behavior?

Tap a component to pin its assumption. “Our eval is wrong” is almost never true of the whole — it’s one of these five violating its assumption.

1. The dataset. A set of examples, standing in for the traffic you actually care about. Silent assumption: these examples are drawn from — or at least representative of — the distribution the decision is about. The polite word for violating this is “sampling bias”; the everyday version is building your eval set from the tickets that were easy to export. And the dataset is a sample, which means everything computed from it has sampling error — the entire subject of the next post.

2. The task definition. What, exactly, is the system asked to do — and what counts as the system? Model alone? Model plus retrieval? Model plus prompt? Plus the agent scaffold? Silent assumption: everyone reports against the same rules. Liang makes a distinction I’ve started using constantly: pre-foundation-model ML evaluated methods on fixed datasets; modern evals mostly evaluate models or whole systems, where anything goes — and agent benchmarks quietly measure the scaffold as much as the model inside it.

3. The scoring function. The thing that turns a response into a number: exact match, unit tests, a rubric, a human rater, an LLM judge. Silent assumption: the scorer itself is accurate. Exact match assumes one correct surface form. Unit tests assume the tests cover the behavior. Human raters assume humans agree (part 5 is about how much they don’t). LLM judges assume the judge shares your definition of good (part 6 is about measuring — and correcting — how much it doesn’t). The scoring function is an instrument inside the instrument, and it is almost never calibrated before use.

4. The aggregation. Almost always: take the mean. Silent assumption: the examples are exchangeable and equally important. A mean over an eval set where 40% of items came from one document, one client, or one template is not measuring what you think it’s measuring — items cluster, and clustered items lie about your effective sample size (part 3). A mean also weights the easy 80% of traffic over the hard 20% your users actually complain about.

5. The decision rule. The threshold, the comparison, the gate: ship if above 0.8; block the PR if it regresses; pick the higher scorer. Two silent assumptions here, and they’re my favorites because this is where I’ve been personally burned. First: that a difference big enough to act on is bigger than the noise — the entire subject of parts 3 and 4. Second, subtler: that the score means what it says numerically. I own a quality-prediction model at work whose ranking is genuinely good — its AUROC would look respectable anywhere — and whose raw probabilities were, for a stretch, nearly useless: compressed into the bottom half of the [0,1] range, so any downstream consumer that applied a threshold rather than a ranking was effectively thresholding noise. Discrimination and calibration are different properties. An eval score can order systems correctly and still be numerically untrustworthy — part 9 walks through that story properly. For now, the rule of thumb: if your decision rule uses the score’s value (not just comparisons between systems), you’ve assumed calibration, and you should check it.

That’s the whole machine: dataset → task → scorer → aggregation → decision rule. Five choices. The reason I insist on the decomposition is that “our eval is wrong” is almost never true of the eval as a whole — it’s one of the five components violating its silent assumption, and they fail differently and get fixed differently.

Two instruments that look identical and share nothing

The taxonomy that saves the most confusion in practice: capability benchmarks and product evals are different instruments that happen to share file formats.

Capability benchmark

measures

the model

cardinal virtue

comparability & difficulty

mortal enemy
contamination

vs

Product eval

measures

your whole system

cardinal virtue

representativeness

mortal enemy
drift

Two instruments, one file format. Say which one you’re building before you argue about how to build it.

A capability benchmark (MMLU, GPQA, SWE-bench) measures a model against a fixed population of tasks. Its purpose is comparison across models and time, so its cardinal virtues are comparability and difficulty, and its mortal enemy is contamination — the moment test items leak into training data, the instrument silently stops measuring capability and starts measuring memorization. Hence the arms race of private splits, fresh-item pipelines, and “verified” cleanups.

A product eval measures a system — your model, your prompts, your retrieval, your guardrails — against your traffic. Nobody else will ever run it, so comparability across organizations is worth nothing; what matters is representativeness, and its mortal enemy is drift: your users change, your traffic mix changes, and an eval set frozen in January is quietly measuring a product that no longer exists by June.

Almost every eval sin I’ve witnessed is a category error between these two. Teams adopt a public benchmark as their product gate because it’s rigorous (it’s rigorous about someone else’s distribution). Teams obsess over contamination in an eval set built from their own private tickets (their real enemy is drift, which nobody assigned an owner). Hamel Husain’s writing on product evals — the best of the process canon — is entirely about the second instrument; leaderboard discourse is entirely about the first. Say which one you’re building before you argue about how to build it.

Design it backwards

Here’s the discipline I’ve settled on, learned partly from watching eval design reviews at work and partly from getting it wrong: start from the decision and derive the eval backwards. Before assembling any dataset, answer four questions in writing:

1. What decision does this number feed? Ship/no-ship? Model A vs B? Vendor acceptance? “We’ll look at it” is not a decision; it’s a dashboard.
2. What’s the baseline? What happens today, without the system — or with the incumbent? A 62% score is meaningless until you know whether the status quo is 40% or 85%.
3. What’s the cost of each error direction? Acting wrongly versus failing to act. A hiring filter and a spam filter can have identical accuracy and wildly different acceptable failure modes.
4. What movement would change someone’s behavior? If a 3-point gain wouldn’t alter anyone’s plans, an eval that can only resolve 5-point differences is the wrong instrument — and one that resolves 1-point differences at 10× the cost is the wrong instrument too. This question — the resolution you actually need — sets your sample size, and it’s the bridge into part 4.

1. The decisionship / don’t ship?
2. Resolution neededthe Δ that changes minds
3. Datasetwhose traffic, what n
4. Scorerwhich sensor, validated

✗ the tempting order — build the eval first, then go looking for a decision it can feed

Start from the decision; the eval falls out. Start from the eval and you get a dashboard.

If you can’t answer these, the fix isn’t a better metric. It’s a conversation with whoever owns the decision.

One more thing I want to put on the record, because in three years of reading eval content I have almost never seen it written down: mature organizations review eval designs the way they review system designs. Where I work, eval plans go through design review — a standing meeting, written scorecards, versioned rubrics, the works. An eval that gates releases or acceptance is load-bearing infrastructure; it deserves the same scrutiny as a database migration. The entire public conversation is about building evals, and almost none of it is about governing them — who reviews the design, who owns the drift, who’s allowed to change the rubric, and what happens to historical comparability when they do. If your evals matter, they need an owner and a change process, not just a dashboard.

Instruments have spec sheets

If an eval is a measurement instrument, then it has the properties every instrument has, whether or not anyone measured them:

1. Variance. Run it twice, get different numbers. How different? (Part 3.)
2. Resolution. The smallest real difference it can reliably detect — set by sample size and design. (Part 4.)
3. Rater reliability. When humans are the sensor, how much of the signal is the humans? (Part 5.)
4. Scorer bias. When an LLM is the sensor, it’s a biased one — measurably, correctably biased. (Part 6 — where this series has been heading all along.)
5. Comparative validity. When instruments rank systems, when do the rankings deserve belief? (Part 7.)
6. Behavior under repetition. Agent evals: reliability versus capability, and the statistics of running things k times. (Part 8.)
7. Assembly and maintenance. Building the whole instrument end to end, and keeping it honest after it ships. (Part 9.)

The rest of this series is that spec sheet, one property at a time, with interactive demos so you can develop intuition instead of taking my word for things. Next up: your eval score is a random variable — error bars, the clustering trap, and why five models tied for first place is an honest leaderboard’s natural condition.

What this means for your eval

• Write down the decision your eval feeds and the construct it claims to measure. One sentence each. If you can’t, stop building and go ask.
• Classify it: capability benchmark or product eval. Then name its enemy — contamination or drift — and give that enemy an owner.
• Audit the five components against their silent assumptions: dataset representative? task rules stated? scorer validated? aggregation cluster-aware? decision rule calibration-checked?
• Fix the decision rule before collecting data — thresholds chosen after seeing results aren’t decision rules, they’re rationalizations.
• If the score’s numeric value (not just its ranking) drives the decision, check calibration — good AUROC does not mean trustworthy probabilities.
• If the eval gates anything that matters, give it governance: an owner, a versioned rubric, and a review before it changes.

References & further reading

1. Percy Liang, CS336 Lecture 12: Evaluation — the construct→metric framing, the four purposes, and the methods-vs-models distinction adapted in this post.
2. Hamel Husain, Your AI Product Needs Evals and the Evals FAQ — the process canon for product evals; this series is the statistics companion to that work, not a replacement.
3. Liang et al., Holistic Evaluation of Language Models (HELM) — evaluation as systematic measurement design.
4. Anthropic, Demystifying Evals for AI Agents — task sourcing and the model-vs-scaffold point.
5. Scale AI, SEAL leaderboard methodology — a public example of decision-focused reporting: confidence intervals and rank-by-upper-bound.

---

Beyond Error Bars · Part 2 of 9 · Previous: Every Eval Era Created the Next Measurement Problem · Next: Error Bars, Applied

Written by a data scientist who builds and lives with evals for a living. Views mine.