Humans Are Instruments — Rater Ops Is the Eval

Every eval bottoms out in a human judgment somewhere. This post treats those humans as what they are — measurement instruments — and covers the statistics and the operational discipline that separate labels you can build on from labels that merely look like data.

Stuck in the low seventies

I once watched a benchmark of real professional work get built — tasks where, in the ordinary economy, somebody invoices for the output: analyses, design documents, working deliverables. Model outputs were graded by experienced professionals, and the team did the responsible thing and tracked pairwise inter-rater agreement live as labels came in. The number parked itself in the low 70s and refused to move.

The first hypothesis was the obvious one — the raters aren’t paying attention — and it was wrong. Reading the actual disagreements revealed raters who were careful, consistent, and answering the wrong question. The scale’s anchors were phrased as preference: which of these is better work? So one rater would look at a deliverable that was beautifully polished but missed a requirement and grade it up on craft, while another graded it down on the miss — and both were faithfully applying the rubric, because the rubric never said which of those things mattered.

The fix wasn’t retraining and it wasn’t replacing anyone. It was rewriting the anchors around a question with an external referent: would a paying client accept this work? A carpenter who installs gorgeous windows an inch too small doesn’t collect ninety percent of the invoice; he collects nothing until they’re fixed. Re-anchored that way, the scale cleanly separated “technically unacceptable” from “acceptable, but not what I’d have chosen,” and agreement climbed several points within days — same raters, same items, better instrument.

Inter-rater agreement is not a score you give your raters. It’s a reading you take from your instrument — and low agreement is the instrument reporting a defect, if you’re willing to read the report.

Raw agreement lies twice

The number that team was watching — the fraction of rater pairs that match — is the number everyone watches first, and it flatters you twice. The first lie is chance. Two raters who stamp pass/fail by flipping independent biased coins will still match constantly: if both stamp “pass” 80% of the time at random, they agree 0.8×0.8 + 0.2×0.2 = 68% of the time while communicating nothing at all. Any agreement number that doesn’t subtract this baseline is grading on a curve it drew itself.

Cohen’s κ is the standard subtraction: κ = (p_o − p_e)/(1 − p_e), where p_o is observed agreement and p_e is the agreement two independent raters with the same marginal label rates would reach by luck. Read it as “the fraction of available beyond-chance agreement you actually achieved”: 1 is perfect, 0 is coin-flipping at the base rates, negative is actively anticorrelated.

The second lie is the one the labeling vendors’ kappa explainers tend to skip, and it cuts the other way: class imbalance. Take 100 items, 90% of which genuinely pass — a perfectly normal prevalence for a mature production system. Rater A marks 90 items pass; rater B marks 95 pass; they overlap on 85. Observed agreement: a healthy-looking 85%. But the chance agreement from those marginals is 0.90×0.95 + 0.10×0.05 = 0.86, so κ = (0.85 − 0.86)/(1 − 0.86) = −0.07. Eighty-five percent agreement, negative kappa. Feinstein and Cicchetti named this the first kappa paradox back in 1990: when one class dominates, p_e soars, the denominator collapses, and κ punishes you for the skew of your data rather than the skill of your raters. Kappa isn’t broken — it is correctly answering “how much better than base-rate stamping is this?” — but it means a bare κ from an imbalanced dataset can look damning while both raters are in fact quite good. Prevalence belongs next to every kappa you report; better, report the full marginals table, which the widget below does for you.

One paragraph on the cousin you’ll actually want in production: Krippendorff’s α generalizes the same chance-corrected idea to missing labels, more than two raters, and ordinal or interval scales — which is to say, to what real collection data looks like, where some items got two labels and some got five and rater seven quit in week three. If your labels are pairwise, complete, and nominal, κ and α will nearly agree; the moment they aren’t, α is the right default. I’m not deriving either here, partly because I already have: a couple of years ago I co-wrote a metric-by-metric field guide to inter-rater reliability — percentage agreement through κ, Fleiss and Krippendorff variants, the paradox you just simulated, and the paradox-resistant coefficients (Brennan–Prediger, Gwet’s AC) that labeling platforms increasingly report. The formulas haven’t changed; go there for the worked examples. But one sentence from that piece is the seed this whole post grew from: “Agreement doesn’t inherently imply accuracy.” Reliability tells you your raters are interchangeable — it does not tell you they’re right, which is a separate measurement against gold labels. Everything else in this post is what the metric guides, mine included, under-emphasize.

The same trap in a judge costume

Everything above transfers, verbatim, to the eval du jour: validating an LLM judge against human labels. The MT-Bench paper reported GPT-4 agreeing with human preferences over 80% of the time — about as often as humans agree with each other — and that single number has done a staggering amount of load-bearing work in the years since. But “agreement with humans” on skewed classes is exactly the paradox you just dragged the slider through. If 85% of your outputs pass, a judge that stamps everything “pass” agrees with your humans 85% of the time while detecting zero failures — and the failures are the entire reason you built the eval.

The minimum honest report is per-class agreement: how often does the judge say fail when humans say fail (true positive rate for defect detection), and how often does it say pass when humans say pass (true negative rate)? One number hides the class you care about inside the class you don’t. And once you have TPR and TNR, something better than reporting becomes possible: you can correct the judge’s aggregate scores for its known error rates, and get valid confidence intervals from a handful of human labels plus a mountain of judge labels. That’s part 6 — the flagship — and it’s why this post insists you collect judge-vs-human agreement per class now.

Disagreement is a diagnostic, not a nuisance

Here’s the habit that separates teams who fix their evals from teams who launder them: when agreement is low, don’t reach for the aggregation function — reach for the disagreements. Pull thirty of them and read them. I mean this literally, and I mean the exchange rate literally too: reading thirty disagreements has taught me more, every single time, than computing fifteen agreement coefficients. The coefficients tell you that the instrument is misreading; only the disagreements tell you why. In my experience they triage into three bins, and each bin has a different fix — which is the point, because “average it away” is the same wrong fix for all three.

1. Rubric ambiguity. The raters are answering different questions because your rubric let them. This is the cold-open story: “which do you prefer?” invites taste; “would a paying client accept this?” has an external referent. The fix is instrument surgery — anchor every scale point to a real-world outcome, and make contested calls (does tone count? does format gate pass?) explicit line items instead of ambient vibes. This bin is the best news in the taxonomy: it’s the cheapest to fix and, in my experience, the biggest.
2. Genuine difficulty. Some items are legitimately contested — safety boundaries, register, verbosity taste. Forcing consensus here doesn’t remove the disagreement; it hides it inside a coin flip. The fix is to treat the label as a distribution, not a scalar: report the split, stratify by rater population where it’s structured. DeepMind’s work on rater disagreement (arXiv 2410.17032) makes the case quantitatively: disagreement across rater groups is often stable and informative — a measurement of the population, not noise in it.
3. Rater error and drift. Somebody missed a checkable fact, rubber-stamped a queue, or slowly wandered off-calibration over weeks. This is the only bin where the rater is the problem, and it’s the bin with the most industrial fix: gold sets with known answers salted into the queue, periodic requalification against them, and retirement of raters who stop clearing the bar.

Tab 2 of the widget above is a practice round: eight disagreements, three bins. The tags matter less than the reflex of asking, for every split verdict, which is this? — because the answer decides whether you edit the rubric, keep the distribution, or fix the rater.

When you must aggregate: majority vote is a flat prior

Eventually you do need one label per item, and the default is majority vote: three raters, two say pass, ship “pass.” Majority vote is fine — it’s also a statement that every rater is equally reliable and every vote equally informative, which you know is false because you’ve been tracking per-rater accuracy against gold sets. Dawid–Skene (1979, originally for error rates among clinicians) is the classic upgrade: it jointly estimates each rater’s confusion matrix and the latent true labels via EM, then weights votes by demonstrated reliability. The intuition is lovely — a rater who says “pass” 98% of the time earns little credence for yet another “pass,” but their rare “fail” is a siren; majority vote hears both votes at the same volume. It’s twenty lines of code and routinely worth a few points of label quality on noisy pools.

Rater ops is engineering

Hamel Husain’s FAQ gives the small-team answer to all of this: appoint a “benevolent dictator” — one trusted domain expert whose judgment is the ground truth — and skip inter-rater agreement entirely. For a three-person product team, that’s correct, and I want to be clear I’m not arguing with it: one good instrument beats three uncalibrated ones. It stops being the answer the moment your labels have to be credible to someone who didn’t appoint the dictator — a client, a lab, a regulator, a leaderboard’s readers. At that point you need multiple sensors, and multiple sensors mean agreement statistics, and agreement statistics are only as good as the operation that produced the labels.

Working at an eval vendor teaches you what that operation looks like when someone’s reputation is priced into it. The pattern, generically: blind the model identity — a rater who knows which lab made output B is measuring the logo, not the output. Randomize assignment, including position (left/right, first/second), because order preference is a real and boring bias. No rater sees the same item twice, so replication comes from three different people, not one person’s memory of themselves. Replicate 3× and resolve by consensus — replication is what makes both Dawid–Skene and disagreement triage possible at all. And the one that separates the serious operations from the rest: track agreement live during collection — full-agreement rate, majority rate, all-three-different rate, on a dashboard, while labeling is happening — not computed once after delivery, when the budget is spent and the only remaining option is to average and apologize. Before any of that starts: a calibration round on shared items with a written report, so systematic rater differences surface before production. On a schedule forever after: requalification against gold sets, monthly, because raters drift the way all sensors drift. Under everything: an audit trail — rater, timestamp, justification per judgment — so that when a label is challenged, the answer is a lookup, not an argument.

If that sounds like industrial overkill, consider where the incentive comes from: serious buyers of human labels now put agreement floors in contracts. I’ve seen quality specs of this shape — kappa at or above 0.75 on a held-out gold set, calibration report before kickoff, monthly requalification, full audit trail — as deliverables with payment attached. A kappa threshold as an SLA. When quality has a price, agreement stops being a courtesy metric in an appendix and becomes the thing you are selling — which, if you believe this series’ premise that the eval is a measurement instrument, is exactly what it always was.

Calibrate the sensor

The reframe this post is selling: your rubric is an instrument spec, and your raters are sensors. Sensors have error rates, drift, and bias; you don’t yell at a sensor, you calibrate it — and when readings disagree, you treat the disagreement as data about the instrument before you treat it as noise in the world. Agreement statistics are the sensor’s self-test; rater ops is the maintenance schedule; and the rubric rewrite in the cold open was nothing more exotic than fixing the spec the sensors were built against.

Which raises the obvious next question. If a rater is a noisy, biased sensor that we calibrate, validate per-class, and correct for — what happens when the sensor is a model? Next post: your LLM judge is a biased, noisy instrument, and there is a proper statistical procedure for debiasing it.

What this means for your eval

• Never report raw % agreement alone. Put κ (or Krippendorff’s α) next to it — and put prevalence and the marginals table next to the kappa, or the paradox will read as rater failure.
• Reach for α over κ the moment your data has missing labels, more than two raters, or an ordinal scale — i.e., almost always in production.
• Validating an LLM judge? Report agreement per class (TPR/TNR against human labels), never as one number. Part 6 builds on exactly these quantities.
• Triage disagreements before aggregating: ambiguous rubric → rewrite anchors; genuinely hard → keep the label distribution; rater error → gold sets and requalification. Read 30 disagreements before computing another coefficient.
• Anchor every scale point to a real-world outcome (“would a paying client accept this?”), not to preference.
• Replicate 3×, blind model identity, randomize assignment and position, and never show a rater the same item twice.
• Track full / majority / all-different agreement rates live during collection — after delivery is too late to do anything but apologize.
• Buying labels? Put the floor in the contract: a kappa threshold on a held-out gold set is an SLA, not a vibe. Selling them? Expect exactly that.

References & further reading

1. Jacob Cohen, A Coefficient of Agreement for Nominal Scales, Educational and Psychological Measurement (1960) — the original κ.
2. Alvan Feinstein & Domenic Cicchetti, High Agreement but Low Kappa: I. The Problems of Two Paradoxes, J. Clinical Epidemiology (1990) — the prevalence paradox, worked in full.
3. Klaus Krippendorff, Content Analysis — Krippendorff’s α; the Wikipedia article and labeling-platform explainers cover the formulas well.
4. Hao Rong, Mit Dhami & Naixin Zhang, Unraveling the Mysteries of Inter-Rater Reliability, Scale AI blog (2024) — my earlier metric-by-metric treatment: κ and its multi-rater variants, the kappa paradox with worked numbers, Brennan–Prediger and Gwet’s AC, and the Landis–Koch interpretation bands.
5. A. P. Dawid & A. M. Skene, Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm, Applied Statistics (1979) — reliability-weighted label aggregation, decades ahead of its use case.
6. Google DeepMind, Insights on Disagreement Patterns in Multimodal Safety Perception across Diverse Rater Groups (2024) — rater disagreement as stable, structured signal; labels as distributions.
7. Lianmin Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023) — the canonical judge-vs-human agreement rates (and the reason to read them per class).
8. Hamel Husain, FAQ on AI Evals — the “benevolent dictator” answer: the right call at small scale, and the respectful counterpoint to this post.

Beyond Error Bars · Part 5 of 9 · Previous: How Many Examples — and When Can You Stop? · Next: Your LLM Judge Is a Biased, Noisy Instrument. Debias It.

Views mine.