Your LLM Judge Is a Biased, Noisy Instrument. Debias It.

Everyone validates their judge now — and then trusts its pass-rate anyway. Judge accuracy is not estimate accuracy. Here is the correction epidemiology worked out in 1978, and the 2023 estimator that makes a small human-labeled set go a very long way.

92% agreement, and wrong anyway

Judge validation has, mercifully, become table stakes. Nearly every serious eval I’ve seen built in the past year performs the same ritual: hold out a hundred-ish outputs, have humans label them, run the LLM judge over the same set, compute agreement. The number comes back — say 92% — someone posts it in the channel with a checkmark, and from that day forward the judge grades everything. Five thousand outputs later the judge says 74% pass, and 74% goes into the deck, the dashboard, and the go/no-go review.

The ritual validates the wrong thing. Hidden inside it is an inference nobody says out loud: the judge agrees with humans 92% of the time, therefore its pass-rate is approximately the true pass-rate. That does not follow, because agreement is one number and judge errors are not symmetric.

Concretely: suppose the judge correctly approves true passes 95% of the time but correctly flunks true failures only 80% of the time — a very ordinary asymmetry; LLM judges are chronically credulous. If the true pass rate is 60%, the judge’s expected reported rate is 0.60·0.95 + 0.40·0.20 = 65%. Out of every hundred outputs it waves through eight failures (20% of the forty) and flunks three passes (5% of the sixty): a net five points of pure inflation. And this judge’s overall agreement on your validation set? 0.60·0.95 + 0.40·0.80 = 89% — a number most teams would celebrate.

true pass rate

60%

→

the LLM judge

TPR (catches passes): 95%

TNR (catches fails): 80%

chronically credulous

→

+5 pts

judge-reported rate

65%

overall agreement: 89% — a number teams celebrate8 failures waved through, 3 passes flunked (per 100)

A good-looking judge can still systematically inflate your metric: asymmetric errors add five points here, and grading more outputs only tightens the interval around the wrong number.

Worse, the bias does not shrink when you grade more outputs. Judge 5,000 instead of 200 and the confidence interval tightens obediently around 65% — you have become precisely wrong. Judge accuracy is a property of the instrument. Estimate accuracy is a property of what you do with its readings. This post is about the second thing: two corrections — one sitting in epidemiology journals since 1978, one published in Science in 2023 — that turn a biased judge plus a small human-labeled set into an unbiased estimate with an honest confidence interval. Five-plus papers, and as far as I can find, zero blog-level explainers. Let’s fix that.

Validate like an epidemiologist

Your LLM judge is a diagnostic test. It examines a specimen (an output) and returns positive or negative for a condition (pass). Epidemiology figured out how to live with imperfect diagnostic tests decades ago, and its first rule is: never characterize a test with a single accuracy number. Characterize it per class, against held-out human labels the judge has never influenced:

• TPR (sensitivity): of the outputs humans call pass, what fraction does the judge also call pass?
• TNR (specificity): of the outputs humans call fail, what fraction does the judge also call fail?

A single “agreement” number is the prevalence-weighted blend of the two — θ·TPR + (1−θ)·TNR — so it hides exactly the asymmetry that causes bias, and it silently changes whenever your output distribution shifts, with no change in the judge at all (part 5’s prevalence paradox, same disease, new host). Hamel Husain’s evals FAQ — deservedly canon at this point — teaches exactly this: validate judges with TPR and TNR on held-out human labels. And it takes one tantalizing step further, noting that with those numbers in hand you can correct the judge’s estimates to recover the true failure rate. The FAQ says the correction exists; it doesn’t print it. Consider the rest of this post the missing appendix.

The correction has been sitting there since 1978

Write the bias down as arithmetic and the fix falls out. What the judge observes is a mixture: true passes it correctly approves, plus true failures it mistakenly approves.

p_obs = θ · TPR + (1−θ) · (1−TNR)

One equation, one unknown. Solve for θ:

θ̂ = (p_obs + TNR − 1) / (TPR + TNR − 1)

This is the Rogan–Gladen estimator, published in 1978 to correct disease-prevalence estimates from imperfect screening tests. Your judge is a screening test; your pass rate is a prevalence. Worked example, with the numbers from the cold open: the judge reports a 65% pass rate over 5,000 outputs; your validation set says TPR = 0.95, TNR = 0.80.

θ̂ = (0.65 + 0.80 − 1) / (0.95 + 0.80 − 1) = 0.45 / 0.75 = 60%

Five points of inflation removed by one line of algebra. (If you took the popular evals course: the judgy library implements exactly this.) Read the pieces aloud: the numerator is what the judge said (65%) plus how often it catches real fails (80%) minus one; the denominator, how often it catches real passes (95%) plus how often it catches real fails (80%) minus one.

Epidemiology solved your judge problem in 1978 — the observed rate plus two per-class error rates recover the true one in a line of algebra.

Three honest caveats. First, TPR and TNR are themselves estimates from a small human-labeled set, and their sampling error propagates into θ — so don’t report the corrected point estimate naked; bootstrap the whole pipeline (resample the validation set and the judged set together, take percentile intervals; the widget below does this). Second, look at that denominator: TPR + TNR − 1 is Youden’s J, a judge’s distance from a coin flip. As it approaches zero the correction divides by nearly nothing and the estimate explodes — no algebra can rescue an uninformative judge, and the formula can return values outside [0, 1]; you clamp, and you treat the clamp as a warning that your validation set is too small or your judge too weak. Third, the formula is binary-only.

Prediction-powered inference: the crown

Rogan–Gladen is a fine emergency repair, and knowing it exists already puts you ahead of most eval reports. But it is binary-only, its confidence interval has to be bolted on with a bootstrap. There is a modern estimator that fixes all three at once — and it’s the one I’d actually build on.

Prediction-powered inference (Angelopoulos, Bates, Fannjiang, Jordan & Zrnic, Science 2023) was designed for exactly the shape of our problem: a huge set of cheap machine predictions, a small set of expensive gold labels, and the desire to use both without lying to yourself. The setup: you have N outputs graded only by the judge (N is large — judges are cheap), and n outputs drawn uniformly at random from the same pool where you have both the judge’s verdict f and a human label y (n is small — humans are expensive). The estimator for the true pass rate is:

θ̂_PP = (1/N)·Σ f(x_i) − [ (1/n)·Σ (f(x_j) − y_j) ]
        └─ judge average ─┘   └──── rectifier ────┘

Read it aloud: the judge’s average, minus the judge’s measured bias. The rectifier is the average amount by which the judge over-calls relative to humans on the paired subset — in our running example it would come out around +5 points, and subtracting it lands you back at the truth. The expectation of the first term is p_obs; the expectation of the second is p_obs − θ; the difference is θ, unbiased, no matter how bad the judge is. Notice what’s absent: no TPR, no TNR, no model of the judge’s error process, no equation to invert. You measure the bias directly and subtract it — which is also why PPI generalizes past binary verdicts to rubric scores, numeric ratings, means of anything bounded, while Rogan–Gladen stays stuck at 2×2.

The confidence interval is just as clean. The two terms are computed on independent samples, so their variances add:

Var(θ̂_PP) = σ²_f / N + σ²_{f−y} / n
  σ²_f  = sample variance of judge verdicts on the N unlabeled items
  σ²_{f−y} = sample variance of (f_j − y_j) on the n paired items
  95% CI:  θ̂_PP ± 1.96 · √Var(θ̂_PP)

The first term is tiny — N is huge and judge labels cost fractions of a cent. The second term is the real price, and here is the elegant part: σ²_{f−y} measures how often the judge disagrees with humans. Three properties follow, and they’re the whole reason this estimator deserves to be standard practice:

1. Validity is unconditional. The CI has correct coverage whether your judge is superb or a coin flip. You cannot be burned by trusting it.
2. The better the judge, the tighter the interval. A judge that mostly agrees with humans makes f − y mostly zero, shrinking σ²_{f−y} toward nothing. A good judge literally earns you effective sample size; a bad one costs you nothing but width.
3. It beats throwing the judge away. Humans-alone gives variance σ²_y/n; PPI gives σ²_{f−y}/n (plus the negligible first term). Whenever the judge has real signal, the disagreement f − y varies less than y itself, and PPI wins — same n as humans, tighter interval. (For a pathologically bad judge, plain PPI can lose; the tuned variant — “power tuning,” a scalar λ on the judge term — provably never does worse than humans-alone. Use a library, get λ for free.)

The follow-up literature has already specialized this to LLM evals: stratified PPI for hybrid human/judge language-model evaluation (NeurIPS 2024), and adaptive versions that choose which items to send to humans (2025). But the base recipe is five steps:

1. Judge everything. Run the (frozen — see below) judge over all N outputs.
2. Sample n items uniformly at random from that same pool — 100–200 is a realistic start — and collect human labels. Keep the judge’s verdicts on those same items. The randomness is load-bearing: a convenience sample (“the ones that looked fishy”) biases the rectifier, and then nothing here protects you.
3. Compute the rectifier: the mean of (judge − human) over the n paired items.
4. Subtract: θ̂_PP = judge’s mean verdict − rectifier.
5. Report the CI: θ̂_PP ± 1.96·√(σ²_f/N + σ²_{f−y}/n). That number — not the raw judge pass-rate — is the metric.

the judge grades everything

N = 5,000

cheap — fractions of a cent

drawn at random

humans grade a random slice

n = 100

gold — expensive

the rectifier

the judge’s measured bias:

mean(judge − human)

→

θ̂_PP

unbiased estimate

honest, tight CI

estimate = judge average − measured bias

A small human sample buys back the judge’s bias — and keeps the judge’s scale: N does the narrowing, n does the truth-telling.

The two headline behaviors: the naive judge bar is the narrowest and the most wrong — at the defaults its “95%” interval contains the truth in roughly none of the 300 experiments, the purest form of confidently wrong that exists in eval reporting. And PPI’s coverage pins to ~95% at every slider setting while its width floats between two anchors: a perfect judge buys you nearly the full N, a useless one leaves you roughly where your n humans already were. Plain PPI — what this widget runs — can give back a sliver of that width to a genuinely bad judge (power-tuning removes even that); mostly the only question is how much you win.

Judge discipline: field notes

Everything above assumes the instrument you validated is the instrument that graded. That assumption dies quietly in practice, so — some patterns from eval builds I’ve been around, none of which appear in the papers:

• Freeze the judge before data collection. On a leaderboard build I watched, the working norm was that the judge prompt froze before the first task was authored. Editing the judge mid-eval is swapping the thermometer mid-experiment: every verdict before the edit lives on a different scale from every verdict after, and your TPR/TNR — measured on which version, exactly? — transfers to neither. Version the judge prompt like a schema; a change means a new validation run and a new rectifier.
• Stress-test with planted lies. The best judge calibration protocol I’ve seen: an engineer building a claim-level verifier injected fabricated false claims into otherwise-real answers and required 100% detection before the judge was trusted with anything. Not 95% — a judge that misses planted, known-false claims at validation time will miss subtler ones at scale, in exactly the direction that inflates your metric.
• Watch for context loss. The same build surfaced a failure mode worth stealing: judging decomposed atomic claims in isolation strips the cross-sentence qualifier that made a claim true, so the judge marks correct claims wrong. The tempting fix — show the judge the full response — has its own documented failure: with full context the judge starts making its own inferences and drifts off the rubric. There’s no free lunch, only a deliberate choice; whichever you choose, your TPR/TNR must be measured on that configuration.
• Assume position, verbosity, and self-preference bias. Pairwise judges systematically prefer the first-listed answer, the longer answer, and their own model family’s answers (Zheng et al.’s MT-Bench paper measured all three). The standard mitigation is cheap: run every comparison both ways and average; a verdict that flips when you swap the order isn’t a verdict, it’s noise — score it a tie.
• Pin the judge’s model version. A judge pointed at a -latest API alias is an instrument that silently recalibrates itself whenever the provider ships an update. Your rectifier and your TPR/TNR describe last month’s judge. Pin the version; when you deliberately upgrade, re-validate and re-estimate before trusting a single aggregate.

The same sensor, twice

If this post felt structurally familiar, that’s because it is part 5 again with the labels swapped. Human raters and LLM judges are the same object: noisy sensors pointed at a construct you can’t observe directly. And the answer is the same two-step, both times: measure the sensor against something better (gold sets and agreement for raters; held-out human labels for judges), then correct the estimate and carry the sensor’s uncertainty into the error bar (Dawid–Skene there; Rogan–Gladen and PPI here). What you never get to do — with either sensor — is read the raw output and call it the measurement.

Next up, the numbers stop being numbers and start being rankings: Bradley–Terry, arena leaderboards, why ranks flip when nothing changed, and what a confidence interval on “#3” even means. That’s part 7.

What this means for your eval

• Never report a raw judge pass-rate as the metric. Report a debiased estimate with a CI that accounts for the judge’s errors.
• Validate per class — TPR and TNR on a held-out human-labeled set (aim for ≥50 examples of each class). A single agreement number hides the asymmetry that biases your aggregate.
• Quick fix: Rogan–Gladen, θ̂ = (p_obs + TNR − 1)/(TPR + TNR − 1). Bootstrap the CI; if TPR + TNR − 1 is near zero, your judge is a coin flip and no correction can save it.
• Better fix: PPI — judge everything, human-label a small uniform random subset, subtract the measured bias, report θ̂_PP ± 1.96·√(σ²_f/N + σ²_{f−y}/n).
• The paired human subset must be a random draw from the judged pool. Convenience-sampling “suspicious” items biases the rectifier and voids the guarantee.
• Freeze and version the judge prompt before collection; any change means a new validation run and a new rectifier.
• Stress-test the judge with injected known-false claims; require 100% detection before trusting it at scale.
• Swap presentation order on pairwise judgments; treat flip-on-swap as a tie. Pin the judge’s model version and re-validate on every upgrade.

References & further reading

1. W. J. Rogan & B. Gladen, Estimating prevalence from the results of a screening test, American Journal of Epidemiology 107(1), 1978 — the original correction.
2. A. N. Angelopoulos, S. Bates, C. Fannjiang, M. I. Jordan & T. Zrnic, Prediction-Powered Inference, Science 382:669–674 (2023) — the framework; PPI++ / power tuning in the follow-up (arXiv 2311.01453).
3. Stratified Prediction-Powered Inference for Hybrid Language Model Evaluation (NeurIPS 2024) — PPI specialized to LM evals with stratified human sampling.
4. Adaptive prediction-powered autoevaluation (2025) — actively choosing which items get human labels.
5. L. Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023) — position, verbosity, and self-preference biases, measured.
6. Hamel Husain, FAQ on AI Evals — the workflow canon; teaches TPR/TNR judge validation and names the correction this post supplies.
7. judgy — a small Python implementation of the Rogan–Gladen correction with bootstrap CIs, from the popular evals course.

---

Beyond Error Bars · Part 6 of 9 · Previous: Humans Are Instruments · Next: Rankings Under Uncertainty

Views mine.