Building a Product Eval, Instrumented

The part I won’t pretend to invent

This is the post where the series cashes out. Eight posts of measurement machinery — variance, power, raters, judges, rankings, repetition — and the obvious question: fine, but how do I actually build the eval for my product?

The skeleton of the answer is not mine, and I’m not going to pretend otherwise. Hamel Husain’s error-analysis loop — read your traces, code the failures into a taxonomy, turn the taxonomy into rubrics, automate the stable rubrics with judges, let the validated judges gate releases — is the settled canon of product-eval practice, and it’s settled because it’s right. If you haven’t read his evals essay and the FAQ, read them before this. I have nothing to subtract from that loop.

What I have is what the canon under-specifies. Every step of the loop is a measurement act, and the FAQ-level treatment gives you the workflow without the instrument spec: how many traces are enough, when a rubric is reliable enough to hand to strangers, what a judge’s agreement number actually licenses, how small a regression your gate can actually see. Those are exactly the questions parts 3–8 built tools for. So this post is what its title says — Hamel’s loop, instrumented: one pass through the build with the toolbox bolted on at each step, then the failure mode none of the canon covers (a war story of mine about the difference between ranking and probability), and finally the governance layer that keeps the whole assembly trustworthy after you ship it.

The canon loop, with measurement bolted on at every station. The loop is Hamel’s; the badges are parts 3–6 of this series, plus the governance pass that keeps the cycle honest.

Step 1 · Error analysis: read until saturation

The canon says: don’t start with metrics; start by reading traces. Open-code the failures in your own words, then cluster the notes into a taxonomy. The FAQ’s stopping heuristic: keep reading until new failure modes stop appearing — in practice, when a stretch of ~20 fresh traces teaches you nothing new.

What to measure: that heuristic is a stopping rule, and it’s worth seeing it as one — the same logic as part 4’s sequential tests, applied qualitatively. You are estimating the probability that an unseen failure mode still exists, and every fresh trace that adds nothing is evidence that probability is small. (The rate at which you’re still seeing first-time failure modes is itself the estimate — readers who want the formal version can look up species-discovery estimators; the folk rule is a decent approximation of them.) One refinement matters in practice: saturate within each traffic segment you care about, not overall. A failure mode confined to 5% of traffic will not surface quickly in a uniform sample, and “we read 100 traces” means little if all 100 came from the easy pool.

The second measurement rule of this step: a taxonomy category is only finished when it’s operationalizable — a question a rater can answer about a single trace, with evidence they can point to. “Response is unhelpful” is a sentiment; “response does not address the user’s stated constraint” is a rubric item. And expect your categories to move: Shankar et al. documented criteria drift — graders discover and revise their own criteria while grading — as a systematic phenomenon, not a personal failing. Plan for revision; version it (step 2).

Step 2 · The rubric is a multi-item instrument

The canon says: turn the taxonomy into rubrics; prefer binary pass/fail per failure mode over Likert scales.

What to measure: a rubric is a multi-item measurement instrument — the exact object psychometrics spent a century learning to build — and it deserves the minimum viable version of that discipline: pilot it on ~20 items with 2 raters before production. Not to compute a kappa you can frame; to harvest the disagreements. Part 5’s triage applies verbatim: each disagreement is either rubric ambiguity (fix the wording), rater error (fix the training), or genuine item ambiguity (the item is the problem — split it or drop it). Twenty items and two raters will find the worst of all three for the price of an afternoon.

Anchor the scale to outcomes, not preferences. Part 5’s central story was a rubric whose agreement jumped when the question was re-anchored from “which response do you prefer?” to “would the customer accept this work?” — re-anchoring bought more agreement than any amount of rater training. And the binary-over-Likert advice, which Hamel argues behaviorally, has a measurement-theory reading: every rubric item is a sensor, and a 1–5 scale on which raters can’t distinguish a 3 from a 4 is a noisy sensor pretending to be a precise one. Ask five binary questions and sum them; you recover the gradation with far more reliability per item.

Finally: version the rubric like code. A wording change is an instrument change, and scores from different instrument versions are different units. You wouldn’t silently swap thermometers mid-experiment; don’t silently edit the rubric mid-quarter. (What to do when you must: governance, below.)

Step 3 · The dataset: clusters, coverage, and life without a gold set

The canon says: curate eval items from real failures; use synthetic data to fill coverage gaps.

What to measure: two things at build time, because you can’t reconstruct them later. First, representativeness — sample against production traffic strata, and record the sampling frame. Second, clustering keys: seed prompt, source document, user, conversation, template. Part 3’s lesson was that items sharing a parent are correlated and your confidence intervals must know it; the metadata costs nothing to keep at generation time and is gone forever if you don’t.

Then there’s the case the textbooks skip: no gold set exists. For open-ended product tasks — write the research memo, plan the migration, draft the report — two competent people produce different, both-defensible outputs. That’s a property of the work, not a gap in your eval. The pattern I’ve seen hold up in serious enterprise eval work is two strictly separated tiers:

1. Trust floors, which gate. Must-not-violate properties: every claim grounded in the provided sources, nothing unsafe, output contract respected. These are checkable near-deterministically — string-level and retrieval-level checks tuned for a very high true-negative rate — and a violation blocks the release, full stop.
2. Value metrics, which hill-climb. Usefulness, coverage, depth, insight. These are judge-scored, noisier, and directional: you track them over versions and try to push them up.

The design rule that carries all the weight: never average the two tiers into one number. A fabricated citation is not offset by eloquence, and any composite that lets it be offset has smuggled a policy decision into an arithmetic mean (part 2’s oldest warning). A related habit worth keeping from the same world: teams often replicate every judge-scored metric a few times and let only the stable ones — run-to-run σ near zero — participate in gating, demoting the noisy ones to diagnostics. That’s an improvised reliability analysis, and it’s worth doing and worth knowing what it doesn’t buy you: σ across reruns measures decoding noise, not sampling error. Part 3 has the full argument; both variances need reporting.

Trust floors — GATE

• every claim grounded in sources
• nothing unsafe
• output contract respected

verdict: any failure blocks the release

Value metrics — HILL-CLIMB

• usefulness
• coverage
• depth & insight

verdict: optimize — don’t gate on these

never average these two into one number

A gate failure isn’t offset by eloquence. Floors block, value metrics climb — any composite that mixes them has smuggled a policy decision into an arithmetic mean.

Step 4 · The judge: validate per class, freeze, stress-test, correct

The canon says: automate only the persistent failure modes with an LLM judge, and validate the judge against held-out human labels — TPR and TNR, not vibes.

What to measure: four disciplines, in order. Validate per class, because overall agreement is a trap: at a 90% pass rate, a judge that waves everything through scores 90% agreement while catching zero failures — the prevalence paradox from part 5. Freeze the judge — pin the model version and prompt, and treat any change as a new instrument requiring re-validation. Stress-test with injected failures: take passing outputs, plant known violations (a fabricated number, a dropped constraint), and measure the catch rate. This is the sensor test with a known signal, and it’s the only honest way to trust a TNR measured on a dataset where failures are rare. Then — the step the canon points at but never performs — correct the aggregate. A judge with known error rates gives you a biased pass rate, and the bias is fixable: Rogan–Gladen inverts it in one line, and prediction-powered inference combines a few hundred human labels with many judge labels into an unbiased estimate with honest intervals. Hamel’s FAQ says the correction exists; part 6 is the recipe, with the widget to build intuition.

Step 5 · The gate: powered, paired, peek-proof

The canon says: run the eval in CI; block releases that regress.

What to measure: a release gate is a hypothesis test whether or not you admit it, so give it the courtesies of one. Pair it: run candidate and incumbent on the same items and test the paired differences (McNemar for binary — part 4); pairing routinely halves the n you need, and it’s free. Power it for the decision’s minimum effect: decide the smallest regression worth blocking — say three points — and size the eval so the gate can actually see it. A gate that can only detect ten-point swings on an 80-item suite is a placebo with a green checkmark. Make it sequential-safe: teams peek — after every batch, on every PR, on every rerun — and repeated testing at a fixed threshold inflates false alarms far past the advertised rate; part 4’s always-valid methods exist for precisely this. And make the intervals cluster-aware (part 3), because eval items that share a seed template will make a naive gate overconfident in both directions. Last, decide the tie policy before you see scores: when the interval spans zero, does the release ship or wait? That’s a product decision, and writing it down beats relitigating it at 6 p.m. on release day.

Calibration ≠ discrimination: the eval that said my model was great

Everything above instruments the loop. This section is about the failure mode that walked through all of it untouched, in a system I own.

I maintain a quality-prediction model — small, unglamorous: it looks at a unit of work in an operational pipeline and predicts the probability that it will turn out badly. The eval story was clean. On a deliberately hard generalization split — held-out cohorts, a later time window — it ranked with an AUROC around 0.75. For a screening problem, that’s genuinely useful signal, honestly measured. The eval looked great, and I said so.

Then I looked at what the score’s consumers actually did with it. One consumer ranked — sort the queue, review the top. Fine. Another thresholded: act when the probability crosses τ. And the probabilities, it turned out, were junk. The model emitted almost everything in the bottom half of [0,1] — the vast majority of predictions below 0.5 while the actual positive rate (its base rate) was in the mid-forties percent, the two classes’ mean predictions separated by a few hundredths. When I swept for the F1-optimal threshold, it collapsed toward zero — operationally indistinguishable from “predict the base rate for everyone.” And underneath that, the base rate itself was drifting: one new cohort’s positive rate came in at less than half of training’s, so a threshold tuned anywhere was miscalibrated somewhere else. The ranking consumers were fine. The threshold consumers were thresholding noise. Same model, same eval — and the AUROC I’d reported was simultaneously true and useless to half its audience.

Discrimination: fine

positives outrank negatives — AUROC 0.75

model score

Calibration: broken

every prediction sits below 0.5

model score (probability)

It ranks like a champion and lies about probabilities — different properties, different checks. Same model, same dots: rank consumers are fine; anyone thresholding is thresholding noise.

These are two independent properties, and the vocabulary matters. Discrimination: do positives outrank negatives? (AUROC’s question.) Calibration: when the model says 0.7, does the event happen 70% of the time? A model can be excellent at one and terrible at the other — Guo et al. showed modern neural networks are routinely like this, discrimination improving while calibration decays. The repair is old, cheap, and satisfying: post-hoc isotonic regression — fit a monotone non-decreasing map from raw scores to observed outcome rates on a validation split, check it on a held-out test split, per cohort where the data allows. Because the map is monotone, the ranking — and therefore AUROC — is preserved; only the values move. You grade the fix with a reliability diagram and expected calibration error, before and after. Below: feel it.

Now the lesson, generalized to LLM evals, because that’s why it’s in this series. The question to ask of every eval score you publish: does its consumer use it as a rank or as a value? If a rank — “pick the better model,” “review the worst outputs first” — discrimination is all you need, and parts 3–7 covered its statistics. But the moment a score is consumed as a value — a threshold (“escalate when risk > 0.6”), an SLA (“ship when mean helpfulness ≥ 4.2”), a weighted composite, a dashboard promising a probability — you owe it a calibration check. Judge scores are the worst offenders: a judge’s 8/10 is not a probability of anything until you calibrate it against outcomes. Two judges’ 8s are not the same 8; this quarter’s 8 is not last quarter’s. Part 6’s corrections fix the bias in aggregate rates; calibration fixes the meaning of individual values — you need both, and they are not the same repair. The check costs one reliability diagram, and the fix, when the diagram bends, is twenty lines of isotonic regression.

Governance: the eval is load-bearing now

An eval that gates releases is not analysis anymore; it’s infrastructure, and it deserves the operational discipline you’d give any other load-bearing system. Nobody blogs about this layer, but mature organizations converge on the same short list, and every item is stealable:

• Eval design review before build — the analog of system design review. What decision does this eval feed? What’s the minimum effect that decision cares about, and does the planned n resolve it? Who consumes ranks, who consumes values? What does a wrong ship cost versus a wrong block?
• Versioned rubrics and judges, with changelogs. A score is only interpretable relative to an instrument version; pin it the way you pin dependencies.
• A comparability plan for instrument changes. When the rubric or judge must change, re-score a reference set of past releases with the new instrument and publish the offset — bridge the series, don’t silently rewrite history.
• A named owner for drift. Datasets age, traffic shifts, providers quietly swap the model behind your judge. An eval without an owner rots on exactly the schedule nobody is watching.
• Scheduled re-audits of frozen environments. Freezing your eval set doesn’t freeze the world around it: a “hard retrieval” task silently becomes an easy lookup as the live web catches up, and scores rise without the system improving — the worst kind of eval rot, because it reads as progress. Re-audit on a calendar, not on an incident.
• A scorecard habit. One line per eval: what decision it feeds, what resolution it has (the minimum detectable effect at current n), when it was last validated against human judgment. Three columns. Most teams cannot fill in any of them from memory, which is the argument for the card.

A word on where the tooling fits, because at this point a reasonable reader is asking which platform does all this for them. The eval platforms — LangSmith, Braintrust, Arize, deepeval, and their cousins — own the pipes: trace logging, dataset management, judge runners, regression dashboards. Use one; hand-rolling that plumbing is a poor use of anyone’s time. But be clear-eyed about the division of labor: almost nothing in this series ships as a platform feature. No platform runs your power analysis, clusters your standard errors, debiases your judge, or checks your calibration — the measurement discipline is what you pour through their pipes. The one genuinely stats-forward capability worth shopping for is judge–human agreement tracking (LangSmith’s Align Evals is the visible example), which is part 6’s first step productized. The rest is still on you — which is, of course, why this series exists.

And one constraint the canon rarely prices in: cost. Once evals gate PRs they sit on the critical path, and latency and dollars become design parameters, not afterthoughts. The pattern that works is a cascade that mirrors the measurement hierarchy: deterministic trust-floor checks on every commit (milliseconds, effectively free), judge-scored value metrics on a sample at merge or nightly (minutes, real dollars), human audit weekly or on escalation (days, expensive). Each decision frequency gets the cheapest sensor with adequate resolution. An eval too slow or too expensive to run gets skipped — and a skipped eval measures nothing at all.

The instrument, assembled

The one-sentence version of this series: an eval is an instrument — it has variance (part 3), resolution (part 4), human sensors (part 5), bias you can measure and correct (part 6), comparative validity (part 7), repetition behavior (part 8) — and now an assembly manual. The loop was never the hard part; the field has had the loop for two years. The hard part is knowing what your instrument can and cannot tell you at every step of it, and that part now fits in a checklist.

One story left: how we got here. Five eras of evaluation, each one built to fix the failure of the last, each one creating the measurement problem the next had to solve — ending at the statistics wave this series was written to continue. That’s part 1, and it’s next.

What this means for your eval

• Read traces until new failure modes stop appearing — per traffic segment, not overall — and make every taxonomy category a question a rater can answer about one trace.
• Pilot the rubric on ~20 items with 2 raters before production; triage every disagreement (rubric, rater, or item); anchor scales to outcomes, not preferences.
• Record clustering keys (seed prompt, source doc, conversation, annotator) at dataset build time — your intervals will need them and you can’t reconstruct them later.
• No gold set? Two tiers: trust floors that gate (near-deterministic, high TNR) and value metrics that hill-climb (judge-scored). Never average them into one number.
• Validate the judge per class, freeze its version, stress-test it with injected failures, and correct the aggregate (Rogan–Gladen / PPI, part 6).
• Make the release gate paired, powered for the minimum effect the decision cares about, sequential-safe if anyone peeks, and cluster-aware. Write the tie policy down first.
• For every score, ask: rank or value? Any value-consumed score — thresholds, SLAs, composites — gets a reliability diagram; if it bends, fit isotonic on validation and check on test.
• A judge’s 8/10 is not a probability until you calibrate it against outcomes; don’t let anyone threshold it as if it were.
• Version rubrics and judges like code, with changelogs and a bridge plan before any instrument change.
• Keep a scorecard per eval — what decision, what resolution, when last validated, who owns drift — and budget it like infrastructure: cheap checks → judge → human, cascaded.

References & further reading

1. Hamel Husain, Your AI Product Needs Evals (2024) and the Evals FAQ (2025) — the canonical build loop this post instruments. Read these first.
2. Shankar, Zamfirescu-Pereira, Hartmann, Parameswaran & Arawjo, Who Validates the Validators? (2024) — criteria drift: graders revise their own criteria while grading.
3. Dawid & Skene, Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm (1979) — the original model of raters as noisy instruments (part 5’s backbone).
4. Zadrozny & Elkan, Transforming Classifier Scores into Accurate Multiclass Probability Estimates (KDD 2002) — isotonic regression as post-hoc calibration.
5. Guo, Pleiss, Sun & Weinberger, On Calibration of Modern Neural Networks (2017) — reliability diagrams, ECE, and why good discrimination coexists with bad calibration.
6. This series: part 3 (error bars, clustering), part 4 (power, sequential testing), part 5 (raters, disagreement triage), part 6 (judge debiasing, Rogan–Gladen, PPI), part 7 (rankings), part 8 (pass@k, pass^k).

---

Beyond Error Bars · Part 9 of 9 · Previous: Agent Evals: The Statistics of pass@k and pass^k · Next: Every Eval Era Created the Next Measurement Problem

Views mine.