8.3 LLM-as-a-Judge Calibration & False Failure Triage

How to calibrate LLM judge rubrics and eliminate false negative evaluations.

Two Different Failure Modes, One Rubric

A miscalibrated judge fails in two directions that look nothing alike and need different fixes. A false fail – the judge rejects a genuinely good response – wastes triage time and erodes trust in the whole suite, since people start re-running or ignoring judge-graded cases instead of trusting a fail. A false pass – the judge accepts a response a human would flag – is the more dangerous of the two for a testing product specifically: it means the suite is quietly lying about a regression, since nothing else is checking a stochastic response the way a deterministic assertion would. Calibration exists to measure both, not just whichever one happens to be annoying you today.

Reading a Calibration Run

Running Calibrate against a rubric re-grades every labeled example through the real judge configuration – the same code path production grading uses, at the same model cost, deliberately not a cheaper mocked comparison, because the point is measuring how the actual judge agrees with human labels, not a stand-in for it. The rubric card in Judge Rubric Manager then shows agreement rate, Cohen’s kappa, sample size, and the last calibrated date. Kappa matters alongside raw agreement because agreement alone is misleading on a lopsided example set – a rubric where 95% of labeled examples are genuinely passing responses gets 95% agreement from a judge that blindly says pass every time, and kappa is what catches that the judge isn’t actually discriminating between good and bad responses at all. The card flags itself amber under 0.80 agreement or a kappa below 0.6, and red under 0.65 agreement – treat either as a signal to review labeled examples and tighten rubric wording before trusting that rubric’s verdicts on new cases.

Of the two directional error rates a run also reports, false-pass rate – of the examples humans labeled as failing, what fraction the judge wrongly passed – deserves more attention than false-fail rate: a judge with a high false-fail rate just generates annoying noise, while a high false-pass rate means the rubric is actively hiding real regressions from everyone relying on it.

Closing the Loop From a Real Disagreement

Rather than hand-authoring the labeled example set as a separate chore, the “📌 Promote to calibration example” button on any judge verdict in the Transcript Drawer lets a human disagreement become a labeled example on the spot – the next Calibrate run measures the rubric against that exact real disagreement, so the calibration set grows from cases people actually cared enough to flag rather than a synthetic sample.

Triaging a Suspected False Failure

• 1. Open the Transcript Drawer on the failed verdict and read the judge’s own reasoning – it usually says which part of the rubric it thinks wasn’t satisfied, which is faster than re-reading the response cold.
• 2. If the reasoning reveals the rubric wording is ambiguous or too strict, tighten it in plain English rather than adding more rubric text – a longer rubric isn’t a more precise one.
• 3. For RAG or retrieval-grounded responses, attach referenceContext so a GROUNDEDNESS rubric grades against what was actually retrieved, not against the judge’s own background knowledge of what the right answer should be.
• 4. Promote the disputed turn to a calibration example either way – agreeing or disagreeing with your own correction, the next calibration run is how you find out whether the rubric change actually helped.

Calibration Spends Real Budget

• A calibration run grades every labeled example through the real model, at real cost – it’s gated behind the AUTOMATER role for that reason, while adding a labeled example only needs AUTHOR and reading calibration history only needs project read access.
• Rubrics can be org-shared, not just project-owned – a calibration run measures the rubric itself, so improving one used across several projects improves grading everywhere it’s referenced, not just in the project you happened to be looking at.