1.3 LLM-as-a-Judge Semantic Rubrics & Groundedness Scoring

How to write plain English semantic rubrics to evaluate AI agent answers for groundedness, accuracy, and tone.

Why an Answer Needs to Be Graded, Not Just Compared

Conversational models phrase the same correct answer differently every time – “Your order shipped yesterday” and “It went out for delivery yesterday” mean the same thing but are not the same string. Asserting text === “…” against that kind of output fails constantly, and loosening it into a contains check just trades false failures for false passes on badly-worded answers. LLM-as-a-Judge sidesteps the whole problem: instead of comparing text to text, an independent evaluator model reads the agent’s response and grades it against criteria you write in plain English, the same way a human reviewer would – but repeatably and at test-suite scale.

Three Rubric Types, Each Answering a Different Question

Rubric Type	What It Checks	Typical Use
GROUNDEDNESS	Whether every factual claim in the response actually traces back to the provided reference context or mock tool outputs – catches hallucination even when the answer sounds confident and well-formed.	RAG pipelines and any agent whose answers should only ever repeat what its tools actually returned.
RESPONSE_QUALITY	Clarity, helpfulness, and correctness against the specific criteria you describe – a broader quality bar than groundedness alone.	General conversational correctness where hallucination isn’t the primary risk but a vague or unhelpful answer is.
GUARDRAIL	Whether the agent adhered to a safety or policy instruction it was supposed to follow – refused what it should refuse, disclosed what it must disclose.	Compliance-sensitive agents where the thing under test is a boundary, not an answer’s content.

Inline Rubrics vs. a Saved Rubric

A rubric can be typed directly into the step (inlineRubric) or referenced by id from a rubric you’ve saved once and reuse everywhere (judgeRubricId) – when a step carries both, the saved rubric wins, so a shared team-maintained rubric always takes precedence over a stray inline override left behind from earlier editing. Saved rubrics matter most once you have more than a handful of cases checking the same kind of thing: Golden Benchmarking datasets (see 1.8) reference a rubricId per row for exactly this reason, so hundreds of dataset rows can share one grading definition instead of hundreds of near-duplicate inline strings that would drift out of sync with each other over time.

Trusting the Judge: Calibration and Ensembles

A judge is itself a model, which means it can be wrong – confidently. Each saved rubric carries a calibration summary (agreement rate, Cohen’s kappa against human-labeled examples, and sample size) with a low-agreement badge if the judge and human reviewers disagree more than expected, so a rubric with a hidden ambiguity in its wording surfaces that fact instead of quietly grading wrong forever. When a verdict in a Run Report looks wrong to you, use “📌 Promote to calibration example” on that turn to file it as a labeled example against its rubric – this is how a rubric’s calibration score actually improves over time, by feeding real disagreements back into it rather than just noting them and moving on.

For a rubric where a single grading call feels noisy – borderline answers that could plausibly go either way – set ensembleSize above 1 to grade the same response with multiple independent judge calls instead of one. The ensemble takes a majority vote for pass/fail (a tie resolves to fail, deliberately, since under-reporting a real failure is the worse mistake for a testing tool to make), averages the numeric score, and reports the variance across members. The transcript drawer flags a verdict with a “⚠ High variance” badge once that variance crosses 0.15 – a visible signal that the members disagreed enough to warrant a second look rather than trusting a single grading pass at face value.

Configuring a Judge Step

• 1. Add an “EXPECT_RESPONSE” step on the canvas.
• 2. Select the Rubric Type – GROUNDEDNESS, RESPONSE_QUALITY, or GUARDRAIL – based on which of the three questions above this step is actually meant to answer.
• 3. Either type an Inline Rubric in plain English, or pick a saved rubric from your project’s rubric library if one already covers this criteria.
• 4. For GROUNDEDNESS checks specifically, also fill in referenceContext so the judge knows which facts were actually available to be grounded in.

Things That Make a Rubric Grade Well or Badly

• Rubric Specificity: “Answer is good” gives the judge nothing concrete to check against and produces inconsistent verdicts run to run. “Response states the package was delivered to the front porch on Tuesday” gives it something it can actually verify.
• Reference Context for RAG: without referenceContext, a GROUNDEDNESS check has no ground truth to compare the answer against and can only fall back to general plausibility – always set it for retrieval-augmented agents, not just when a test happens to fail without it.
• A judge verdict is a grade, not a certainty – for anything safety- or compliance-critical, treat a low calibration score or a high-variance ensemble result as a reason to have a human review that specific rubric’s wording, not as noise to ignore.