9.1 Online Evaluation & Drift Monitoring

How to keep grading a shipped agent’s real production traffic after the test suite has already passed, and get flagged automatically when its behavior drifts.

Testing Doesn’t Stop at the Merge

A test suite (Surface 1) tells you an agent behaved correctly against the scenarios you thought to write. An online evaluation policy keeps checking a sample of its live production traces against the same kind of deterministic matcher used by ASSERT_OUTPUT – contains, notContains, regex, notRegex, jsonPath+equals – against the agent’s recorded OpenTelemetry GenAI trace, so a behavior change that only shows up in real traffic (a prompt change upstream, a model swap, a tool that started returning a slightly different shape) gets caught between test runs, not at the next release.

Standing Up a Policy

• 1. Open the 📡 Online Eval tab.
• 2. Create a policy: name it, set a sample rate (what fraction of new production traces get graded), and add one or more matchers.
• 3. The policy scores newly ingested traces on a schedule – no LLM judge call in this path, so scoring stays cheap enough to run continuously.
• 4. Review the scored-trace list: pass/fail per trace, and which matcher fired on a failure.

Drift Alerts

Each policy can also watch its own pass rate over time: a trailing 7-day baseline compared against a trailing 24-hour current window, using a two-sided statistical significance test (the same kind of test used for run-over-run case comparisons – see “Run-over-Run Significance” in the flaky-quarantine playbook) so a noisy day doesn’t read as drift. A statistically significant drop opens a drift alert, visible on the policy’s dashboard and acknowledgeable once investigated; a per-policy cooldown keeps one ongoing drift from paging the same alert repeatedly.

Where the Online Scorer Still Has Gaps

• Matchers here are the same vocabulary as ASSERT_OUTPUT, with one gap: jsonSchema matchers are not evaluated in the online scorer today – a policy matcher using jsonSchema is rejected at save time rather than silently skipped at scoring time.
• Drift alerts are surfaced in-app, not pushed to Slack/email yet – check the dashboard, or poll the drift-alerts endpoint from your own alerting, until push delivery ships.
• A drift alert tells you a policy’s pass rate moved; it doesn’t tell you why. Pair it with Trace Promotion to pull the failing traces straight into a dataset you can debug and regress against.