1.4 Statistical Repeat Evaluations & Non-Deterministic Variance

How to evaluate AI agents across multiple iterations (N repeats) with pass threshold percentages to eliminate prompt flakiness.

Why Statistical Evaluation is Essential for LLMs

Because LLM temperature > 0 causes stochastic variance, running an agent test once does not prove reliability. An agent might pass 9 times and hallucinate on the 10th. CertiQA allows you to configure Repeat Count (N) and Pass Threshold (T) sliders directly on the canvas.

Setting Repeat Count and Threshold

• 1. Open any Test Case settings on the right panel.
• 2. Adjust the “Repeat Count” slider (e.g. 5 iterations) – this is how many independent times the whole case runs, not how many times a single assertion is checked.
• 3. Adjust the “Pass Threshold” slider (e.g. 80% pass required) – the minimum share of those iterations that must succeed for the case as a whole to pass.
• 4. Click “Run Case”. CertiQA computes the pass percentage once every iteration has reported back.

How many repeats is enough depends on what stage of the pipeline the case sits in. Early in development, when you’re iterating on a prompt and mainly want a quick directional signal, 3-5 repeats at a permissive threshold is usually enough to catch an obviously broken change without burning excessive model calls on every save. For a release-gating suite that actually blocks a deploy, more repeats narrow the Wilson interval below and make the pass/fail call itself trustworthy rather than just fast – 10-20 repeats at a stricter threshold is a reasonable starting point, tuned down over time once you have a feel for how noisy a given case actually is in practice.

pass^k and the Wilson Confidence Interval

A plain pass rate over a handful of repeats looks more precise than it is: “80%” from 4/5 runs and “80%” from 400/500 runs are not the same claim. CertiQA reports two additional statistics alongside the plain rate. pass^k (the tau-bench estimator) is the unbiased probability that k independently drawn trials from this run would ALL pass – a stricter, more conservative read on reliability than the single-run rate. The 95% Wilson score interval gives the plausible range for the true pass rate given how few trials were actually run; with only 5 repeats that range is wide, and CertiQA shows it rather than pretending the point estimate is exact.

Repeat Pass Rate: 80% (4/5)  pass^2 60%  pass^3 40%  [95% CI 38–96%]  ✅ PASS

Gating changed accordingly: a case used to fail whenever the plain pass rate fell below its Pass Threshold. Now it fails only when the plain pass rate misses the threshold AND the Wilson interval’s upper bound also falls short – a 3/5 run against an 80% threshold has an upper bound above 80%, so it is not statistically distinguishable from passing, and failing CI on it just trains people to ignore CI. Pass –strict-threshold to the CLI to restore the old point-estimate-only gate. The run report shows pass^2/pass^3 and the confidence interval in the repeat-pass-rate chip’s tooltip.