Skip to content

Trial policies

Use a trial policy when output can vary between otherwise identical calls.

export const measure = {
trials: 5,
verdict: { atLeast: 4 },
earlyStop: false,
errorRetries: 1,
};
  • verdict is 'all', 'majority' or { atLeast: n }, counted over trials. earlyStop stops when the remaining trials cannot change the verdict. errorRetries adds trials for infrastructure errors only; errors never count toward trials.
  • Put these fields at the top level of defineEval or describeEval: there is no policy key. The core reference has the defaults and the validation rules.
  • ctx.skip(reason) is not a trial. Before a verdict it skips the case; after a verdict it keeps that verdict. Its tokens still count toward a token budget.
  • Give a case that needs another bar its own eval with a distinct name, not a per-case policy: describeEval('support-triage-strict', { task, trials: 20, verdict: { atLeast: 16 } }, declare). Each eval is its own run and report series. To compare runs, use the same trials and earlyStop: false.

Use majority for a smoke signal, not as a rate measurement.

True pass rate Majority of 3 Majority of 5
50% 50% 50%
60% 65% 68%
70% 78% 84%
80% 90% 94%
90% 97% 99%

At a true 70% pass rate, majority-of-three passes 78% of runs. Compare fixed trial pass rates with intervals; one trial is not a gate for a nondeterministic task.