Trial policies
Use a trial policy when output can vary between otherwise identical calls.
export const measure = { trials: 5, verdict: { atLeast: 4 }, earlyStop: false, errorRetries: 1,};verdictis'all','majority'or{ atLeast: n }, counted overtrials.earlyStopstops when the remaining trials cannot change the verdict.errorRetriesadds trials for infrastructure errors only; errors never count towardtrials.- Put these fields at the top level of
defineEvalordescribeEval: there is nopolicykey. The core reference has the defaults and the validation rules. ctx.skip(reason)is not a trial. Before a verdict it skips the case; after a verdict it keeps that verdict. Its tokens still count toward a token budget.- Give a case that needs another bar its own eval with a distinct name, not a per-case policy:
describeEval('support-triage-strict', { task, trials: 20, verdict: { atLeast: 16 } }, declare). Each eval is its own run and report series. To compare runs, use the sametrialsandearlyStop: false.
What a majority verdict tells you
Section titled “What a majority verdict tells you”Use majority for a smoke signal, not as a rate measurement.
| True pass rate | Majority of 3 | Majority of 5 |
|---|---|---|
| 50% | 50% | 50% |
| 60% | 65% | 68% |
| 70% | 78% | 84% |
| 80% | 90% | 94% |
| 90% | 97% | 99% |
At a true 70% pass rate, majority-of-three passes 78% of runs. Compare fixed trial pass rates with intervals; one trial is not a gate for a nondeterministic task.