7. Trials, outcomes and statistics
Each case runs up to trials counted trials, and a verdict rule turns their passes and fails into one case outcome (reference, guide).
Outcomes
Section titled “Outcomes”Only pass and fail count in pass rates. An error says nothing about quality, so it is in no rate. A thrown value is a fail unless it is about infrastructure (Errors). A case also ends in error when its counted trials did not decide the verdict and no more trial starts. A skip did not run to a verdict. A known-failure mark turns a decided fail into xfail and a decided pass into xpass, which says to remove the mark.
Trial policy
Section titled “Trial policy”A default pass-rate threshold of 100% for a nondeterministic task makes a red exit code mean little, so the policy is explicit: trials, verdict, earlyStop and errorRetries.
earlyStop: falsegives a fixed number of trials per case, to measure pass rates and to compare versions with intervals.errorRetriesretries onlyerrortrials (an outage, a rate limit, an unreadable judge reply). Afailcounts towardtrialsand is never retried: another trial is whattrialsis for.- Transport retries (429, 5xx, a reset connection) belong in the model client or the task, and core does not record them.
- A majority verdict for one case is a smoke signal, not a measurement: at a true pass rate of 70%, a majority of 3 passes in 78% of runs (guide).
When a case stops
Section titled “When a case stops”A decided verdict stands, whatever ends the case later, except that a runner feature that core refuses ends the case in error after a decided verdict too. An open verdict takes its outcome and reason from the cause: the core reference has the table, and shared scenarios pin it in both languages.
Statistics
Section titled “Statistics”wilson gives the 95% interval of one pass rate, rateDifference the Newcombe interval of the change between two rates, and passRates the counts that rate(), the CLI and the MLflow reporter share, so a set check and a report give the same numbers (reference). rateDifference computes sqrt(x*x + y*y) itself and does not call Math.hypot. Why: Math.hypot and Python’s math.hypot round differently in the last bit, and plain operations give both languages the same bits.
Set checks
Section titled “Set checks”A set check turns the cases of a slice into one outcome, and it never passes or fails over a subset: a filtered, sharded or stopped run would pass on the cases that happened to run. A check that covers no case is an error (it is misconfigured). A covered case that did not run makes the check skip, or fail with partialChecks: 'fail' for a CI job that requires every check. A covered case in error makes it error, because its quality is unknown (table). Set a threshold from the low bound of a baseline, not from one run.
In a runner adapter, an eval’s set checks are one test, set checks, declared after the cases and never a hook. Why: some reporters copy hook output to every test of the file. The check reads the results recorded in the same process, so it does not depend on test order: one that runs too early is skip.