10. Runner adapters
Runner adapters put the two eval forms in a native test runner. Core owns trials, verdicts and records, and each adapter maps runner selection, results and lifecycle to core. The references are Vitest, Jest, node:test, pytest and the adapter contract.
Forms and adapter boundary
Section titled “Forms and adapter boundary”describeEvalsupplies its ownit, and core runs the body once for each trial. Why: a runner calls a test body once, and its retries cannot expressmajorityoratLeastverdicts while keeping infrastructure errors apart from quality failures. The supplieditalso works across Vitest 4 and 5 without a runner-specific harness.- Vitest 4 and 5 share one adapter that uses the public
vitestentry and detects a feature instead of comparing version strings. - pytest uses the top-level
pytest_trialkitplugin. Why: in a plugin inside thetrialkitnamespace package, pytest stripstrialkit/and tries to rewrite the realpytestmodule, so non-editable installs warned on every run.
Selection and ordering
Section titled “Selection and ordering”- A case that runs after
set checksfails with an order error, so random test order is not supported in Jest and node:test. Vitest pins the order withshuffle: false, and pytest movesset checksafter the last selected case. - Runner selection (
it.only,it.skip, name filters, deselection) can make a check partial, andpartialCheckschoosesskiporfail. Askipmark excludes a case from checks on purpose; a runtime skip or a corefilterdoes not.
Trial features and matchers
Section titled “Trial features and matchers”Runner features must not change a trial’s verdict or stop its only deadline.
- A failed
expect.softdoes not throw, assertion counts span all trials, and snapshots change with the number of trials, so the adapters refuse them and the case ends inerror. Use ordinary assertions and trialkit’sxfailmark. - Fake timers can stop the trial timeout, the only deadline left once adapters turn off the runner’s. A detected modern fake timer ends the case in
error. Detection is not reliable for every fake (Jest legacy timers, a clock hidden on every method, node:test mock timers), so do not use fake timers in evals. scoreis the primary scorer API. Vitest and Jest also offer the opt-intoPassmatcher: the app registers it, and trialkit patches noexpect. The runner consumes a discarded matcher promise, so core cannot detect it: always awaittoPass, or usescore.
Outcomes and closing
Section titled “Outcomes and closing”The adapters preserve core outcomes and fail the run when no test can report an incomplete result. Every case test shows its trial counts in the runner’s own output. The closing hook finishes each started run and fails the suite when a result is incomplete (a reporter error, a stopped run, a partial check, cases without a verdict). The Python adapter does this in a session hook, as a section of the terminal summary and exit status 1; no test fails for it.
Contract coverage
Section titled “Contract coverage”The contract fixtures run against each runner version, with a capability table for the points where runners differ, so every other result stays shared. Migration notes map the vitest-evals test shape to trialkit.