Skip to content

10. Runner adapters

Runner adapters put the two eval forms in a native test runner. Core owns trials, verdicts and records, and each adapter maps runner selection, results and lifecycle to core. The references are Vitest, Jest, node:test, pytest and the adapter contract.

  • describeEval supplies its own it, and core runs the body once for each trial. Why: a runner calls a test body once, and its retries cannot express majority or atLeast verdicts while keeping infrastructure errors apart from quality failures. The supplied it also works across Vitest 4 and 5 without a runner-specific harness.
  • Vitest 4 and 5 share one adapter that uses the public vitest entry and detects a feature instead of comparing version strings.
  • pytest uses the top-level pytest_trialkit plugin. Why: in a plugin inside the trialkit namespace package, pytest strips trialkit/ and tries to rewrite the real pytest module, so non-editable installs warned on every run.
  • A case that runs after set checks fails with an order error, so random test order is not supported in Jest and node:test. Vitest pins the order with shuffle: false, and pytest moves set checks after the last selected case.
  • Runner selection (it.only, it.skip, name filters, deselection) can make a check partial, and partialChecks chooses skip or fail. A skip mark excludes a case from checks on purpose; a runtime skip or a core filter does not.

Runner features must not change a trial’s verdict or stop its only deadline.

  • A failed expect.soft does not throw, assertion counts span all trials, and snapshots change with the number of trials, so the adapters refuse them and the case ends in error. Use ordinary assertions and trialkit’s xfail mark.
  • Fake timers can stop the trial timeout, the only deadline left once adapters turn off the runner’s. A detected modern fake timer ends the case in error. Detection is not reliable for every fake (Jest legacy timers, a clock hidden on every method, node:test mock timers), so do not use fake timers in evals.
  • score is the primary scorer API. Vitest and Jest also offer the opt-in toPass matcher: the app registers it, and trialkit patches no expect. The runner consumes a discarded matcher promise, so core cannot detect it: always await toPass, or use score.

The adapters preserve core outcomes and fail the run when no test can report an incomplete result. Every case test shows its trial counts in the runner’s own output. The closing hook finishes each started run and fails the suite when a result is incomplete (a reporter error, a stopped run, a partial check, cases without a verdict). The Python adapter does this in a session hook, as a section of the terminal summary and exit status 1; no test fails for it.

The contract fixtures run against each runner version, with a capability table for the points where runners differ, so every other result stays shared. Migration notes map the vitest-evals test shape to trialkit.