@trialkit/jest
Run trialkit evals as Jest tests. The shared test form explains cases, marks, run, score, policy, and trial outcomes.
Install
Section titled “Install”npm install --save-dev @trialkit/jest @trialkit/core jestPeers: @trialkit/core, Jest ^29.0.0 || ^30.0.0. No other dependencies: your installed Jest is used. Node.js 22.23.4+, TypeScript 5.4+; ESM and CommonJS builds. Jest needs your usual TypeScript transformer; the adapter needs no transformer settings.
Example
Section titled “Example”import { expect } from '@jest/globals';import { describeEval } from '@trialkit/jest';import { classify } from '../src/classify';
describeEval('classify', { task: classify, trials: 3, verdict: 'majority' }, (it) => { it('billing', async ({ run }) => expect(await run('charged twice')).toBe('billing'));});- Declare at the top level and run
jest; CommonJS can userequire('@trialkit/jest'). - Exports:
describeEval,evalSuite,evalMatchers.it.forpasses(row, ctx), unlike Jestit.each. - The default reporter writes
eval-results/<eval name>.<run id>.jsonl;reporters: []writes nothing.
Reference
Section titled “Reference”describeEval(name, options, declare) and evalSuite(definition, options = {}) declare one test per case and a final set checks test. Shared validation, marks, and policy are in the Vitest reference; outcomes and closing rules are in runner adapters. The eval name and equal labels must be unique per file; invalid or unknown options, empty cases, and async declarations throw.
The shared option table applies to Jest. evalSuite takes its task, trial policy and checks from the definition.
Optional toPass matcher
Section titled “Optional toPass matcher”Register expect.extend(evalMatchers) with expect from @jest/globals. The adapter does not augment types; include this declaration in tsconfig.json:
import type { Scorer } from '@trialkit/core';declare module 'expect' { interface Matchers<R extends void | Promise<void>, T = unknown> { toPass<I, O, E>(scorer: Scorer<I, O, E>, expected?: E): Promise<void>; }}Always await expect(output).toPass(scorer, expected?) inside a running describeEval trial. It records the same score as score; .not, outside-trial and late calls are errors. Jest consumes discarded matcher promises, so use score for strict await checks.
Jest behavior
Section titled “Jest behavior”| Feature | Rule |
|---|---|
| Filtering | jest -t <case id> matches a regex against suite and test names joined by a space; matching the suite selects all. it.only and it.skip use Jest’s modes. |
| Output | console.info prints <suite> > <case>: <outcome>, <counts> once, even on retry; --silent hides it. Failures start 0/2 trials passed, needs 2; <last failure>, followed by other non-passing trials and the original frame. Infrastructure errors show InfrastructureError: 0/0 trials passed, 1 error, needs 2; <reason>, the task stack and cause; misuse keeps its class, such as TypeError. |
| Skips and warnings | Jest cannot skip a running test: skip/xfail passes it and prints skipped (<reason>). Xpass and partially skipped checks use console.warn; all skipped checks pass with a note. Reporters retain true outcomes. |
| Refused | expect.assertions, expect.hasAssertions, snapshots and detected modern fake timers end the case in error. A call-through spy on the real setTimeout is allowed, including when it is excluded from faking. A spy on a fake timeout is detected when another fake method exposes the same clock. A clock hidden everywhere else, legacy timers, or manually replacing timers while fakes are active make detection unreliable; do not use fake timers in evals. Jest has no expect.soft. |
| Lifecycle | beforeEach/afterEach run once per test, so trials share state. it.concurrent and it.failing are unavailable; use xfail. jest.retryTimes reports the first result without rerunning work. |
| Timeout / cancellation | Eval tests and their closing hook use Jest’s maximum timeout, 2147483647 ms, so testTimeout does not apply to them; other tests keep it. trialTimeoutMs limits each trial. Jest supplies no test abort signal. |
| Order / sharding | Do not use --randomize: set checks must run last. Jest shards by file; use separate files and distinct eval names for parallel blocks, with each check and its cases together. |
| Hooks | describeEval calls declare outside Jest describe; hooks there apply to the file. A failing afterAll fails the file, not the individual tests. |
| Globals / workers | Uses @jest/globals, including with injectGlobals: false. Workers have separate files; use { dir } for custom file reporters. |
Failed or errored cases print bounded trial logs with console.info before the error, once even when Jest retries the test. --silent hides them. Passing-case logs stay in reporters and the history report.
The closing hook fails for reporter errors, early stops, cases running without any verdict, runtime-skipped partial checks, unreported failed/errored checks, or a token budget without reported tokens. If set checks is filtered out, the hook reports failed or errored checks without repeating already reported failures. Definition-case and check failures use the declaration stack.
Definition-form look-ahead starts with the first case. Jest’s test-name pattern, including a pattern from configuration or -t, and focused tests disable it. When Jest’s selection state is unavailable, look-ahead is also disabled. Otherwise cases may run ahead; results stay in dataset order, events may interleave, a thrown callback aborts in-flight work, and ahead-started tasks cannot rely on their test’s beforeEach. A token budget can overshoot by about one trial per concurrent case.
Import the production code in a component eval, so that --changedSince and --findRelatedTests select the eval when that code changes.