Skip to content

@trialkit/jest

Run trialkit evals as Jest tests. The shared test form explains cases, marks, run, score, policy, and trial outcomes.

Terminal window
npm install --save-dev @trialkit/jest @trialkit/core jest

Peers: @trialkit/core, Jest ^29.0.0 || ^30.0.0. No other dependencies: your installed Jest is used. Node.js 22.23.4+, TypeScript 5.4+; ESM and CommonJS builds. Jest needs your usual TypeScript transformer; the adapter needs no transformer settings.

import { expect } from '@jest/globals';
import { describeEval } from '@trialkit/jest';
import { classify } from '../src/classify';
describeEval('classify', { task: classify, trials: 3, verdict: 'majority' }, (it) => {
it('billing', async ({ run }) => expect(await run('charged twice')).toBe('billing'));
});
  • Declare at the top level and run jest; CommonJS can use require('@trialkit/jest').
  • Exports: describeEval, evalSuite, evalMatchers. it.for passes (row, ctx), unlike Jest it.each.
  • The default reporter writes eval-results/<eval name>.<run id>.jsonl; reporters: [] writes nothing.

describeEval(name, options, declare) and evalSuite(definition, options = {}) declare one test per case and a final set checks test. Shared validation, marks, and policy are in the Vitest reference; outcomes and closing rules are in runner adapters. The eval name and equal labels must be unique per file; invalid or unknown options, empty cases, and async declarations throw.

The shared option table applies to Jest. evalSuite takes its task, trial policy and checks from the definition.

Register expect.extend(evalMatchers) with expect from @jest/globals. The adapter does not augment types; include this declaration in tsconfig.json:

import type { Scorer } from '@trialkit/core';
declare module 'expect' {
interface Matchers<R extends void | Promise<void>, T = unknown> {
toPass<I, O, E>(scorer: Scorer<I, O, E>, expected?: E): Promise<void>;
}
}

Always await expect(output).toPass(scorer, expected?) inside a running describeEval trial. It records the same score as score; .not, outside-trial and late calls are errors. Jest consumes discarded matcher promises, so use score for strict await checks.

Feature Rule
Filtering jest -t <case id> matches a regex against suite and test names joined by a space; matching the suite selects all. it.only and it.skip use Jest’s modes.
Output console.info prints <suite> > <case>: <outcome>, <counts> once, even on retry; --silent hides it. Failures start 0/2 trials passed, needs 2; <last failure>, followed by other non-passing trials and the original frame. Infrastructure errors show InfrastructureError: 0/0 trials passed, 1 error, needs 2; <reason>, the task stack and cause; misuse keeps its class, such as TypeError.
Skips and warnings Jest cannot skip a running test: skip/xfail passes it and prints skipped (<reason>). Xpass and partially skipped checks use console.warn; all skipped checks pass with a note. Reporters retain true outcomes.
Refused expect.assertions, expect.hasAssertions, snapshots and detected modern fake timers end the case in error. A call-through spy on the real setTimeout is allowed, including when it is excluded from faking. A spy on a fake timeout is detected when another fake method exposes the same clock. A clock hidden everywhere else, legacy timers, or manually replacing timers while fakes are active make detection unreliable; do not use fake timers in evals. Jest has no expect.soft.
Lifecycle beforeEach/afterEach run once per test, so trials share state. it.concurrent and it.failing are unavailable; use xfail. jest.retryTimes reports the first result without rerunning work.
Timeout / cancellation Eval tests and their closing hook use Jest’s maximum timeout, 2147483647 ms, so testTimeout does not apply to them; other tests keep it. trialTimeoutMs limits each trial. Jest supplies no test abort signal.
Order / sharding Do not use --randomize: set checks must run last. Jest shards by file; use separate files and distinct eval names for parallel blocks, with each check and its cases together.
Hooks describeEval calls declare outside Jest describe; hooks there apply to the file. A failing afterAll fails the file, not the individual tests.
Globals / workers Uses @jest/globals, including with injectGlobals: false. Workers have separate files; use { dir } for custom file reporters.

Failed or errored cases print bounded trial logs with console.info before the error, once even when Jest retries the test. --silent hides them. Passing-case logs stay in reporters and the history report.

The closing hook fails for reporter errors, early stops, cases running without any verdict, runtime-skipped partial checks, unreported failed/errored checks, or a token budget without reported tokens. If set checks is filtered out, the hook reports failed or errored checks without repeating already reported failures. Definition-case and check failures use the declaration stack.

Definition-form look-ahead starts with the first case. Jest’s test-name pattern, including a pattern from configuration or -t, and focused tests disable it. When Jest’s selection state is unavailable, look-ahead is also disabled. Otherwise cases may run ahead; results stay in dataset order, events may interleave, a thrown callback aborts in-flight work, and ahead-started tasks cannot rely on their test’s beforeEach. A token budget can overshoot by about one trial per concurrent case.

Import the production code in a component eval, so that --changedSince and --findRelatedTests select the eval when that code changes.