Skip to content

@trialkit/vitest

Run trialkit evals as Vitest tests: each case is one test, repeated by its trial policy.

Terminal window
npm install --save-dev @trialkit/vitest @trialkit/core vitest

Peers: @trialkit/core, Vitest ^4.1.12 || ^5.0.3; Vitest 5 also needs vite. No other dependencies: your installed Vitest is used. CI runs the adapter contract on the floor of each supported major. ESM only; Node.js 22.23.4+ and TypeScript 5.4+. In a CommonJS project use moduleResolution: bundler, TypeScript 5.8+ nodenext, or a dynamic import with node16.

import { describeEval } from '@trialkit/vitest';
import { expect } from 'vitest';
import { classify } from '../src/classify';
describeEval('classify', { task: classify, trials: 3, verdict: 'majority' }, (it) => {
it('billing', async ({ run }) => {
expect(await run('I was charged twice.')).toBe('billing');
});
});
  • Declare at the top level; run vitest run or select a case with vitest run -t billing.
  • The default reporter writes eval-results/<eval name>.<run id>.jsonl; reporters: [] writes nothing.
  • evalSuite runs a reusable defineEval definition. Jest and node:test share the test form below.

Exports: describeEval, evalSuite, evalMatchers. Both forms declare one test per case and a final set checks test when checks exist. See runner adapters for outcome mapping and closing rules.

describeEval(name, options, declare) uses name as eval name and dataset id. Its options is EvalOptions from @trialkit/core/adapter. The table also lists evalSuite options; its task, trial policy and checks come from the definition.

Option Default Meaning
task Required Async or synchronous (input, ctx) => output called by run.
trials 1 Maximum counted trials; integer >= 1.
verdict 'all' 'all', 'majority', or { atLeast: n }.
earlyStop true Stop when remaining trials cannot change the verdict.
errorRetries 0 Extra attempts for errors; non-negative integer.
checks None Uniquely named set checks, run in set checks.
name Eval name Runner suite name.
reporters JSONL file in eval-results/ Replace default reporters; [] disables them.
trialTimeoutMs 120000 Per-trial limit; integer 1–2147483647 ms.
runTimeoutMs Off Whole-run limit; integer 1–2147483647 ms.
maxErrors Off Positive error-trial limit, including retries.
labels {} String-valued run labels.
config None JSON configuration snapshot and hash; see run config.
partialChecks 'skip' Outcome for partial checks: 'skip' or 'fail'.
tokenBudget No budget Positive input-plus-output token limit, counting usage reported to ctx.addUsage.
filter All cases Predicate; nonmatching cases skip with filtered out.
concurrency 1 evalSuite only; cases that run at once, a positive integer or Infinity.

Invalid options, unknown keys, duplicate eval name and value-equal labels in one file, no tests, it.for([]), and async declaration callbacks throw at declaration. Label key order, display name, and config do not distinguish duplicates; load data before declaring.

Each block has its own policy, run, and report series. Use a separately named block for a different policy. Errors do not count toward trials; only errorRetries adds attempts. An atLeast greater than trials is invalid. With early stop, 16 of 20 stops at 16 passes or 5 failures; earlyStop: false runs all 20 counted trials.

Form Meaning
it(name, body) Declare a case with a non-empty, unique id; set checks is reserved when checks exist.
it(name, marks, body) Marks: tags is a string array; metadata has string, number, or boolean values; skip and xfail are reasons. Other keys throw.
it.for(rows)(name, body) One case per row; $key or $a.b expands fields, row marks apply, and the body receives (row, ctx).
it.only, it.skip Runner selection; excluded cases can make checks partial.

A new context is passed on each trial, with caseId, trial, signal, traceId, attach, log, span, spans, addUsage, metric, skip, run, and score.

  • await run(input) calls the task and records a run attachment. Input must be JSON; the first call sets the case input, which must stay the same across trials. An unawaited or unsettled call makes the trial an error even if the body also fails.
  • await score(scorer, output, expected?) records a score and throws on failure. output has the output type of the scorer: the output of the task, or evidence that the body builds from it (for example the object that metricJudge reads). An unawaited or unsettled call is an error.

Register evalMatchers in a Vitest setup file:

import { evalMatchers } from '@trialkit/vitest';
import { expect } from 'vitest';
expect.extend(evalMatchers);

The adapter does not augment types. Include a consumer declaration in tsconfig.json, using the signature for your Vitest major:

import type { Scorer } from '@trialkit/core';
import 'vitest';
declare module 'vitest' {
// Vitest 5:
interface Matchers<R extends void | Promise<void>, T = unknown> {
toPass<I, E>(scorer: Scorer<I, T, E>, expected?: E): Promise<void>;
}
}

For Vitest 4 replace interface Matchers<R extends void | Promise<void>, T = unknown> with interface Matchers<T = any>.

await expect(output).toPass(scorer, expected?) records the same score as score. It is valid only inside an active describeEval trial; .not, outside-trial calls, and calls after the trial ends are errors. Always await it: Vitest consumes discarded matcher promises, so use score for strict await checks.

A returning body passes; throws follow the shared outcome and error rules. A task throw may be caught by the body, but infrastructure errors still end the trial in error. ctx.skip(reason) skips unless a verdict is already decided. Each pass or fail records assert@1; each case records its input hash for comparison. Misuse wins over a body error: a refused runner feature first, then an unawaited call, then a rejected core call, then the error that the body threw.

Result Output
Counts trials annotation, such as 2/3 trials passed.
Failure 0/2 trials passed, needs 2; <last failure> plus other non-passing trials; Vitest retains the thrown error’s diff and frame.
Infrastructure error InfrastructureError: 0/0 trials passed, 1 error, needs 2; <reason>, with the task stack and cause.
Misuse Keeps its error class and call frame, e.g. TypeError: ... run() was not awaited.
Skip / warning ctx.skip(note) for skip or xfail; warning annotation for xpass or partly skipped checks.

Failed or errored cases add a bounded logs annotation before the error. Passing-case logs stay in reporters and the history report.

The afterAll hook fails the suite for reporter errors, early stop, no verdict despite cases running, failed or errored checks whose test did not run, checks partial because a case skipped while running, or a token budget with no reported tokens. Runner-excluded cases do not themselves fail the hook. Check failures and definition-form case failures use the declaration stack.

Runner features that do not combine with trials

Section titled “Runner features that do not combine with trials”
Feature Rule
A failed expect.soft, expect.assertions, expect.hasAssertions, snapshots Refused; the case ends in error.
Fake timers Clock-marked fake timers are refused; use vi.useRealTimers(). A call-through spy on the real setTimeout is allowed.
beforeEach, afterEach, test.extend Run once per test; trials share fixture state. Create per-trial state in the body.
Retries, repeats, runner timeout Disabled; trial policy and trialTimeoutMs control attempts and time.
it.concurrent, it.fails Not provided; use xfail and separate files for parallel blocks.
  • Tests are ordered with concurrent: false, shuffle: false, retry: 0, repeats: 0, even under sequence flags. Vitest 4 joins suite and case names with a space for -t; Vitest 5 uses >. Match the case id alone.
  • Each test and the closing hook have Vitest timeout 0; trialTimeoutMs defaults to 120 seconds. Vitest’s test signal reaches the case and aborts the trial on cancellation; the closing hook still finishes the run.
  • describeEval calls declare immediately, outside describe; hooks declared there apply to the whole file. Workers run files separately; use { dir } for custom file reporters.
  • Import the production code in a component eval, so that vitest run --changed selects the eval when that code changes.

evalSuite(definition, options = {}) takes a defineEval definition and SuiteOptions: the options from name through filter above plus concurrency (default 1, integer >= 1 or Infinity). Policy and checks belong to the definition.

Look-ahead begins with the first selected test and uses Vitest’s selected modes, so -t and only do not start excluded cases. Tests report in order, results remain in dataset order, events may interleave, and each case’s trials are sequential. A thrown callback or cancelled test aborts in-flight cases. Ahead-started tasks cannot rely on their own test’s beforeEach; a token budget can overshoot by about one trial per concurrent case.