@trialkit/vitest
Run trialkit evals as Vitest tests: each case is one test, repeated by its trial policy.
Install
Section titled “Install”npm install --save-dev @trialkit/vitest @trialkit/core vitestPeers: @trialkit/core, Vitest ^4.1.12 || ^5.0.3; Vitest 5 also needs vite. No other dependencies: your installed Vitest is used. CI runs the adapter contract on the floor of each supported major. ESM only; Node.js 22.23.4+ and TypeScript 5.4+. In a CommonJS project use moduleResolution: bundler, TypeScript 5.8+ nodenext, or a dynamic import with node16.
Example
Section titled “Example”import { describeEval } from '@trialkit/vitest';import { expect } from 'vitest';import { classify } from '../src/classify';
describeEval('classify', { task: classify, trials: 3, verdict: 'majority' }, (it) => { it('billing', async ({ run }) => { expect(await run('I was charged twice.')).toBe('billing'); });});- Declare at the top level; run
vitest runor select a case withvitest run -t billing. - The default reporter writes
eval-results/<eval name>.<run id>.jsonl;reporters: []writes nothing. evalSuiteruns a reusabledefineEvaldefinition. Jest and node:test share the test form below.
Reference
Section titled “Reference”Exports: describeEval, evalSuite, evalMatchers. Both forms declare one test per case and a final set checks test when checks exist. See runner adapters for outcome mapping and closing rules.
Options of describeEval
Section titled “Options of describeEval”describeEval(name, options, declare) uses name as eval name and dataset id. Its options is EvalOptions from @trialkit/core/adapter. The table also lists evalSuite options; its task, trial policy and checks come from the definition.
| Option | Default | Meaning |
|---|---|---|
task |
Required | Async or synchronous (input, ctx) => output called by run. |
trials |
1 |
Maximum counted trials; integer >= 1. |
verdict |
'all' |
'all', 'majority', or { atLeast: n }. |
earlyStop |
true |
Stop when remaining trials cannot change the verdict. |
errorRetries |
0 |
Extra attempts for errors; non-negative integer. |
checks |
None | Uniquely named set checks, run in set checks. |
name |
Eval name | Runner suite name. |
reporters |
JSONL file in eval-results/ |
Replace default reporters; [] disables them. |
trialTimeoutMs |
120000 |
Per-trial limit; integer 1–2147483647 ms. |
runTimeoutMs |
Off | Whole-run limit; integer 1–2147483647 ms. |
maxErrors |
Off | Positive error-trial limit, including retries. |
labels |
{} |
String-valued run labels. |
config |
None | JSON configuration snapshot and hash; see run config. |
partialChecks |
'skip' |
Outcome for partial checks: 'skip' or 'fail'. |
tokenBudget |
No budget | Positive input-plus-output token limit, counting usage reported to ctx.addUsage. |
filter |
All cases | Predicate; nonmatching cases skip with filtered out. |
concurrency |
1 |
evalSuite only; cases that run at once, a positive integer or Infinity. |
Invalid options, unknown keys, duplicate eval name and value-equal labels in one file, no tests, it.for([]), and async declaration callbacks throw at declaration. Label key order, display name, and config do not distinguish duplicates; load data before declaring.
One trial policy for each block
Section titled “One trial policy for each block”Each block has its own policy, run, and report series. Use a separately named block for a different policy. Errors do not count toward trials; only errorRetries adds attempts. An atLeast greater than trials is invalid. With early stop, 16 of 20 stops at 16 passes or 5 failures; earlyStop: false runs all 20 counted trials.
The tests
Section titled “The tests”| Form | Meaning |
|---|---|
it(name, body) |
Declare a case with a non-empty, unique id; set checks is reserved when checks exist. |
it(name, marks, body) |
Marks: tags is a string array; metadata has string, number, or boolean values; skip and xfail are reasons. Other keys throw. |
it.for(rows)(name, body) |
One case per row; $key or $a.b expands fields, row marks apply, and the body receives (row, ctx). |
it.only, it.skip |
Runner selection; excluded cases can make checks partial. |
A new context is passed on each trial, with caseId, trial, signal, traceId, attach, log, span, spans, addUsage, metric, skip, run, and score.
await run(input)calls the task and records arunattachment. Input must be JSON; the first call sets the case input, which must stay the same across trials. An unawaited or unsettled call makes the trial an error even if the body also fails.await score(scorer, output, expected?)records a score and throws on failure.outputhas the output type of the scorer: the output of the task, or evidence that the body builds from it (for example the object thatmetricJudgereads). An unawaited or unsettled call is an error.
Optional toPass matcher
Section titled “Optional toPass matcher”Register evalMatchers in a Vitest setup file:
import { evalMatchers } from '@trialkit/vitest';import { expect } from 'vitest';expect.extend(evalMatchers);The adapter does not augment types. Include a consumer declaration in tsconfig.json, using the signature for your Vitest major:
import type { Scorer } from '@trialkit/core';import 'vitest';declare module 'vitest' { // Vitest 5: interface Matchers<R extends void | Promise<void>, T = unknown> { toPass<I, E>(scorer: Scorer<I, T, E>, expected?: E): Promise<void>; }}For Vitest 4 replace interface Matchers<R extends void | Promise<void>, T = unknown> with interface Matchers<T = any>.
await expect(output).toPass(scorer, expected?) records the same score as score. It is valid only inside an active describeEval trial; .not, outside-trial calls, and calls after the trial ends are errors. Always await it: Vitest consumes discarded matcher promises, so use score for strict await checks.
How a trial ends
Section titled “How a trial ends”A returning body passes; throws follow the shared outcome and error rules. A task throw may be caught by the body, but infrastructure errors still end the trial in error. ctx.skip(reason) skips unless a verdict is already decided. Each pass or fail records assert@1; each case records its input hash for comparison. Misuse wins over a body error: a refused runner feature first, then an unawaited call, then a rejected core call, then the error that the body threw.
Results in the Vitest output
Section titled “Results in the Vitest output”| Result | Output |
|---|---|
| Counts | trials annotation, such as 2/3 trials passed. |
| Failure | 0/2 trials passed, needs 2; <last failure> plus other non-passing trials; Vitest retains the thrown error’s diff and frame. |
| Infrastructure error | InfrastructureError: 0/0 trials passed, 1 error, needs 2; <reason>, with the task stack and cause. |
| Misuse | Keeps its error class and call frame, e.g. TypeError: ... run() was not awaited. |
| Skip / warning | ctx.skip(note) for skip or xfail; warning annotation for xpass or partly skipped checks. |
Failed or errored cases add a bounded logs annotation before the error. Passing-case logs stay in reporters and the history report.
The afterAll hook fails the suite for reporter errors, early stop, no verdict despite cases running, failed or errored checks whose test did not run, checks partial because a case skipped while running, or a token budget with no reported tokens. Runner-excluded cases do not themselves fail the hook. Check failures and definition-form case failures use the declaration stack.
Runner features that do not combine with trials
Section titled “Runner features that do not combine with trials”| Feature | Rule |
|---|---|
A failed expect.soft, expect.assertions, expect.hasAssertions, snapshots |
Refused; the case ends in error. |
| Fake timers | Clock-marked fake timers are refused; use vi.useRealTimers(). A call-through spy on the real setTimeout is allowed. |
beforeEach, afterEach, test.extend |
Run once per test; trials share fixture state. Create per-trial state in the body. |
| Retries, repeats, runner timeout | Disabled; trial policy and trialTimeoutMs control attempts and time. |
it.concurrent, it.fails |
Not provided; use xfail and separate files for parallel blocks. |
In Vitest
Section titled “In Vitest”- Tests are ordered with
concurrent: false,shuffle: false,retry: 0,repeats: 0, even under sequence flags. Vitest 4 joins suite and case names with a space for-t; Vitest 5 uses>. Match the case id alone. - Each test and the closing hook have Vitest timeout
0;trialTimeoutMsdefaults to 120 seconds. Vitest’s test signal reaches the case and aborts the trial on cancellation; the closing hook still finishes the run. describeEvalcallsdeclareimmediately, outsidedescribe; hooks declared there apply to the whole file. Workers run files separately; use{ dir }for custom file reporters.- Import the production code in a component eval, so that
vitest run --changedselects the eval when that code changes.
The definition form
Section titled “The definition form”evalSuite(definition, options = {}) takes a defineEval definition and SuiteOptions: the options from name through filter above plus concurrency (default 1, integer >= 1 or Infinity). Policy and checks belong to the definition.
Look-ahead begins with the first selected test and uses Vitest’s selected modes, so -t and only do not start excluded cases. Tests report in order, results remain in dataset order, events may interleave, and each case’s trials are sequential. A thrown callback or cancelled test aborts in-flight cases. Ahead-started tasks cannot rely on their own test’s beforeEach; a token budget can overshoot by about one trial per concurrent case.