Level 2: a component eval
Use a component eval when a prompt or model change affects one production step.
import { exact } from '@trialkit/scorers';import { describeEval } from '@trialkit/vitest';import { triageTicket } from '../src/triage.ts';
const category = exact({ name: 'category', output: (value: { category: string }) => value.category,});
describeEval( 'support-triage', { task: async (ticket: string, ctx) => { const { triage, usage } = await triageTicket(ticket, { client, model: 'gpt-5-mini', signal: ctx.signal, }); ctx.addUsage(usage); return triage; }, trials: 3, verdict: 'majority', }, (it) => { it.for([{ id: 'double-charge', input: 'I was charged twice.', expected: 'billing' }])( '$id', async ({ input, expected }, { run, score }) => { await score(category, await run(input), expected); }, ); },);- Call the production function. Pass
ctx.signalto its model call and send its usage toctx.addUsage, so cancellation, token budgets and reports work. run(input)callstaskfor a trial.score(scorer, output, expected?)records a score. TypeScript infers both fromtaskand the scorer: write no generic argument.it(name, marks, body)declares one case, with the markstags,metadata,skipandxfail.it.for(rows)declares a table. Keep held-out and prompt-quoted cases distinct in metadata.- A failed
expector score fails a trial. A body can catch a task error when refusal itself is the assertion.
This is the test form: the cases belong in the runner test file. Options and defaults: the Vitest adapter and the core reference. Jest and node:test have the same API.
The definition form: when a script shares the eval
Section titled “The definition form: when a script shares the eval”Use a definition when a test, a script or calibration must share the same dataset and scoring rules.
import { evalSuite } from '@trialkit/vitest';import { evaluation } from './evaluation.ts';
evalSuite(evaluation, { concurrency: 4, trialTimeoutMs: 60_000 });defineEvalis data and functions; importing it runs nothing.evalSuitemakes one test per case and a finalset checkstest.runEvalruns the same definition outside a test runner. It has no default file reporter, so give a script one, such asjsonlFile({ dir: 'eval-results' }).- A runner name filter leaves a covered set check
skipwithpartial: <k> of <n> cases ran. A test-form eval must stay in order: do not usetest.concurrentorit.concurrent.
Cases from a file
Section titled “Cases from a file”Use a reviewed file when a test and a script share cases.
import { readCases } from '@trialkit/core/node';
const dataset = { id: 'support-tickets-v1', cases: await readCases('tickets.jsonl') };JSON Lines rows have id and input, with optional expected, tags, metadata, skip and xfail; values can be objects. CSV uses id,input,expected,tags. The reader returns unknown values and names a bad row by path and physical line. See the core reader.
For long or multi-line inputs, keep cases in YAML or TOML with @trialkit/cases (Python: trialkit-cases). The file has one cases list with the same fields. The readers are synchronous, so a test file can declare cases without top-level await:
# yaml-language-server: $schema=https://raw.githubusercontent.com/lzehrung/trialkit/main/spec/schemas/cases.schema.jsoncases: - id: double-charge tags: [regression] input: | Hi, I was charged twice for my March invoice. The second charge appeared on the 14th. expected: billingimport { readYamlCases } from '@trialkit/cases/yaml';import { describeEval } from '@trialkit/vitest';import { expect } from 'vitest';
describeEval('triage', { task, trials: 3, verdict: 'majority' }, (it) => { it.for(readYamlCases('evals/tickets.yaml'))('$id', async ({ input, expected }, { run }) => { expect((await run(input)).category).toBe(expected); });});Cases from a database or API
Section titled “Cases from a database or API”Use casesFromRows when the cases are in a database or behind an API. It checks each row with the rules of the file readers and returns cases. It reads no file.
import { casesFromRows } from '@trialkit/core';import { describeEval } from '@trialkit/vitest';import { expect } from 'vitest';
const cases = casesFromRows(await db.query('select id, input, expected from eval_cases'), { source: 'tickets table',});
describeEval('triage', { task, trials: 3, verdict: 'majority' }, (it) => { it.for(cases)('$id', async ({ input, expected }, { run }) => { expect((await run(input)).category).toBe(expected); });});sourcenames the origin in messages: a table,tickets-apior a file path. A bad row givestickets table: row 3: <reason>. If the origin has line numbers, passlines(one for each row) to getsource:lineinstead.- Query before
describeEval: Vitest,node:testand Jest in ESM mode (node --experimental-vm-modules) accept top-levelawait. - Jest in CommonJS mode has no top-level
await, and pytest builds its test list before any fixture runs. For these two, snapshot the rows to a file in a step before the tests start (a JestglobalSetupmodule, or a script that runs before pytest). Then read that file with a synchronous reader:readYamlCasesorreadTomlCasesin TypeScript,read_casesin Python. - Select the case fields in the query. A column that is not a case field is an error.
See the core reference and the Python reference. The alpha studio can edit these cases on a page and run them.
Draft cases with a model
Section titled “Draft cases with a model”Use generateCases when you have a few real cases and want more to review. It asks a model once and gives rows that writeCases saves as a draft. A person edits the draft and copies the approved cases into the dataset: the references that a model writes are not ground truth.
import { writeCases } from '@trialkit/cases/write';import { readCases } from '@trialkit/core/node';import { generateCases } from '@trialkit/judge/generate';import { openaiModel } from '@trialkit/judge/openai';
const model = openaiModel({ client, model: 'gpt-5-mini' });const result = await generateCases({ model, source: { cases: await readCases('seeds.jsonl') }, count: 12, seed: 7,});await writeCases('draft.yaml', result.rows);- The source is seed cases,
{ documents: [{ id, text }] }that you loaded, or{ description }. Pass the dataset that the draft will join asexisting, so that the model does not repeat it. Keep the generation metadata of each row after the review: it records the origin. - There is no command: run this in a script of your app, with your own credentials.
examples/generate-casesruns offline with a fake model.
The TypeScript and Python references have the options, limits and errors.