Skip to content

Level 2: a component eval

Use a component eval when a prompt or model change affects one production step.

import { exact } from '@trialkit/scorers';
import { describeEval } from '@trialkit/vitest';
import { triageTicket } from '../src/triage.ts';
const category = exact({
name: 'category',
output: (value: { category: string }) => value.category,
});
describeEval(
'support-triage',
{
task: async (ticket: string, ctx) => {
const { triage, usage } = await triageTicket(ticket, {
client,
model: 'gpt-5-mini',
signal: ctx.signal,
});
ctx.addUsage(usage);
return triage;
},
trials: 3,
verdict: 'majority',
},
(it) => {
it.for([{ id: 'double-charge', input: 'I was charged twice.', expected: 'billing' }])(
'$id',
async ({ input, expected }, { run, score }) => {
await score(category, await run(input), expected);
},
);
},
);
  • Call the production function. Pass ctx.signal to its model call and send its usage to ctx.addUsage, so cancellation, token budgets and reports work.
  • run(input) calls task for a trial. score(scorer, output, expected?) records a score. TypeScript infers both from task and the scorer: write no generic argument.
  • it(name, marks, body) declares one case, with the marks tags, metadata, skip and xfail. it.for(rows) declares a table. Keep held-out and prompt-quoted cases distinct in metadata.
  • A failed expect or score fails a trial. A body can catch a task error when refusal itself is the assertion.

This is the test form: the cases belong in the runner test file. Options and defaults: the Vitest adapter and the core reference. Jest and node:test have the same API.

The definition form: when a script shares the eval

Section titled “The definition form: when a script shares the eval”

Use a definition when a test, a script or calibration must share the same dataset and scoring rules.

import { evalSuite } from '@trialkit/vitest';
import { evaluation } from './evaluation.ts';
evalSuite(evaluation, { concurrency: 4, trialTimeoutMs: 60_000 });
  • defineEval is data and functions; importing it runs nothing. evalSuite makes one test per case and a final set checks test.
  • runEval runs the same definition outside a test runner. It has no default file reporter, so give a script one, such as jsonlFile({ dir: 'eval-results' }).
  • A runner name filter leaves a covered set check skip with partial: <k> of <n> cases ran. A test-form eval must stay in order: do not use test.concurrent or it.concurrent.

Use a reviewed file when a test and a script share cases.

import { readCases } from '@trialkit/core/node';
const dataset = { id: 'support-tickets-v1', cases: await readCases('tickets.jsonl') };

JSON Lines rows have id and input, with optional expected, tags, metadata, skip and xfail; values can be objects. CSV uses id,input,expected,tags. The reader returns unknown values and names a bad row by path and physical line. See the core reader.

For long or multi-line inputs, keep cases in YAML or TOML with @trialkit/cases (Python: trialkit-cases). The file has one cases list with the same fields. The readers are synchronous, so a test file can declare cases without top-level await:

# yaml-language-server: $schema=https://raw.githubusercontent.com/lzehrung/trialkit/main/spec/schemas/cases.schema.json
cases:
- id: double-charge
tags: [regression]
input: |
Hi, I was charged twice for my March invoice.
The second charge appeared on the 14th.
expected: billing
import { readYamlCases } from '@trialkit/cases/yaml';
import { describeEval } from '@trialkit/vitest';
import { expect } from 'vitest';
describeEval('triage', { task, trials: 3, verdict: 'majority' }, (it) => {
it.for(readYamlCases('evals/tickets.yaml'))('$id', async ({ input, expected }, { run }) => {
expect((await run(input)).category).toBe(expected);
});
});

Use casesFromRows when the cases are in a database or behind an API. It checks each row with the rules of the file readers and returns cases. It reads no file.

import { casesFromRows } from '@trialkit/core';
import { describeEval } from '@trialkit/vitest';
import { expect } from 'vitest';
const cases = casesFromRows(await db.query('select id, input, expected from eval_cases'), {
source: 'tickets table',
});
describeEval('triage', { task, trials: 3, verdict: 'majority' }, (it) => {
it.for(cases)('$id', async ({ input, expected }, { run }) => {
expect((await run(input)).category).toBe(expected);
});
});
  • source names the origin in messages: a table, tickets-api or a file path. A bad row gives tickets table: row 3: <reason>. If the origin has line numbers, pass lines (one for each row) to get source:line instead.
  • Query before describeEval: Vitest, node:test and Jest in ESM mode (node --experimental-vm-modules) accept top-level await.
  • Jest in CommonJS mode has no top-level await, and pytest builds its test list before any fixture runs. For these two, snapshot the rows to a file in a step before the tests start (a Jest globalSetup module, or a script that runs before pytest). Then read that file with a synchronous reader: readYamlCases or readTomlCases in TypeScript, read_cases in Python.
  • Select the case fields in the query. A column that is not a case field is an error.

See the core reference and the Python reference. The alpha studio can edit these cases on a page and run them.

Use generateCases when you have a few real cases and want more to review. It asks a model once and gives rows that writeCases saves as a draft. A person edits the draft and copies the approved cases into the dataset: the references that a model writes are not ground truth.

import { writeCases } from '@trialkit/cases/write';
import { readCases } from '@trialkit/core/node';
import { generateCases } from '@trialkit/judge/generate';
import { openaiModel } from '@trialkit/judge/openai';
const model = openaiModel({ client, model: 'gpt-5-mini' });
const result = await generateCases({
model,
source: { cases: await readCases('seeds.jsonl') },
count: 12,
seed: 7,
});
await writeCases('draft.yaml', result.rows);
  • The source is seed cases, { documents: [{ id, text }] } that you loaded, or { description }. Pass the dataset that the draft will join as existing, so that the model does not repeat it. Keep the generation metadata of each row after the review: it records the origin.
  • There is no command: run this in a script of your app, with your own credentials. examples/generate-cases runs offline with a fake model.

The TypeScript and Python references have the options, limits and errors.