Skip to content

Example: minimal

Copy this for the smallest Vitest eval: one model call, an exact category check, and a rubric judge, replayed from cassette.json.

Run these in this directory. Node.js 22.23.4 or later. In PowerShell, set the key first: $env:OPENROUTER_API_KEY = 'sk-or-...'.

Command What it does Cost
pnpm test Replay: cassette.json answers. No network and no key. Nothing
TRIALKIT_LIVE=1 OPENROUTER_API_KEY=sk-or-... pnpm test The same eval calls the real models. Under one cent
OPENROUTER_API_KEY=sk-or-... pnpm record Replay known calls and add each new reply to cassette.json. New calls only
pnpm report Reads eval-results/ and writes trialkit-report.html. --format markdown prints a table. Nothing

pnpm test -t crash runs the case crash. The verbose reporter prints:

✓ classify.eval.test.ts > classify > double-charge
❯ trials
↳ 2/2 trials passed
  • trials: 3 and verdict: 'majority'. A case stops when the verdict is sure, so a steady pass prints 2/2. 2/3 means one trial failed. A fail prints counts such as 0/2 trials passed, needs 2. A trial in error (timeout, HTTP error status, replay 404) prints apart, for example 1 error, and is not a fail. A reply that is not a classification fails the trial.
  • The suite sets no reporters, so each run writes eval-results/classify.<run id>.jsonl (Git ignores that folder). A judge score records the template (rubric@1), a hash of the judge’s system prompt, and the judge model. The model label is z-ai/glm-5.3-flash, or replay:z-ai/glm-5.3-flash on replay, so a report keeps the two series apart.
  • The committed recording made 14 model calls (10 task, 4 judge): 3,827 input tokens, 2,615 output tokens, $0.0013 (usage.cost in cassette.json). A changed prompt, schema, ticket, rubric, or model gets 404 No recording matches POST … and the trial ends in error. pnpm record adds misses. Delete cassette.json and record every case in one run (no -t) to replace every reply. Do not edit the cassette. saveCassette wraps short arrays; pnpm format at the repository root puts them on one line. A case that fails in a recording is a result: fix the prompt and record again.
File Role
src/classify.ts Production call: category and a one-sentence summary. Takes the OpenAI client.
classify.eval.test.ts The eval. expect checks the category (crash is tagged regression). score runs rubricJudge on the summary.
openrouter.ts One client for the task (z-ai/glm-5.3-flash) and the judge (anthropic/claude-haiku-5.5). Replay, live (TRIALKIT_LIVE=1), or record misses.
cassette.json Replies from one real run.
vitest.config.ts Verbose reporter. pnpm record is vitest run --mode record, which sets TRIALKIT_RECORD on Windows too.

Next: pytest, replay, history and PR gates, black-box HTTP, compare models, support triage.