Example: minimal
Copy this for the smallest Vitest eval: one model call, an exact category check, and a rubric judge, replayed from cassette.json.
Run these in this directory. Node.js 22.23.4 or later. In PowerShell, set the key first: $env:OPENROUTER_API_KEY = 'sk-or-...'.
| Command | What it does | Cost |
|---|---|---|
pnpm test |
Replay: cassette.json answers. No network and no key. |
Nothing |
TRIALKIT_LIVE=1 OPENROUTER_API_KEY=sk-or-... pnpm test |
The same eval calls the real models. | Under one cent |
OPENROUTER_API_KEY=sk-or-... pnpm record |
Replay known calls and add each new reply to cassette.json. |
New calls only |
pnpm report |
Reads eval-results/ and writes trialkit-report.html. --format markdown prints a table. |
Nothing |
pnpm test -t crash runs the case crash. The verbose reporter prints:
✓ classify.eval.test.ts > classify > double-charge
❯ trials ↳ 2/2 trials passedtrials: 3andverdict: 'majority'. A case stops when the verdict is sure, so a steady pass prints2/2.2/3means one trial failed. A fail prints counts such as0/2 trials passed, needs 2. A trial inerror(timeout, HTTP error status, replay 404) prints apart, for example1 error, and is not a fail. A reply that is not a classification fails the trial.- The suite sets no
reporters, so each run writeseval-results/classify.<run id>.jsonl(Git ignores that folder). A judge score records the template (rubric@1), a hash of the judge’s system prompt, and the judge model. Themodellabel isz-ai/glm-5.3-flash, orreplay:z-ai/glm-5.3-flashon replay, so a report keeps the two series apart. - The committed recording made 14 model calls (10 task, 4 judge): 3,827 input tokens, 2,615 output tokens, $0.0013 (
usage.costincassette.json). A changed prompt, schema, ticket, rubric, or model gets404 No recording matches POST …and the trial ends inerror.pnpm recordadds misses. Deletecassette.jsonand record every case in one run (no-t) to replace every reply. Do not edit the cassette.saveCassettewraps short arrays;pnpm formatat the repository root puts them on one line. A case that fails in a recording is a result: fix the prompt and record again.
| File | Role |
|---|---|
src/classify.ts |
Production call: category and a one-sentence summary. Takes the OpenAI client. |
classify.eval.test.ts |
The eval. expect checks the category (crash is tagged regression). score runs rubricJudge on the summary. |
openrouter.ts |
One client for the task (z-ai/glm-5.3-flash) and the judge (anthropic/claude-haiku-5.5). Replay, live (TRIALKIT_LIVE=1), or record misses. |
cassette.json |
Replies from one real run. |
vitest.config.ts |
Verbose reporter. pnpm record is vitest run --mode record, which sets TRIALKIT_RECORD on Windows too. |
Next: pytest, replay, history and PR gates, black-box HTTP, compare models, support triage.