Example: support-ticket triage
Copy this when one app needs unit tests, a component eval, a black-box HTTP eval, and a calibrated judge, and CI replays recordings. For one of those, copy minimal, pytest, replay, history and PR gates, or black-box HTTP. The guide explains each level.
From the repository root, pnpm install && pnpm build. Then run these in this directory. Node.js 22.23.4 or later. In PowerShell, set a variable before the command: $env:OPENROUTER_API_KEY = 'sk-or-...'.
| Command | What it does | Cost |
|---|---|---|
pnpm test |
Projects unit and replay. No key. No network except the loopback interface. |
Nothing |
pnpm eval |
Project evals. Real models named in TRIAGE_EVAL_MODELS. With that variable unset, the file skips. |
Model calls |
pnpm record |
Project record. Real models, the judge, and the calibration. Writes evals/cassettes/. |
Model calls |
pnpm blackbox |
Project blackbox, always live. Uses TRIAGE_URL, or starts the service in this process for each model in TRIAGE_EVAL_MODELS. |
Model calls |
pnpm serve |
The service on 127.0.0.1:8787 (PORT). Requires TRIAGE_MODEL. |
Model calls |
pnpm calibrate |
The judge against evals/summary-labels.json. TRIALKIT_CASSETTES=replay uses the recording. |
Nothing on replay |
pnpm eval:script <provider>:<model> ... |
scripts/run-evals.ts. The same eval, cases in parallel. |
Model calls |
pnpm report |
trialkit report eval-results --by model,prompt. |
Nothing |
pnpm report:sample |
The report for history-sample/. |
Nothing |
pnpm record -t glm records one model. pnpm eval -t outage runs one case on each model.
| Variable | Default | Meaning |
|---|---|---|
TRIAGE_EVAL_MODELS |
Unset: the Vitest file skips. replay and record set the three cassette models. |
Comma-separated <provider>:<model>. One component-eval suite per model. Black-box starts one in-process service per model. |
TRIAGE_URL |
Unset: black-box starts the service. replay sets it empty. |
Service for pnpm blackbox, for example http://127.0.0.1:8787. |
TRIAGE_JUDGE_MODEL |
openrouter:anthropic/claude-haiku-5.5 |
Judge model. Calibrate again after a change. A comparison that matters should use a judge from another model family: this default also grades its own summaries in one suite. |
TRIALKIT_CASSETTES |
Unset: live. The projects set replay, record, or empty (blackbox). |
record, replay, or a live run. |
TRIALKIT_RESULTS_DIR |
eval-results |
JSON Lines directory. Git ignores it. |
TRIAGE_MODEL |
Required for pnpm serve. |
<provider>:<model> for the service process. |
PORT |
8787 |
Port of pnpm serve on 127.0.0.1. |
| Provider | Variables | Endpoint |
|---|---|---|
openai |
OPENAI_API_KEY |
OpenAI |
openrouter |
OPENROUTER_API_KEY |
https://openrouter.ai/api/v1 |
databricks |
DATABRICKS_HOST, DATABRICKS_TOKEN |
<host>/ai-gateway/mlflow/v1 |
compatible |
OPENAI_BASE_URL, OPENAI_API_KEY |
Any OpenAI-compatible server |
The cassette models are openrouter:z-ai/glm-5.3-flash, openrouter:anthropic/claude-haiku-5.5, and openrouter:deepseek/deepseek-v4.1-flash.
Output
Section titled “Output”history-sample/ holds 18 real runs: 3 models, prompt versions v1 to v3, 2 runs each. They were recorded before the judge, so they have category and priority scores only. pnpm report:sample:
The output of trialkit report examples/support-triage/history-sample --by model,prompt --format markdown, without its Source JSONL lines:
trialkit report
Section titled “trialkit report”1 eval, 18 runs, from 2026-10-31 14:36 to 2026-10-31 14:42 UTC. Series by model · prompt.
support-triage
Section titled “support-triage”| Series | Runs | Cases | Case pass rate (95% CI) | Trial pass rate (95% CI) | Trend | Error trials | Mean trial | Latest run (UTC) |
|---|---|---|---|---|---|---|---|---|
| openrouter:anthropic/claude-haiku-5.5 · triage@1+b590cb | 2 | 16 pass | 100.0% (80.6–100.0%) | 100.0% (92.6–100.0%) | ██ |
0 | 1.3 s | 2026-10-31 14:39 |
| openrouter:anthropic/claude-haiku-5.5 · triage@2+1721e6 | 2 | 16 pass | 100.0% (80.6–100.0%) | 100.0% (92.6–100.0%) | ██ |
0 | 1.4 s | 2026-10-31 14:41 |
| openrouter:anthropic/claude-haiku-5.5 · triage@3+6de1a1 | 2 | 16 pass | 100.0% (80.6–100.0%) | 100.0% (92.6–100.0%) | ██ |
0 | 1.4 s | 2026-10-31 14:42 |
| openrouter:deepseek/deepseek-v4.1-flash · triage@1+b590cb | 2 | 16 pass | 100.0% (80.6–100.0%) | 100.0% (92.6–100.0%) | ██ |
0 | 1.2 s | 2026-10-31 14:39 |
| openrouter:deepseek/deepseek-v4.1-flash · triage@2+1721e6 | 2 | 16 pass | 100.0% (80.6–100.0%) | 100.0% (92.6–100.0%) | ██ |
0 | 1.2 s | 2026-10-31 14:41 |
| openrouter:deepseek/deepseek-v4.1-flash · triage@3+6de1a1 | 2 | 16 pass | 100.0% (80.6–100.0%) | 100.0% (92.6–100.0%) | ██ |
0 | 980 ms | 2026-10-31 14:42 |
| openrouter:z-ai/glm-5.3-flash · triage@1+b590cb | 2 | 14 pass, 2 fail | 87.5% (64.0–96.5%) | 87.5% (75.3–94.1%) | ▇▇ |
0 | 1.7 s | 2026-10-31 14:39 |
| openrouter:z-ai/glm-5.3-flash · triage@2+1721e6 | 2 | 15 pass, 1 fail | 93.8% (71.7–98.9%) | 91.7% (80.4–96.7%) | █▇ |
0 | 1.9 s | 2026-10-31 14:41 |
| openrouter:z-ai/glm-5.3-flash · triage@3+6de1a1 | 2 | 16 pass | 100.0% (80.6–100.0%) | 97.9% (89.1–99.6%) | ██ |
0 | 1.8 s | 2026-10-31 14:42 |
Trend: the trial pass rate of each completed run in the window, oldest first, on a fixed scale from ▁ 0% to █ 100%, so rows compare.
| Series | Tokens in | Tokens out |
|---|---|---|
| openrouter:anthropic/claude-haiku-5.5 · triage@1+b590cb | 48390 | 10667 |
| openrouter:anthropic/claude-haiku-5.5 · triage@2+1721e6 | 51270 | 12268 |
| openrouter:anthropic/claude-haiku-5.5 · triage@3+6de1a1 | 54438 | 14237 |
| openrouter:deepseek/deepseek-v4.1-flash · triage@1+b590cb | 18364 | 23641 |
| openrouter:deepseek/deepseek-v4.1-flash · triage@2+1721e6 | 17922 | 22438 |
| openrouter:deepseek/deepseek-v4.1-flash · triage@3+6de1a1 | 19854 | 25991 |
| openrouter:z-ai/glm-5.3-flash · triage@1+b590cb | 14082 | 18683 |
| openrouter:z-ai/glm-5.3-flash · triage@2+1721e6 | 16524 | 26971 |
| openrouter:z-ai/glm-5.3-flash · triage@3+6de1a1 | 18390 | 25833 |
No flips since the previous completed run of each series.
The same command without --format markdown writes an HTML report. The site build made the one in this frame. Open it on its own page.
Recorded judge (summaryJudge sets temperature: 0; a model that rejects 0, such as the gpt-5 models, needs that option removed): 158 calls, every reply readable, about $0.00015 each. OpenRouter did not list temperature for this model when it was recorded, so verdicts vary a little between runs. Calibration on the recording:
| Measure | Value |
|---|---|
| Agreement | 0.929 (13 of 14); the test needs at least 0.9 |
| Cohen’s kappa | 0.857 |
| Confusion | human pass: 7 judged pass; human fail: 6 judged fail, 1 judged pass |
| Disagreement | refund-policy-misread: the judge passed a summary that turns a question about the refund policy into a refund request |
| Judge errors | 0 |
| Tokens | 11,627 in, 2,218 out |
One more disagreement gives agreement 0.857, under 0.9. On these 14 summaries the judge missed only refund-policy-misread, in 2 of 4 runs.
pnpm record makes 302 model calls when nothing is retried, and takes about nine minutes. The recordings in this repository hold 181,282 input tokens and 66,103 output tokens. OpenRouter billed $0.0515.
- Category is
exact. Priority isnumeric({ tolerance: 1, scale: 3 }): one step off passes with value 0.67. The summary is arubricJudge(faithful, no invented fact, at most 15 words). Prompt v4 asks for 12 words. The judge sees the ticket and the summary.expectedis an object, so the judge gets no reference answer. Trials: 3,majority,earlyStop: false,errorRetries: 1. Set checks: everyregressioncase passes, and at least 75% of trials pass. In the committed recordings every summary passes the judge. - A status other than 2xx, a body that is not JSON, or a body that is not a triage ends the trial in
error(infrastructureError). A wrong category or priority fails a scorer. A reply the strict parser rejects fails the trial: that reply is the answer the eval measures. A failedGET /healthskips every case, andpartialChecks: 'fail'fails the run. The service reports no task tokens, so a black-box trial counts the judge only. See Errors and failures and Level 3. vitest run --changedfollows imports, includingevals/tickets.json. The selection table is in CI layout.pnpm check:changedasserts it. A cassette change selects no test file: runpnpm testafterpnpm record.
| File | Role |
|---|---|
src/triage.ts |
Prompt v4, the request, and the strict parser. The eval calls triageTicket. |
src/model.ts |
OpenAI-compatible client. Takes fetch. |
src/server.ts, src/main.ts |
POST /triage, GET /health, x-request-id. The service passes the ticket through unchanged, so one cassette serves the component eval and the black-box eval. |
test/triage.test.ts, test/server.test.ts, test/blackbox.test.ts, test/eval.test.ts |
Unit tests. |
test/fake-openai.ts |
A real OpenAI client with a fake fetch. |
test/no-network.ts |
Setup for unit and replay. Global fetch fails except on the loopback interface. |
evals/triage.eval.ts |
Dataset, scorers, judge, trial policy, set checks, labels. |
evals/triage.blackbox.ts |
The same checks. The task is POST /triage. It does not import the service. |
evals/tickets.json |
16 tickets. |
evals/setup.ts |
Models, judge, and cassette mode from the environment. |
evals/cassettes/*.json |
One file per model under test (task calls and the judge calls that grade them), plus calibration.json. |
evals/triage.eval.test.ts |
Component eval for projects replay, evals, and record. |
evals/triage.blackbox.test.ts |
Black-box eval for projects replay and blackbox. Name: support-triage-blackbox. |
evals/summary-labels.json, evals/calibration.ts, evals/summary-judge.calibration.test.ts |
14 human labels. Agreement must be at least 0.9. |
scripts/run-evals.ts, scripts/calibrate.ts |
The eval and the calibration outside the Vitest projects. |
vitest.config.ts |
Projects unit, replay, evals, blackbox, record. |
history-sample/ |
The report above. |
../../scripts/changed-check.mjs |
Asserts which files vitest run --changed selects. Update its lists when you add a test file or an import. |
../../.github/workflows/evals.yml |
A manual real run, the eval-history branch, and a trend report. |
Replayed runs are labeled model=replay:<model> and judge=replay:<judge>. A real run also records prompt (triage@<version>+<six hex characters of the prompt text>), judge, commit, branch, and env.
Record again
Section titled “Record again”A changed prompt, model list, judge model, rubric, or ticket is a new request. Replay answers 404 and the trial ends in error. A change of a scorer, a set check, the trial policy, or a label needs no new recording. See Record and replay.
- Run
pnpm recordwith the provider key set.pnpm record -t glmrecords one model and leaves the other cassettes. The black-box eval replays these cassettes. - Run
pnpm formatat the repository root.saveCassettewrites short arrays on many lines, and the formatter puts them on one line. - Read the diff and commit it. Do not edit a cassette. A case that fails in a recording is a result: fix the prompt or the dataset, or record again.
The evals environment
Section titled “The evals environment”.github/workflows/evals.yml runs only when someone starts it. It commits new JSON Lines files to the eval-history branch, builds the report from that branch, uploads the report, and writes the markdown table to the job summary. The judge uses the same key as the models. The eval job uses this environment. The publish job can push and has no key. The report job has neither.
The workflow reads the secret OPENROUTER_API_KEY of a GitHub environment named evals, not a repository secret. A workflow on another branch cannot read a secret of an environment that allows only main.
- In OpenRouter, make a key for this workflow only, with a credit limit (for example USD 5).
- In the repository: Settings, Environments, New environment. Name it
evals. - Set Deployment branches to
mainonly. A required reviewer is optional. - Add the secret
OPENROUTER_API_KEY. Delete a repository secret of the same name.