Skip to content

Example: support-ticket triage

Copy this when one app needs unit tests, a component eval, a black-box HTTP eval, and a calibrated judge, and CI replays recordings. For one of those, copy minimal, pytest, replay, history and PR gates, or black-box HTTP. The guide explains each level.

From the repository root, pnpm install && pnpm build. Then run these in this directory. Node.js 22.23.4 or later. In PowerShell, set a variable before the command: $env:OPENROUTER_API_KEY = 'sk-or-...'.

Command What it does Cost
pnpm test Projects unit and replay. No key. No network except the loopback interface. Nothing
pnpm eval Project evals. Real models named in TRIAGE_EVAL_MODELS. With that variable unset, the file skips. Model calls
pnpm record Project record. Real models, the judge, and the calibration. Writes evals/cassettes/. Model calls
pnpm blackbox Project blackbox, always live. Uses TRIAGE_URL, or starts the service in this process for each model in TRIAGE_EVAL_MODELS. Model calls
pnpm serve The service on 127.0.0.1:8787 (PORT). Requires TRIAGE_MODEL. Model calls
pnpm calibrate The judge against evals/summary-labels.json. TRIALKIT_CASSETTES=replay uses the recording. Nothing on replay
pnpm eval:script <provider>:<model> ... scripts/run-evals.ts. The same eval, cases in parallel. Model calls
pnpm report trialkit report eval-results --by model,prompt. Nothing
pnpm report:sample The report for history-sample/. Nothing

pnpm record -t glm records one model. pnpm eval -t outage runs one case on each model.

Variable Default Meaning
TRIAGE_EVAL_MODELS Unset: the Vitest file skips. replay and record set the three cassette models. Comma-separated <provider>:<model>. One component-eval suite per model. Black-box starts one in-process service per model.
TRIAGE_URL Unset: black-box starts the service. replay sets it empty. Service for pnpm blackbox, for example http://127.0.0.1:8787.
TRIAGE_JUDGE_MODEL openrouter:anthropic/claude-haiku-5.5 Judge model. Calibrate again after a change. A comparison that matters should use a judge from another model family: this default also grades its own summaries in one suite.
TRIALKIT_CASSETTES Unset: live. The projects set replay, record, or empty (blackbox). record, replay, or a live run.
TRIALKIT_RESULTS_DIR eval-results JSON Lines directory. Git ignores it.
TRIAGE_MODEL Required for pnpm serve. <provider>:<model> for the service process.
PORT 8787 Port of pnpm serve on 127.0.0.1.
Provider Variables Endpoint
openai OPENAI_API_KEY OpenAI
openrouter OPENROUTER_API_KEY https://openrouter.ai/api/v1
databricks DATABRICKS_HOST, DATABRICKS_TOKEN <host>/ai-gateway/mlflow/v1
compatible OPENAI_BASE_URL, OPENAI_API_KEY Any OpenAI-compatible server

The cassette models are openrouter:z-ai/glm-5.3-flash, openrouter:anthropic/claude-haiku-5.5, and openrouter:deepseek/deepseek-v4.1-flash.

history-sample/ holds 18 real runs: 3 models, prompt versions v1 to v3, 2 runs each. They were recorded before the judge, so they have category and priority scores only. pnpm report:sample:

The output of trialkit report examples/support-triage/history-sample --by model,prompt --format markdown, without its Source JSONL lines:

1 eval, 18 runs, from 2026-10-31 14:36 to 2026-10-31 14:42 UTC. Series by model · prompt.

Series Runs Cases Case pass rate (95% CI) Trial pass rate (95% CI) Trend Error trials Mean trial Latest run (UTC)
openrouter:anthropic/claude-haiku-5.5 · triage@1+b590cb 2 16 pass 100.0% (80.6–100.0%) 100.0% (92.6–100.0%) ██ 0 1.3 s 2026-10-31 14:39
openrouter:anthropic/claude-haiku-5.5 · triage@2+1721e6 2 16 pass 100.0% (80.6–100.0%) 100.0% (92.6–100.0%) ██ 0 1.4 s 2026-10-31 14:41
openrouter:anthropic/claude-haiku-5.5 · triage@3+6de1a1 2 16 pass 100.0% (80.6–100.0%) 100.0% (92.6–100.0%) ██ 0 1.4 s 2026-10-31 14:42
openrouter:deepseek/deepseek-v4.1-flash · triage@1+b590cb 2 16 pass 100.0% (80.6–100.0%) 100.0% (92.6–100.0%) ██ 0 1.2 s 2026-10-31 14:39
openrouter:deepseek/deepseek-v4.1-flash · triage@2+1721e6 2 16 pass 100.0% (80.6–100.0%) 100.0% (92.6–100.0%) ██ 0 1.2 s 2026-10-31 14:41
openrouter:deepseek/deepseek-v4.1-flash · triage@3+6de1a1 2 16 pass 100.0% (80.6–100.0%) 100.0% (92.6–100.0%) ██ 0 980 ms 2026-10-31 14:42
openrouter:z-ai/glm-5.3-flash · triage@1+b590cb 2 14 pass, 2 fail 87.5% (64.0–96.5%) 87.5% (75.3–94.1%) ▇▇ 0 1.7 s 2026-10-31 14:39
openrouter:z-ai/glm-5.3-flash · triage@2+1721e6 2 15 pass, 1 fail 93.8% (71.7–98.9%) 91.7% (80.4–96.7%) █▇ 0 1.9 s 2026-10-31 14:41
openrouter:z-ai/glm-5.3-flash · triage@3+6de1a1 2 16 pass 100.0% (80.6–100.0%) 97.9% (89.1–99.6%) ██ 0 1.8 s 2026-10-31 14:42

Trend: the trial pass rate of each completed run in the window, oldest first, on a fixed scale from ▁ 0% to █ 100%, so rows compare.

Series Tokens in Tokens out
openrouter:anthropic/claude-haiku-5.5 · triage@1+b590cb 48390 10667
openrouter:anthropic/claude-haiku-5.5 · triage@2+1721e6 51270 12268
openrouter:anthropic/claude-haiku-5.5 · triage@3+6de1a1 54438 14237
openrouter:deepseek/deepseek-v4.1-flash · triage@1+b590cb 18364 23641
openrouter:deepseek/deepseek-v4.1-flash · triage@2+1721e6 17922 22438
openrouter:deepseek/deepseek-v4.1-flash · triage@3+6de1a1 19854 25991
openrouter:z-ai/glm-5.3-flash · triage@1+b590cb 14082 18683
openrouter:z-ai/glm-5.3-flash · triage@2+1721e6 16524 26971
openrouter:z-ai/glm-5.3-flash · triage@3+6de1a1 18390 25833

No flips since the previous completed run of each series.

The same command without --format markdown writes an HTML report. The site build made the one in this frame. Open it on its own page.

Recorded judge (summaryJudge sets temperature: 0; a model that rejects 0, such as the gpt-5 models, needs that option removed): 158 calls, every reply readable, about $0.00015 each. OpenRouter did not list temperature for this model when it was recorded, so verdicts vary a little between runs. Calibration on the recording:

Measure Value
Agreement 0.929 (13 of 14); the test needs at least 0.9
Cohen’s kappa 0.857
Confusion human pass: 7 judged pass; human fail: 6 judged fail, 1 judged pass
Disagreement refund-policy-misread: the judge passed a summary that turns a question about the refund policy into a refund request
Judge errors 0
Tokens 11,627 in, 2,218 out

One more disagreement gives agreement 0.857, under 0.9. On these 14 summaries the judge missed only refund-policy-misread, in 2 of 4 runs.

pnpm record makes 302 model calls when nothing is retried, and takes about nine minutes. The recordings in this repository hold 181,282 input tokens and 66,103 output tokens. OpenRouter billed $0.0515.

  • Category is exact. Priority is numeric({ tolerance: 1, scale: 3 }): one step off passes with value 0.67. The summary is a rubricJudge (faithful, no invented fact, at most 15 words). Prompt v4 asks for 12 words. The judge sees the ticket and the summary. expected is an object, so the judge gets no reference answer. Trials: 3, majority, earlyStop: false, errorRetries: 1. Set checks: every regression case passes, and at least 75% of trials pass. In the committed recordings every summary passes the judge.
  • A status other than 2xx, a body that is not JSON, or a body that is not a triage ends the trial in error (infrastructureError). A wrong category or priority fails a scorer. A reply the strict parser rejects fails the trial: that reply is the answer the eval measures. A failed GET /health skips every case, and partialChecks: 'fail' fails the run. The service reports no task tokens, so a black-box trial counts the judge only. See Errors and failures and Level 3.
  • vitest run --changed follows imports, including evals/tickets.json. The selection table is in CI layout. pnpm check:changed asserts it. A cassette change selects no test file: run pnpm test after pnpm record.
File Role
src/triage.ts Prompt v4, the request, and the strict parser. The eval calls triageTicket.
src/model.ts OpenAI-compatible client. Takes fetch.
src/server.ts, src/main.ts POST /triage, GET /health, x-request-id. The service passes the ticket through unchanged, so one cassette serves the component eval and the black-box eval.
test/triage.test.ts, test/server.test.ts, test/blackbox.test.ts, test/eval.test.ts Unit tests.
test/fake-openai.ts A real OpenAI client with a fake fetch.
test/no-network.ts Setup for unit and replay. Global fetch fails except on the loopback interface.
evals/triage.eval.ts Dataset, scorers, judge, trial policy, set checks, labels.
evals/triage.blackbox.ts The same checks. The task is POST /triage. It does not import the service.
evals/tickets.json 16 tickets.
evals/setup.ts Models, judge, and cassette mode from the environment.
evals/cassettes/*.json One file per model under test (task calls and the judge calls that grade them), plus calibration.json.
evals/triage.eval.test.ts Component eval for projects replay, evals, and record.
evals/triage.blackbox.test.ts Black-box eval for projects replay and blackbox. Name: support-triage-blackbox.
evals/summary-labels.json, evals/calibration.ts, evals/summary-judge.calibration.test.ts 14 human labels. Agreement must be at least 0.9.
scripts/run-evals.ts, scripts/calibrate.ts The eval and the calibration outside the Vitest projects.
vitest.config.ts Projects unit, replay, evals, blackbox, record.
history-sample/ The report above.
../../scripts/changed-check.mjs Asserts which files vitest run --changed selects. Update its lists when you add a test file or an import.
../../.github/workflows/evals.yml A manual real run, the eval-history branch, and a trend report.

Replayed runs are labeled model=replay:<model> and judge=replay:<judge>. A real run also records prompt (triage@<version>+<six hex characters of the prompt text>), judge, commit, branch, and env.

A changed prompt, model list, judge model, rubric, or ticket is a new request. Replay answers 404 and the trial ends in error. A change of a scorer, a set check, the trial policy, or a label needs no new recording. See Record and replay.

  1. Run pnpm record with the provider key set. pnpm record -t glm records one model and leaves the other cassettes. The black-box eval replays these cassettes.
  2. Run pnpm format at the repository root. saveCassette writes short arrays on many lines, and the formatter puts them on one line.
  3. Read the diff and commit it. Do not edit a cassette. A case that fails in a recording is a result: fix the prompt or the dataset, or record again.

.github/workflows/evals.yml runs only when someone starts it. It commits new JSON Lines files to the eval-history branch, builds the report from that branch, uploads the report, and writes the markdown table to the job summary. The judge uses the same key as the models. The eval job uses this environment. The publish job can push and has no key. The report job has neither.

The workflow reads the secret OPENROUTER_API_KEY of a GitHub environment named evals, not a repository secret. A workflow on another branch cannot read a secret of an environment that allows only main.

  1. In OpenRouter, make a key for this workflow only, with a credit limit (for example USD 5).
  2. In the repository: Settings, Environments, New environment. Name it evals.
  3. Set Deployment branches to main only. A required reviewer is optional.
  4. Add the secret OPENROUTER_API_KEY. Delete a repository secret of the same name.