Example: compare models
Copy this when you compare models on one eval and a wrong answer should stay in the record as a result. Two sets: compare-classify (the default) and routing (Routing).
Run these in this directory. Node.js 22.23.4 or later. In PowerShell, set the key first: $env:OPENROUTER_API_KEY = 'sk-or-...'.
| Command | What it does | Cost |
|---|---|---|
pnpm test |
Replay. Every case gets a verdict, and the outcomes equal the latest run in history/<set>/<model>/. A wrong answer does not fail the test. |
Nothing |
pnpm compare |
Replay of compare-classify. Prints each model. Exit code 1 only when a case gets no verdict. |
Nothing |
pnpm compare --set routing |
The same for routing. --live and pnpm record take --set too. |
Nothing |
pnpm compare --live |
Live models. Prints the result and keeps no record. | About 1.3 cents (routing: 3) |
OPENROUTER_API_KEY=sk-or-... pnpm record |
Live run. Replaces both cassettes of the set and adds one run under history/<set>/. |
About 1.3 cents (routing: 3) |
pnpm report |
trialkit report history --by model writes trialkit-report.html. --format markdown prints a table. |
Nothing |
pnpm exec trialkit compare history/compare-classify/z-ai-glm-5.3-flash history/compare-classify/inclusionai-ling-3.0-flash --by= |
Latest run of each model, case by case. --by= pairs by eval name, so each directory holds one model. |
Nothing |
$ pnpm comparecompare-classify [model=replay:z-ai/glm-5.3-flash, env=local]: 11 pass, 1 fail (run bfcaa22b-3310-404f-8a07-3a8015c37781) ✗ invite-greyed fail 1 of 3 trials passed, needs 2; category: expected "account", got "bug"compare-classify [model=replay:inclusionai/ling-3.0-flash, env=local]: 12 pass (run db9b6ffa-a92a-464a-8618-d4b54500b443)The run ids change in each run. The label replay: keeps a replayed run apart from a real one.
The models
Section titled “The models”Both task models think before they answer, and a thinking token is billed as output. Of the task-call output tokens in the recording, 78% (GLM) and 91% (Ling) are thinking tokens. The cost table in the next section is the sum of usage.cost in the cassette of the second run (36 task replies and 36 judge replies), so it holds what the providers that answered charged. The provider field of each reply names them: Wafer, OpenInference, InferenceNet, Cohere, DeepInfra and DigitalOcean answered for GLM, DeepInfra for Ling, and Azure for the judge. For list prices, see the OpenRouter pages of z-ai/glm-5.3-flash, inclusionai/ling-3.0-flash and anthropic/claude-haiku-5.5.
The result
Section titled “The result”The output of trialkit report examples/compare-models/history/compare-classify --by model --format markdown, without its Source JSONL lines:
trialkit report
Section titled “trialkit report”1 eval, 4 runs, from 2026-11-02 03:28 to 2026-11-02 03:29 UTC. Series by model.
compare-classify
Section titled “compare-classify”| Series | prompt | Runs | Cases | Case pass rate (95% CI) | Trial pass rate (95% CI) | Trend | Error trials | Mean trial | Latest run (UTC) |
|---|---|---|---|---|---|---|---|---|---|
| inclusionai/ling-3.0-flash | - | 2 | 12 pass | 100.0% (75.8–100.0%) | 97.2% (85.8–99.5%) | ██ |
0 | 5.1 s | 2026-11-02 03:29 |
| z-ai/glm-5.3-flash | - | 2 | 11 pass, 1 fail | 91.7% (64.6–98.5%) | 94.4% (81.9–98.5%) | ▇█ |
0 | 2.7 s | 2026-11-02 03:29 |
Trend: the trial pass rate of each completed run in the window, oldest first, on a fixed scale from ▁ 0% to █ 100%, so rows compare.
| Series | Tokens in | Tokens out |
|---|---|---|
| inclusionai/ling-3.0-flash | 54563 | 8185 |
| z-ai/glm-5.3-flash | 54673 | 8415 |
No flips since the previous completed run of each series.
The same command without --format markdown writes an HTML report. The site build made the one in this frame. Open it on its own page.
The report has no cost. This table adds it, from the cassette, and the cases that did not pass.
| Model | Cost of the run | Task calls | Judge calls | Cases that failed | Cases with trials that disagreed |
|---|---|---|---|---|---|
z-ai/glm-5.3-flash |
$0.0067 | $0.0019 | $0.0048 | invite-greyed in both runs |
Run 2: invite-greyed (1 of 3 trials passed) |
inclusionai/ling-3.0-flash |
$0.0062 | $0.0014 | $0.0049 | None | Run 2: sarcasm-crash (2 of 3 trials passed) |
- The 95% intervals of the trial pass rates overlap (94.4%, 81.9–98.5%, and 97.2%, 85.8–99.5%), and so do the case pass rates.
trialkit comparereports Ling 2.8 points higher, 95% interval −9.3 to +15.6, on 36 trials (an interval about 14 to 17 points wide). The models differ oninvite-greyedonly: the prompt rule isaccount; GLM answeredbugin 5 of 6 trials and Ling answeredaccountin 6 of 6. In the cassette, OpenInference served GLM’s three trials of that ticket. Nothing pins a provider, so the comparison measures what OpenRouter routed. Pin one provider withonlyandallow_fallbacks(provider routing). In run 2 the judge failed one Ling summary for adding the word “recent”. - The 36 judge calls cost 2.5 to 3.5 times the 36 task calls. The judge prompt holds the rubric, the ticket, and the summary.
- A case in
error(a stale cassette answers 404) failspnpm test. A wrong recorded answer leaves the test green. The example writes noeval-results/and stays out of check.yml. The comparison is a script, the shape in the guide.
Routing
Section titled “Routing”The routing set compares typesafe/jev-router, a router model on OpenRouter that picks a model and a reasoning effort for each request, with z-ai/glm-5.3-flash. The models API lists no price (-1): a request costs what the picked model costs. So src/ask.ts returns the answer, completion.model and usage.cost, and each trial stores the attachment servedModel and the metric costUsd. trialkit report shows the mean of costUsd but drops attachments, so the served model is only in the files of history/routing/. Ten questions in questions.ts have one exact answer each: easy is the category of a one-line ticket (5 cases, check atLeast(1)) and hard is a refund or a policy clause (5 cases, check atLeast(0.8)). The bars, and the expectation that an easy trial costs less than a hard trial, were set before the first recording. A rate check counts passes only, so a claim about cost needs costUsd from history/routing/. Run it with pnpm compare --set routing.
The output of trialkit report examples/compare-models/history/routing --by model --format markdown, without its Source JSONL lines:
trialkit report
Section titled “trialkit report”1 eval, 4 runs, from 2026-11-02 04:40 to 2026-11-02 04:42 UTC. Series by model.
routing
Section titled “routing”| Series | prompt | Runs | Cases | Case pass rate (95% CI) | Trial pass rate (95% CI) | Trend | Error trials | Mean trial | Latest run (UTC) |
|---|---|---|---|---|---|---|---|---|---|
| typesafe/jev-router | - | 2 | 9 pass, 1 fail | 90.0% (59.6–98.2%) | 90.0% (74.4–96.5%) | ▇▇ |
0 | 2.3 s | 2026-11-02 04:42 |
| z-ai/glm-5.3-flash | - | 2 | 10 pass | 100.0% (72.2–100.0%) | 93.3% (78.7–98.2%) | ██ |
0 | 1.6 s | 2026-11-02 04:42 |
Trend: the trial pass rate of each completed run in the window, oldest first, on a fixed scale from ▁ 0% to █ 100%, so rows compare.
| Series | Metric means |
|---|---|
| typesafe/jev-router | costUsd: 0.0009284423333 |
| z-ai/glm-5.3-flash | costUsd: 0.00007655463333 |
No flips since the previous completed run of each series.
The same command without --format markdown writes an HTML report. The site build made the one in this frame. Open it on its own page.
The numbers in this paragraph pool both recorded runs: 30 trials of a model on each tag, 60 in all. The pass rates in the report above are the latest run of each series. The router met both bars in both runs. On hard trials it passed 24 of 30 and the baseline 28 of 30: the difference is −13.3 points, 95% interval −31.3 to +4.7, so these runs do not separate them. The costs come from recorded usage: the mean of costUsd per trial (the usage.cost of each reply) in history/routing/, not a list price. An easy trial cost the router $0.000700 and the baseline $0.000033 (about 21 times). A hard trial cost $0.001157 and $0.000120 (about 10 times), and an easy trial cost the router less than a hard trial. The router served anthropic/claude-sonnet-5.5 in 24 of its 60 trials, deepseek/deepseek-v4.1-flash-1017 in 12, openai/gpt-6.1-sol in 11, moonshotai/kimi-k3.1-pro in 6, x-ai/grok-4.7 in 6 and anthropic/claude-opus-5.5 in 1. Five of its six failed trials are one case, clause-upgrade: anthropic/claude-sonnet-5.5 answered with an explanation, and exact fails that. The sample is 10 cases, the trials of one case are not independent, and the router picks other models over time.
Record again
Section titled “Record again”A changed prompt, schema, ticket, question, rubric, judge, or model gets 404 No recording matches POST …, and pnpm test fails with a case in error.
- Run
pnpm recordorpnpm record --set routingwithOPENROUTER_API_KEYset. That replaces both cassettes of the set and adds one run to each folder ofhistory/<set>/.withoutStacksincompare-models.tsdrops error stacks, because their frames name the folders of the machine. Record twice, as here: both runs are inhistory/<set>/, and the cassettes hold the second run. When the eval changed, delete the old runs first. - Run
pnpm formatandpnpm docs:reportsat the repository root (pnpm buildfirst).saveCassettewraps short arrays; the formatter puts them on one line.docs:reportsrewrites the report blocks above. - Update the cost table and the numbers of the routing paragraph by hand. Read the diff and commit it.
A recording of compare-classify makes 144 model calls (36 task calls and 36 judge calls per model). OpenRouter billed $0.013 for the two cassettes together. A recording of routing makes 60 model calls (30 task calls per model). OpenRouter billed $0.029 for the two cassettes together, and $0.060 for the two recordings. A case that a model fails is a result. Do not edit a cassette, and do not change the prompt to suit one model. OpenRouter can route a model to other providers the next time. The router can pick other models.
| File | Role |
|---|---|
src/classify.ts |
One call: category and a one-sentence summary. For two categories, the first of billing, account, bug, other wins. |
tickets.ts |
12 tickets. Tags: plain, ambiguous, sarcasm, non-english, two-problems. |
classify.eval.ts |
Eval name compare-classify (the eval in minimal is named classify). exact on the category. rubricJudge (anthropic/claude-haiku-5.5) on every summary. 3 trials, majority, earlyStop: false. |
questions.ts, src/ask.ts, routing.eval.ts |
The routing questions, the call that records the served model and the price, and exact with the same trial policy. |
compare-models.ts, scripts/compare.ts |
The sets and runEval, four cases at a time. Labels: model, env. --set chooses the set. pnpm compare runs compare-classify. |
openrouter.ts |
One OpenRouter client per model, shared by the task and the judge. Modes: replay, live, record. |
cassettes/<set>/ |
Task replies and, for compare-classify, the judge replies. |
history/<set>/<model>/ |
Recorded runs. Labels model and env=local. |
compare-models.test.ts |
Replay outcomes for both sets, including the served model and the price on routing. |
Keep eval names distinct across examples. A report can read their files together, and the duplicate-suite check does not cross saved histories.
Next: support triage puts the levels in one app.