Skip to content

Example: compare models

Copy this when you compare models on one eval and a wrong answer should stay in the record as a result. Two sets: compare-classify (the default) and routing (Routing).

Run these in this directory. Node.js 22.23.4 or later. In PowerShell, set the key first: $env:OPENROUTER_API_KEY = 'sk-or-...'.

Command What it does Cost
pnpm test Replay. Every case gets a verdict, and the outcomes equal the latest run in history/<set>/<model>/. A wrong answer does not fail the test. Nothing
pnpm compare Replay of compare-classify. Prints each model. Exit code 1 only when a case gets no verdict. Nothing
pnpm compare --set routing The same for routing. --live and pnpm record take --set too. Nothing
pnpm compare --live Live models. Prints the result and keeps no record. About 1.3 cents (routing: 3)
OPENROUTER_API_KEY=sk-or-... pnpm record Live run. Replaces both cassettes of the set and adds one run under history/<set>/. About 1.3 cents (routing: 3)
pnpm report trialkit report history --by model writes trialkit-report.html. --format markdown prints a table. Nothing
pnpm exec trialkit compare history/compare-classify/z-ai-glm-5.3-flash history/compare-classify/inclusionai-ling-3.0-flash --by= Latest run of each model, case by case. --by= pairs by eval name, so each directory holds one model. Nothing
$ pnpm compare
compare-classify [model=replay:z-ai/glm-5.3-flash, env=local]: 11 pass, 1 fail (run bfcaa22b-3310-404f-8a07-3a8015c37781)
✗ invite-greyed fail 1 of 3 trials passed, needs 2; category: expected "account", got "bug"
compare-classify [model=replay:inclusionai/ling-3.0-flash, env=local]: 12 pass (run db9b6ffa-a92a-464a-8618-d4b54500b443)

The run ids change in each run. The label replay: keeps a replayed run apart from a real one.

Both task models think before they answer, and a thinking token is billed as output. Of the task-call output tokens in the recording, 78% (GLM) and 91% (Ling) are thinking tokens. The cost table in the next section is the sum of usage.cost in the cassette of the second run (36 task replies and 36 judge replies), so it holds what the providers that answered charged. The provider field of each reply names them: Wafer, OpenInference, InferenceNet, Cohere, DeepInfra and DigitalOcean answered for GLM, DeepInfra for Ling, and Azure for the judge. For list prices, see the OpenRouter pages of z-ai/glm-5.3-flash, inclusionai/ling-3.0-flash and anthropic/claude-haiku-5.5.

The output of trialkit report examples/compare-models/history/compare-classify --by model --format markdown, without its Source JSONL lines:

1 eval, 4 runs, from 2026-11-02 03:28 to 2026-11-02 03:29 UTC. Series by model.

Series prompt Runs Cases Case pass rate (95% CI) Trial pass rate (95% CI) Trend Error trials Mean trial Latest run (UTC)
inclusionai/ling-3.0-flash - 2 12 pass 100.0% (75.8–100.0%) 97.2% (85.8–99.5%) ██ 0 5.1 s 2026-11-02 03:29
z-ai/glm-5.3-flash - 2 11 pass, 1 fail 91.7% (64.6–98.5%) 94.4% (81.9–98.5%) ▇█ 0 2.7 s 2026-11-02 03:29

Trend: the trial pass rate of each completed run in the window, oldest first, on a fixed scale from ▁ 0% to █ 100%, so rows compare.

Series Tokens in Tokens out
inclusionai/ling-3.0-flash 54563 8185
z-ai/glm-5.3-flash 54673 8415

No flips since the previous completed run of each series.

The same command without --format markdown writes an HTML report. The site build made the one in this frame. Open it on its own page.

The report has no cost. This table adds it, from the cassette, and the cases that did not pass.

Model Cost of the run Task calls Judge calls Cases that failed Cases with trials that disagreed
z-ai/glm-5.3-flash $0.0067 $0.0019 $0.0048 invite-greyed in both runs Run 2: invite-greyed (1 of 3 trials passed)
inclusionai/ling-3.0-flash $0.0062 $0.0014 $0.0049 None Run 2: sarcasm-crash (2 of 3 trials passed)
  • The 95% intervals of the trial pass rates overlap (94.4%, 81.9–98.5%, and 97.2%, 85.8–99.5%), and so do the case pass rates. trialkit compare reports Ling 2.8 points higher, 95% interval −9.3 to +15.6, on 36 trials (an interval about 14 to 17 points wide). The models differ on invite-greyed only: the prompt rule is account; GLM answered bug in 5 of 6 trials and Ling answered account in 6 of 6. In the cassette, OpenInference served GLM’s three trials of that ticket. Nothing pins a provider, so the comparison measures what OpenRouter routed. Pin one provider with only and allow_fallbacks (provider routing). In run 2 the judge failed one Ling summary for adding the word “recent”.
  • The 36 judge calls cost 2.5 to 3.5 times the 36 task calls. The judge prompt holds the rubric, the ticket, and the summary.
  • A case in error (a stale cassette answers 404) fails pnpm test. A wrong recorded answer leaves the test green. The example writes no eval-results/ and stays out of check.yml. The comparison is a script, the shape in the guide.

The routing set compares typesafe/jev-router, a router model on OpenRouter that picks a model and a reasoning effort for each request, with z-ai/glm-5.3-flash. The models API lists no price (-1): a request costs what the picked model costs. So src/ask.ts returns the answer, completion.model and usage.cost, and each trial stores the attachment servedModel and the metric costUsd. trialkit report shows the mean of costUsd but drops attachments, so the served model is only in the files of history/routing/. Ten questions in questions.ts have one exact answer each: easy is the category of a one-line ticket (5 cases, check atLeast(1)) and hard is a refund or a policy clause (5 cases, check atLeast(0.8)). The bars, and the expectation that an easy trial costs less than a hard trial, were set before the first recording. A rate check counts passes only, so a claim about cost needs costUsd from history/routing/. Run it with pnpm compare --set routing.

The output of trialkit report examples/compare-models/history/routing --by model --format markdown, without its Source JSONL lines:

1 eval, 4 runs, from 2026-11-02 04:40 to 2026-11-02 04:42 UTC. Series by model.

Series prompt Runs Cases Case pass rate (95% CI) Trial pass rate (95% CI) Trend Error trials Mean trial Latest run (UTC)
typesafe/jev-router - 2 9 pass, 1 fail 90.0% (59.6–98.2%) 90.0% (74.4–96.5%) ▇▇ 0 2.3 s 2026-11-02 04:42
z-ai/glm-5.3-flash - 2 10 pass 100.0% (72.2–100.0%) 93.3% (78.7–98.2%) ██ 0 1.6 s 2026-11-02 04:42

Trend: the trial pass rate of each completed run in the window, oldest first, on a fixed scale from ▁ 0% to █ 100%, so rows compare.

Series Metric means
typesafe/jev-router costUsd: 0.0009284423333
z-ai/glm-5.3-flash costUsd: 0.00007655463333

No flips since the previous completed run of each series.

The same command without --format markdown writes an HTML report. The site build made the one in this frame. Open it on its own page.

The numbers in this paragraph pool both recorded runs: 30 trials of a model on each tag, 60 in all. The pass rates in the report above are the latest run of each series. The router met both bars in both runs. On hard trials it passed 24 of 30 and the baseline 28 of 30: the difference is −13.3 points, 95% interval −31.3 to +4.7, so these runs do not separate them. The costs come from recorded usage: the mean of costUsd per trial (the usage.cost of each reply) in history/routing/, not a list price. An easy trial cost the router $0.000700 and the baseline $0.000033 (about 21 times). A hard trial cost $0.001157 and $0.000120 (about 10 times), and an easy trial cost the router less than a hard trial. The router served anthropic/claude-sonnet-5.5 in 24 of its 60 trials, deepseek/deepseek-v4.1-flash-1017 in 12, openai/gpt-6.1-sol in 11, moonshotai/kimi-k3.1-pro in 6, x-ai/grok-4.7 in 6 and anthropic/claude-opus-5.5 in 1. Five of its six failed trials are one case, clause-upgrade: anthropic/claude-sonnet-5.5 answered with an explanation, and exact fails that. The sample is 10 cases, the trials of one case are not independent, and the router picks other models over time.

A changed prompt, schema, ticket, question, rubric, judge, or model gets 404 No recording matches POST …, and pnpm test fails with a case in error.

  1. Run pnpm record or pnpm record --set routing with OPENROUTER_API_KEY set. That replaces both cassettes of the set and adds one run to each folder of history/<set>/. withoutStacks in compare-models.ts drops error stacks, because their frames name the folders of the machine. Record twice, as here: both runs are in history/<set>/, and the cassettes hold the second run. When the eval changed, delete the old runs first.
  2. Run pnpm format and pnpm docs:reports at the repository root (pnpm build first). saveCassette wraps short arrays; the formatter puts them on one line. docs:reports rewrites the report blocks above.
  3. Update the cost table and the numbers of the routing paragraph by hand. Read the diff and commit it.

A recording of compare-classify makes 144 model calls (36 task calls and 36 judge calls per model). OpenRouter billed $0.013 for the two cassettes together. A recording of routing makes 60 model calls (30 task calls per model). OpenRouter billed $0.029 for the two cassettes together, and $0.060 for the two recordings. A case that a model fails is a result. Do not edit a cassette, and do not change the prompt to suit one model. OpenRouter can route a model to other providers the next time. The router can pick other models.

File Role
src/classify.ts One call: category and a one-sentence summary. For two categories, the first of billing, account, bug, other wins.
tickets.ts 12 tickets. Tags: plain, ambiguous, sarcasm, non-english, two-problems.
classify.eval.ts Eval name compare-classify (the eval in minimal is named classify). exact on the category. rubricJudge (anthropic/claude-haiku-5.5) on every summary. 3 trials, majority, earlyStop: false.
questions.ts, src/ask.ts, routing.eval.ts The routing questions, the call that records the served model and the price, and exact with the same trial policy.
compare-models.ts, scripts/compare.ts The sets and runEval, four cases at a time. Labels: model, env. --set chooses the set. pnpm compare runs compare-classify.
openrouter.ts One OpenRouter client per model, shared by the task and the judge. Modes: replay, live, record.
cassettes/<set>/ Task replies and, for compare-classify, the judge replies.
history/<set>/<model>/ Recorded runs. Labels model and env=local.
compare-models.test.ts Replay outcomes for both sets, including the served model and the price on routing.

Keep eval names distinct across examples. A report can read their files together, and the duplicate-suite check does not cross saved histories.

Next: support triage puts the levels in one app.