Example: config from a database
Copy this when a model config lives outside the repository and trialkit compare must block a bad candidate before it goes live. trialkit stores nothing: it records the config of each run, and the studio edits the candidate through the functions of this app.
db.json stands in for the product database. Its table configs has one row for each named config. The app reads the row production. The row candidate is the edit someone wants to ship. The model, the first prompt, and the response schema match. effort and systemPrompt differ.
| Field | production |
candidate |
|---|---|---|
effort |
medium |
low |
systemPrompt |
The four categories, and this rule: the first of billing, account, bug, other wins. | The four categories, and a shorter rule: the first problem in the ticket wins. |
Its table tickets holds the cases of the eval: 7 rows with the fields of an eval case (id, input, expected, tags), and a revision that each save raises. src/db.ts reads it with an async query, and ticketsDataset in triage.eval.ts checks the rows with casesFromRows. The dataset id of a run is tickets-r<revision>, so a run whose dataset hash differs shows which revision of the table it used. The tag of a ticket is its kind: trialkit compare shows a pass rate for each tag, and its gate fails when a case tagged regression goes from pass to fail.
Run these in this directory. Node.js 22.23.4 or later. In PowerShell, set the key first: $env:OPENROUTER_API_KEY = 'sk-or-...'.
| Command | What it does | Cost |
|---|---|---|
pnpm test |
Replay cassette.json. The gate on production, the unit tests, and the recorded promotion check. No network and no key. |
Nothing |
pnpm promote candidate |
Replay on production and on candidate, then trialkit compare. Exit code 0: promote (the script prints the update). 1: do not. |
Nothing |
pnpm promote candidate --live |
The real model. Use it after you edit a row: a replay knows only the recorded configs. A new sample can differ from the recording. | About 0.1 cent |
OPENROUTER_API_KEY=sk-or-... pnpm record |
Live run. Replaces cassette.json and adds one run of each config to history/. |
About 0.1 cent |
OPENROUTER_API_KEY=sk-or-... pnpm studio |
Serve the studio for the row candidate and the table tickets. See The studio. |
About 0.04 cent for each run of 21 calls |
pnpm report |
trialkit report history --by config,configName. --format markdown prints a table. |
Nothing |
The promotion check
Section titled “The promotion check”$ pnpm promote candidateticket-category [model=replay:z-ai/glm-5.3-flash, configName=production]: 7 pass (run f578b95e-b13b-474d-a2a6-24c90a86d85c)ticket-category [model=replay:z-ai/glm-5.3-flash, configName=candidate]: 5 pass, 2 fail (run 33dcf4a6-57f8-462e-8cec-4086a20d5d39) ✗ crash-and-charge fail 0 of 3 trials passed, needs 2; category: expected "billing", got "bug" ✗ feature-and-login fail 0 of 3 trials passed, needs 2; category: expected "account", got "other"trialkit compare================
1 eval compared; 1 with a regression. Runs are paired by eval name. A change is head minus base, in percentage points (pp), with its 95% interval.
ticket-category---------------
Regression: 2 cases tagged regression changed from pass to fail or error (crash-and-charge, feature-and-login); the trial pass rate changed by -23.8 pp (95% CI -45.6 to -0.6).
- Base: run f578b95e-b13b-474d-a2a6-24c90a86d85c, started 2026-11-02 05:05 UTC- Head: run 33dcf4a6-57f8-462e-8cec-4086a20d5d39, started 2026-11-02 05:05 UTC- Config that differs: effort: medium → low · systemPrompt: sha256:70d45ff02609… → sha256:5765b13da035…- Labels that differ: configName production → candidate
2 cases changed outcome (outcome, passed / counted trials), regressions first:
Case Base Head Kind----------------- ---------- ---------- ----------crash-and-charge pass (3/3) fail (0/3) regressionfeature-and-login pass (2/3) fail (0/3) regression
…trialkit: regression in 1 eval:- ticket-category: 2 cases tagged regression changed from pass to fail or error (crash-and-charge, feature-and-login); the trial pass rate changed by -23.8 pp (95% CI -45.6 to -0.6)
Do not promote "candidate": trialkit compare exited with code 1. "production" stays as it is.The run ids change in each run. The output above cuts the pass-rate tables (…). The history is history/ for a recording and a temporary folder otherwise. This command, with the short hashes a report shows, gives the same decision on history/:
trialkit compare history history --by config --base-config sha256:e45aace9b6d9 --head-config sha256:3224dc6fbb33 --fail-on-regressiontriage.eval.test.tsruns the rowproduction, the row the app reads. A failing case fails the test. An edited row, prompt, ticket, orsrc/classify.tsis a new request: replay answers 404 and the trial ends inerror. The run records the config and itssha256:hash. The labelconfigNameis the row. The labelmodelisreplay:<model>on a replay.--by configpairs the two runs.--base-configand--head-configtake a hash (full, or the digits a report shows) orfield=value, for exampleeffort=low. A config difference stays comparable: it is the change under test. A dataset change does not.--fail-on-regressionexits 1 when aregressioncase goes from pass to fail orerror, or when the interval of the trial-pass-rate change is below 0. Exit code 2 is a bad argument. The script uses that code and never writes the database. When the candidate passes, the script prints the update and the deploy step applies it.- The same steps gate a file in the pull request, an MLflow alias (
loadPrompt({ host, name, alias })from@trialkit/report-mlflow), or a gateway route: pass the active config asconfigand compare it with the candidate. See Gate a config change.
The studio
Section titled “The studio”pnpm studio serves a page that edits the row candidate and the table tickets, runs the eval on your changes, and compares them with the saved row. triage.studio.ts connects the page to src/db.ts:
export default studio({ eval: (config: Config) => triageEval(clientFor('live'), config, dataset), runOptions: (config) => ({ labels: runOptions('live', CANDIDATE, config).labels, concurrency: 4, }), config: { load: () => readConfig(CANDIDATE), save: (next, { loaded }) => writeCandidate(next, { expect: loaded }), }, cases: { load: async () => (await queryTickets()).rows, save: (next, { loaded }) => writeTickets(next, { expect: loaded }), },});- Run
OPENROUTER_API_KEY=sk-or-... pnpm studioand open the address that it prints. A run calls the real model: the 7 tickets with 3 trials make 21 calls. Runs go toeval-studio/, which Git ignores. - A save writes
db.json, the committed stand-in database, and refuses a write when the row or the table changed since the page loaded it. The stand-in does the check and the write in one synchronous step and replaces the file atomically; a real database does both in one transaction (or anUPDATE … WHERE revision = ?), and one studio serializes its own actions. No function writesproduction.pnpm testthen fails until you record again;git checkout db.jsonundoes the save. - The studio does not promote. After a save,
pnpm promote candidate --liveruns both configs andtrialkit comparedecides. The package README has the options and the limits.
The result
Section titled “The result”The output of trialkit report examples/config-from-db/history --by config,configName --format markdown, without its Source JSONL lines:
trialkit report
Section titled “trialkit report”1 eval, 4 runs, from 2026-11-02 05:03 to 2026-11-02 05:03 UTC. Series by config · configName.
ticket-category
Section titled “ticket-category”| Series | prompt | Runs | Cases | Case pass rate (95% CI) | Trial pass rate (95% CI) | Trend | Error trials | Mean trial | Latest run (UTC) |
|---|---|---|---|---|---|---|---|---|---|
| sha256:3224dc6fbb33… · candidate | - | 2 | 5 pass, 2 fail | 71.4% (35.9–91.8%) | 71.4% (50.0–86.2%) | ▆▆ |
0 | 447 ms | 2026-11-02 05:03 |
| sha256:e45aace9b6d9… · production | - | 2 | 7 pass | 100.0% (64.6–100.0%) | 95.2% (77.3–99.2%) | ██ |
0 | 648 ms | 2026-11-02 05:03 |
Trend: the trial pass rate of each completed run in the window, oldest first, on a fixed scale from ▁ 0% to █ 100%, so rows compare.
No flips since the previous completed run of each series.
The same command without --format markdown writes an HTML report. The site build made the one in this frame. Open it on its own page.
--by config,configName makes one series per config hash and row name. Production passed all 7 cases in both recordings. The candidate failed crash-and-charge and feature-and-login in both.
Record again
Section titled “Record again”A changed row of db.json (a config or a ticket), prompt, or src/classify.ts gets 404 No recording matches POST …. pnpm test then fails with cases in error.
- Run
pnpm recordwithOPENROUTER_API_KEYset. That replacescassette.jsonand adds one run of each config tohistory/. Record twice, as here: both runs are inhistory/, and the cassette holds the second. When the tickets or the check changed, deletehistory/first.withoutStacksinpromote.tsdrops error stacks.pnpm check:docsfails for a history that names a folder of the recording machine. - Run
pnpm formatandpnpm docs:reportsat the repository root (pnpm buildfirst).saveCassettewraps short arrays; the formatter puts them on one line.docs:reportsrewrites the report above. - Read the diff, update the numbers in this file, and commit. Do not edit the cassette.
promote.test.tsexpects this recorded decision, a candidate that does not pass. When you change the candidate so that it passes, record again and update that test and the numbers here.
| File | Role |
|---|---|
db.json, src/db.ts |
The tables configs and tickets, and the functions that read and write them: readConfig, writeCandidate, queryTickets and writeTickets. DB_FILE names another copy of the file. |
src/classify.ts |
One model call. The request comes from the config and the ticket: model, reasoning_effort, prompts, response schema. |
triage.eval.ts |
ticketsDataset() checks the rows of the table with casesFromRows. triageEval(client, config, dataset): 3 trials, earlyStop: false, exact on the category. |
triage.eval.test.ts |
The gate on production. |
triage.studio.ts |
The studio file. |
triage.studio.test.ts |
Its functions on a copy of db.json: a stale save is refused, and a save round-trips. |
promote.ts, scripts/promote.ts |
Both configs, then trialkit compare. |
openrouter.ts |
The OpenRouter client. Replay, live, or record. One cassette for every config. |
cassette.json, history/ |
The latest recording, and each run with its config. |