Skip to content

Example: config from a database

Copy this when a model config lives outside the repository and trialkit compare must block a bad candidate before it goes live. trialkit stores nothing: it records the config of each run, and the studio edits the candidate through the functions of this app.

db.json stands in for the product database. Its table configs has one row for each named config. The app reads the row production. The row candidate is the edit someone wants to ship. The model, the first prompt, and the response schema match. effort and systemPrompt differ.

Field production candidate
effort medium low
systemPrompt The four categories, and this rule: the first of billing, account, bug, other wins. The four categories, and a shorter rule: the first problem in the ticket wins.

Its table tickets holds the cases of the eval: 7 rows with the fields of an eval case (id, input, expected, tags), and a revision that each save raises. src/db.ts reads it with an async query, and ticketsDataset in triage.eval.ts checks the rows with casesFromRows. The dataset id of a run is tickets-r<revision>, so a run whose dataset hash differs shows which revision of the table it used. The tag of a ticket is its kind: trialkit compare shows a pass rate for each tag, and its gate fails when a case tagged regression goes from pass to fail.

Run these in this directory. Node.js 22.23.4 or later. In PowerShell, set the key first: $env:OPENROUTER_API_KEY = 'sk-or-...'.

Command What it does Cost
pnpm test Replay cassette.json. The gate on production, the unit tests, and the recorded promotion check. No network and no key. Nothing
pnpm promote candidate Replay on production and on candidate, then trialkit compare. Exit code 0: promote (the script prints the update). 1: do not. Nothing
pnpm promote candidate --live The real model. Use it after you edit a row: a replay knows only the recorded configs. A new sample can differ from the recording. About 0.1 cent
OPENROUTER_API_KEY=sk-or-... pnpm record Live run. Replaces cassette.json and adds one run of each config to history/. About 0.1 cent
OPENROUTER_API_KEY=sk-or-... pnpm studio Serve the studio for the row candidate and the table tickets. See The studio. About 0.04 cent for each run of 21 calls
pnpm report trialkit report history --by config,configName. --format markdown prints a table. Nothing
$ pnpm promote candidate
ticket-category [model=replay:z-ai/glm-5.3-flash, configName=production]: 7 pass (run f578b95e-b13b-474d-a2a6-24c90a86d85c)
ticket-category [model=replay:z-ai/glm-5.3-flash, configName=candidate]: 5 pass, 2 fail (run 33dcf4a6-57f8-462e-8cec-4086a20d5d39)
✗ crash-and-charge fail 0 of 3 trials passed, needs 2; category: expected "billing", got "bug"
✗ feature-and-login fail 0 of 3 trials passed, needs 2; category: expected "account", got "other"
trialkit compare
================
1 eval compared; 1 with a regression. Runs are paired by eval name. A change is head minus base, in percentage points (pp), with its 95% interval.
ticket-category
---------------
Regression: 2 cases tagged regression changed from pass to fail or error (crash-and-charge, feature-and-login); the trial pass rate changed by -23.8 pp (95% CI -45.6 to -0.6).
- Base: run f578b95e-b13b-474d-a2a6-24c90a86d85c, started 2026-11-02 05:05 UTC
- Head: run 33dcf4a6-57f8-462e-8cec-4086a20d5d39, started 2026-11-02 05:05 UTC
- Config that differs: effort: medium → low · systemPrompt: sha256:70d45ff02609… → sha256:5765b13da035…
- Labels that differ: configName production → candidate
2 cases changed outcome (outcome, passed / counted trials), regressions first:
Case Base Head Kind
----------------- ---------- ---------- ----------
crash-and-charge pass (3/3) fail (0/3) regression
feature-and-login pass (2/3) fail (0/3) regression
…
trialkit: regression in 1 eval:
- ticket-category: 2 cases tagged regression changed from pass to fail or error (crash-and-charge, feature-and-login); the trial pass rate changed by -23.8 pp (95% CI -45.6 to -0.6)
Do not promote "candidate": trialkit compare exited with code 1. "production" stays as it is.

The run ids change in each run. The output above cuts the pass-rate tables (…). The history is history/ for a recording and a temporary folder otherwise. This command, with the short hashes a report shows, gives the same decision on history/:

Terminal window
trialkit compare history history --by config --base-config sha256:e45aace9b6d9 --head-config sha256:3224dc6fbb33 --fail-on-regression
  • triage.eval.test.ts runs the row production, the row the app reads. A failing case fails the test. An edited row, prompt, ticket, or src/classify.ts is a new request: replay answers 404 and the trial ends in error. The run records the config and its sha256: hash. The label configName is the row. The label model is replay:<model> on a replay.
  • --by config pairs the two runs. --base-config and --head-config take a hash (full, or the digits a report shows) or field=value, for example effort=low. A config difference stays comparable: it is the change under test. A dataset change does not. --fail-on-regression exits 1 when a regression case goes from pass to fail or error, or when the interval of the trial-pass-rate change is below 0. Exit code 2 is a bad argument. The script uses that code and never writes the database. When the candidate passes, the script prints the update and the deploy step applies it.
  • The same steps gate a file in the pull request, an MLflow alias (loadPrompt({ host, name, alias }) from @trialkit/report-mlflow), or a gateway route: pass the active config as config and compare it with the candidate. See Gate a config change.

pnpm studio serves a page that edits the row candidate and the table tickets, runs the eval on your changes, and compares them with the saved row. triage.studio.ts connects the page to src/db.ts:

export default studio({
eval: (config: Config) => triageEval(clientFor('live'), config, dataset),
runOptions: (config) => ({
labels: runOptions('live', CANDIDATE, config).labels,
concurrency: 4,
}),
config: {
load: () => readConfig(CANDIDATE),
save: (next, { loaded }) => writeCandidate(next, { expect: loaded }),
},
cases: {
load: async () => (await queryTickets()).rows,
save: (next, { loaded }) => writeTickets(next, { expect: loaded }),
},
});
  • Run OPENROUTER_API_KEY=sk-or-... pnpm studio and open the address that it prints. A run calls the real model: the 7 tickets with 3 trials make 21 calls. Runs go to eval-studio/, which Git ignores.
  • A save writes db.json, the committed stand-in database, and refuses a write when the row or the table changed since the page loaded it. The stand-in does the check and the write in one synchronous step and replaces the file atomically; a real database does both in one transaction (or an UPDATE … WHERE revision = ?), and one studio serializes its own actions. No function writes production. pnpm test then fails until you record again; git checkout db.json undoes the save.
  • The studio does not promote. After a save, pnpm promote candidate --live runs both configs and trialkit compare decides. The package README has the options and the limits.

The output of trialkit report examples/config-from-db/history --by config,configName --format markdown, without its Source JSONL lines:

1 eval, 4 runs, from 2026-11-02 05:03 to 2026-11-02 05:03 UTC. Series by config · configName.

Series prompt Runs Cases Case pass rate (95% CI) Trial pass rate (95% CI) Trend Error trials Mean trial Latest run (UTC)
sha256:3224dc6fbb33… · candidate - 2 5 pass, 2 fail 71.4% (35.9–91.8%) 71.4% (50.0–86.2%) ▆▆ 0 447 ms 2026-11-02 05:03
sha256:e45aace9b6d9… · production - 2 7 pass 100.0% (64.6–100.0%) 95.2% (77.3–99.2%) ██ 0 648 ms 2026-11-02 05:03

Trend: the trial pass rate of each completed run in the window, oldest first, on a fixed scale from ▁ 0% to █ 100%, so rows compare.

No flips since the previous completed run of each series.

The same command without --format markdown writes an HTML report. The site build made the one in this frame. Open it on its own page.

--by config,configName makes one series per config hash and row name. Production passed all 7 cases in both recordings. The candidate failed crash-and-charge and feature-and-login in both.

A changed row of db.json (a config or a ticket), prompt, or src/classify.ts gets 404 No recording matches POST …. pnpm test then fails with cases in error.

  1. Run pnpm record with OPENROUTER_API_KEY set. That replaces cassette.json and adds one run of each config to history/. Record twice, as here: both runs are in history/, and the cassette holds the second. When the tickets or the check changed, delete history/ first. withoutStacks in promote.ts drops error stacks. pnpm check:docs fails for a history that names a folder of the recording machine.
  2. Run pnpm format and pnpm docs:reports at the repository root (pnpm build first). saveCassette wraps short arrays; the formatter puts them on one line. docs:reports rewrites the report above.
  3. Read the diff, update the numbers in this file, and commit. Do not edit the cassette. promote.test.ts expects this recorded decision, a candidate that does not pass. When you change the candidate so that it passes, record again and update that test and the numbers here.
File Role
db.json, src/db.ts The tables configs and tickets, and the functions that read and write them: readConfig, writeCandidate, queryTickets and writeTickets. DB_FILE names another copy of the file.
src/classify.ts One model call. The request comes from the config and the ticket: model, reasoning_effort, prompts, response schema.
triage.eval.ts ticketsDataset() checks the rows of the table with casesFromRows. triageEval(client, config, dataset): 3 trials, earlyStop: false, exact on the category.
triage.eval.test.ts The gate on production.
triage.studio.ts The studio file.
triage.studio.test.ts Its functions on a copy of db.json: a stale save is refused, and a save round-trips.
promote.ts, scripts/promote.ts Both configs, then trialkit compare.
openrouter.ts The OpenRouter client. Replay, live, or record. One cassette for every config.
cassette.json, history/ The latest recording, and each run with its config.