History and reports
Use JSON Lines history when you need trends, a pull-request gate or a comparison between runs.
trialkit report eval-history --by model,prompttrialkit report eval-history --by model --format markdowntrialkit compare eval-history eval-results --format markdown --out comparison.md --fail-on-regression- Keep one file per run in a branch, bucket or CI artifact. Label runs with
model;trialkit comparepairs by eval name and model label, and leaves out an input that changed.examples/compare-modelscompares models this way. - Reports show run trends with 95% intervals, case consistency, score histograms and flips. They warn when a dataset, policy, scorer version or judge changes;
earlyStop: truecan bias multi-trial rates. --fail-on-regressiongates regression-tag flips, set-check flips, a former verdict that becomes error, a significant trial-rate fall, incomplete head runs and removed gated cases. Other flips are reported as noise.
For first history, use --allow-missing-base deliberately. This repository instead requires base preparation and at least one compared series; unmatched head series are notCompared. See the CLI reference and examples/history, which gates a pull request.
Gate a config change
Section titled “Gate a config change”Use config when the setting under test lives outside the code.
trialkit compare history history --by config --base-config sha256:<12-hex> --head-config sha256:<12-hex> --fail-on-regressionconfig is a JSON record in RunStarted, not an applied setting. Reports group with --by config or a field such as --by config.effort; comparison lists changed config fields without treating them as a regression. Keep secrets out of it. examples/config-from-db shows the pattern, and its studio (alpha, parked) edits the candidate before the gate sees it.
An example report
Section titled “An example report”The generated report below comes from the committed history of examples/compare-models. pnpm docs:reports rewrites it; do not edit the generated block. The example READMEs show the other reports: support-triage, config-from-db and the routing report of compare-models.
The output of trialkit report examples/compare-models/history/compare-classify --by model --format markdown, without its Source JSONL lines:
trialkit report
Section titled “trialkit report”1 eval, 4 runs, from 2026-11-02 03:28 to 2026-11-02 03:29 UTC. Series by model.
compare-classify
Section titled “compare-classify”| Series | prompt | Runs | Cases | Case pass rate (95% CI) | Trial pass rate (95% CI) | Trend | Error trials | Mean trial | Latest run (UTC) |
|---|---|---|---|---|---|---|---|---|---|
| inclusionai/ling-3.0-flash | - | 2 | 12 pass | 100.0% (75.8–100.0%) | 97.2% (85.8–99.5%) | ██ |
0 | 5.1 s | 2026-11-02 03:29 |
| z-ai/glm-5.3-flash | - | 2 | 11 pass, 1 fail | 91.7% (64.6–98.5%) | 94.4% (81.9–98.5%) | ▇█ |
0 | 2.7 s | 2026-11-02 03:29 |
Trend: the trial pass rate of each completed run in the window, oldest first, on a fixed scale from ▁ 0% to █ 100%, so rows compare.
| Series | Tokens in | Tokens out |
|---|---|---|
| inclusionai/ling-3.0-flash | 54563 | 8185 |
| z-ai/glm-5.3-flash | 54673 | 8415 |
No flips since the previous completed run of each series.
The same command without --format markdown writes an HTML report. The site build made the one in this frame. Open it on its own page.