Skip to content

History and reports

Use JSON Lines history when you need trends, a pull-request gate or a comparison between runs.

Terminal window
trialkit report eval-history --by model,prompt
trialkit report eval-history --by model --format markdown
trialkit compare eval-history eval-results --format markdown --out comparison.md --fail-on-regression
  • Keep one file per run in a branch, bucket or CI artifact. Label runs with model; trialkit compare pairs by eval name and model label, and leaves out an input that changed. examples/compare-models compares models this way.
  • Reports show run trends with 95% intervals, case consistency, score histograms and flips. They warn when a dataset, policy, scorer version or judge changes; earlyStop: true can bias multi-trial rates.
  • --fail-on-regression gates regression-tag flips, set-check flips, a former verdict that becomes error, a significant trial-rate fall, incomplete head runs and removed gated cases. Other flips are reported as noise.

For first history, use --allow-missing-base deliberately. This repository instead requires base preparation and at least one compared series; unmatched head series are notCompared. See the CLI reference and examples/history, which gates a pull request.

Use config when the setting under test lives outside the code.

Terminal window
trialkit compare history history --by config --base-config sha256:<12-hex> --head-config sha256:<12-hex> --fail-on-regression

config is a JSON record in RunStarted, not an applied setting. Reports group with --by config or a field such as --by config.effort; comparison lists changed config fields without treating them as a regression. Keep secrets out of it. examples/config-from-db shows the pattern, and its studio (alpha, parked) edits the candidate before the gate sees it.

The generated report below comes from the committed history of examples/compare-models. pnpm docs:reports rewrites it; do not edit the generated block. The example READMEs show the other reports: support-triage, config-from-db and the routing report of compare-models.

The output of trialkit report examples/compare-models/history/compare-classify --by model --format markdown, without its Source JSONL lines:

1 eval, 4 runs, from 2026-11-02 03:28 to 2026-11-02 03:29 UTC. Series by model.

Series prompt Runs Cases Case pass rate (95% CI) Trial pass rate (95% CI) Trend Error trials Mean trial Latest run (UTC)
inclusionai/ling-3.0-flash - 2 12 pass 100.0% (75.8–100.0%) 97.2% (85.8–99.5%) ██ 0 5.1 s 2026-11-02 03:29
z-ai/glm-5.3-flash - 2 11 pass, 1 fail 91.7% (64.6–98.5%) 94.4% (81.9–98.5%) ▇█ 0 2.7 s 2026-11-02 03:29

Trend: the trial pass rate of each completed run in the window, oldest first, on a fixed scale from ▁ 0% to █ 100%, so rows compare.

Series Tokens in Tokens out
inclusionai/ling-3.0-flash 54563 8185
z-ai/glm-5.3-flash 54673 8415

No flips since the previous completed run of each series.

The same command without --format markdown writes an HTML report. The site build made the one in this frame. Open it on its own page.