trialkit
trialkit report turns recorded eval JSON Lines into HTML, Markdown, or JSON. trialkit compare compares a base history with a head history and can gate regressions. For outcome rules, see @trialkit/core outcomes.
Install
Section titled “Install”npm install --save-dev trialkit @trialkit/coreNode.js 22.23.4 or later; ESM only. trialkit --version writes the version and exits 0.
Example
Section titled “Example”trialkit report eval-historytrialkit compare eval-history eval-results --fail-on-regression- Record one JSON Lines file per run with
jsonlFile, labels, and case tags. - The default series is one
modelandpromptpair. Tag cases that must not get worse withregression. - Give files or directories, not a shell glob. Directories are scanned recursively for
.jsonlfiles.
Record runs
Section titled “Record runs”import { runEval } from '@trialkit/core';import { jsonlFile } from '@trialkit/core/node';import { classifyEval } from './classify.eval';
await runEval(classifyEval, { reporters: [jsonlFile({ dir: 'eval-results' })], labels: { model: 'openai:gpt-5-mini', prompt: 'triage@3', commit: 'abc1234' }, config: { model: 'gpt-5-mini', prompt: 'triage@3' },});labels are string-valued. model and prompt are conventional names, not requirements. config is a plain JSON object recorded with its canonical hash; use it for settings that the run tested. See the config of a run.
Report
Section titled “Report”trialkit report eval-historytrialkit report eval-history --by model --last 30trialkit report eval-history --format markdown --out summary.mdA report groups runs by eval and series, shows pass-rate trends, score and consistency distributions, usage and cost totals, recorded metric means, changes, the latest run, and recorded spans and logs when available. HTML and Markdown show the same tables from one model: the latest completed run of each series, usage totals (one table per eval, without columns that are unknown for every series), metric means, flips and incomplete runs. HTML adds charts, a list of every run, and a page per run with its usage, cases and trials. A trial lists the judge of each score: its template, model, prompt hash and settings hash. The template is faithfulness@2 for the built-in prompt of a metric, and faithfulness@2 · override sha256:0123456789ab… when the judge has a prompt that replaced it (the hash is the prompt hash, cut to 12 hex digits). A shown unknown value is unknown, not zero. It warns and skips unusable lines or evidence; incomplete runs are shown but do not contribute to trends, distributions, or flips.
HTML previews recorded values at 300 characters and includes at most 4 KiB per value. Trace and log fragments use at most 256 KiB per trial; evidence shares a 10 MiB report budget. If the complete HTML would exceed 10 MiB, it is replaced by a notice to filter the history and consult the source JSONL. Both formats show at most 20 flips per series, and the reason of a flip cut at 300 characters. Markdown puts eval tables before details and evidence, shows at most 20 spans and 20 logs per trial, and is capped at 64 KiB. It omits table rows, shortens very long eval names, and omits source lines and evidence when they do not fit. HTML and Markdown validate selected-run logs and warn about malformed logs and sequence gaps; JSON reads the summary only. Capture is opt-in; redact before recording. A collapsed value is not redacted.
| Option | Default | Meaning |
|---|---|---|
--by <names> |
model,prompt |
Comma-separated labels, config, or config.<path> that define a series. --by= makes one series per eval. |
--mark <labels> |
prompt when --by omits prompt; otherwise none |
Comma-separated labels whose changes inside a series are marked. --mark= marks none. |
--eval <name> |
All evals | Keep one eval. |
--since <date> |
No limit | Keep runs at or after an ISO 8601 date or date-time. A date is UTC; a zone-less date-time is local time. |
--last <n> |
No limit | Keep the latest n runs per series after --since; n is an integer at least 1. |
--format <format> |
html |
html, markdown, or json. |
--out <file> |
trialkit-report.html for HTML; stdout otherwise |
Output file, or - for stdout. |
-h, --help |
— | Print usage and exit 0. |
--version |
— | Print the version and exit 0. |
config means the run’s configHash; config.<path> reads object fields only. A missing value displays as -. --by config. is invalid. Report exit codes are 0 when written, 1 for no matching runs or an error, and 2 for invalid arguments.
Compare
Section titled “Compare”trialkit compare main-results pr-resultstrialkit compare main-results pr-results --format markdown --out comparison.mdtrialkit compare main-results pr-results --by= --fail-on-regressionEach side is a JSON Lines file or a directory of them. For every eval and --by series, compare uses the latest completed run on each side. It compares only shared cases with the same recorded input hash (or a missing hash); rates use cases that are pass or fail in both runs. It lists changed inputs, uncounted cases, and series that were not compared. A config change is reported but is not itself a comparability warning.
A case in both runs whose outcome differs is a regression (a gated case from pass to fail or error), errored (any other case from pass or fail to error; it fails the gate too), flipped (any other pass to fail), an improvement (fail or error to pass) or a change. A change in dataset hash, policy, scorer version, judge template, prompt hash, settings hash, prompt source or model warns; a change in config or labels only informs. The prompt source of a judge is built-in or override (@trialkit/judge: a metricJudge with a prompt), and a history that records none is built-in. It has its own line, judge <scorer> prompt source: built-in → override, so an override that has the text of the built-in prompt is still a change. Long or multiline config text displays as a short hash.
| Option | Default | Meaning |
|---|---|---|
--by <names> |
model |
Comma-separated labels, config, or config.<path> that pair runs. --by= pairs the latest completed run of each eval. |
--base-config <match> |
All base runs | Repeatable base selector: a sha256: hash prefix or <field>=<value>. |
--head-config <match> |
All head runs | Repeatable head selector, with the same syntax. |
--eval <name> |
All evals | Keep one eval. |
--format <format> |
text |
text, markdown, or json. |
--out <file> |
stdout | Output file, or - for stdout. |
--fail-on-regression |
Off | Exit 1 when a series meets the regression rule below. |
--gate-flips <tags> |
regression |
Repeatable, comma-separated case tags for the flip gate; all and none stand alone. |
--no-gate-rate |
Off | Disable the trial-pass-rate condition in the regression rule. |
--max-cost <usd> |
No cost gate | Finite non-negative USD limit for the latest completed head run; unknown cost fails it. |
--allow-missing-base |
Off | Treat a missing base path or a base with no run as the first run, with a stderr notice. |
-h, --help |
— | Print usage and exit 0. |
--version |
— | Print the version and exit 0. |
A selector’s field value is JSON when valid JSON, otherwise a string. Repeat selectors require every match. A config selector removes whole config from pairing because the sides intentionally differ; config.<path> still pairs. Ambiguous hash prefixes and malformed selectors exit 2.
Exit code 0 means the comparison was written and, with --fail-on-regression, no regression occurred. Exit code 1 means a selected side has no completed run, nothing was comparable, a series cannot choose a run, a gate failed, or an operational error occurred. Exit code 2 means invalid arguments. The command writes output before code 1 for a comparison or cost gate; it writes no output when it cannot select a completed run. --allow-missing-base applies only to the base; a missing head still fails, and an enabled cost gate still runs.
The regression rule
Section titled “The regression rule”A series fails the gate when any applicable condition is true:
- A case covered by
--gate-flipschanges frompasstofailorerror; a tag on either side covers it. - A covered base
passorfailhas no head result or becomesskip,xfailorxpass. A case with a different recorded input hash is not compared or gated. - A base
passorfailends in headerror, regardless of tags. - A set check changes from
passtofailorerror, or a basepassorfailcheck is absent, skipped or errored in head. Checks present on only one side are listed. - Unless
--no-gate-rateis set, the 95% interval of the overall trial-pass-rate change is wholly below 0. - The head has
stoppedBy, has fewer case results than its recorded dataset size, or has runs but none completed. - With
--max-cost, the latest completed head run exceeds the limit or has unknown cost.
--max-cost also checks a head-only series or an allowed missing base, and fails with code 1 even without --fail-on-regression. Other pass to fail changes are listed as flips but do not gate the command. A series absent on one side is notCompared; it does not fail by itself, but the command exits 1 with no compared series unless --allow-missing-base allowed a missing base. A stopped, incomplete, or partial head still fails when no case can be compared, including a head-only series.
Library API
Section titled “Library API”The package root exports these functions:
| Export | Contract |
|---|---|
readHistory(paths) |
Reads files without attachments and returns { runs, warnings }; warnings cover invalid lines and spans. |
summarizeHistory(runs, options?) |
Builds the report model. SummaryOptions: by (['model', 'prompt']), mark (prompt when appropriate), evalName, since, and last. |
readTrials(runs, onWarning?) |
Reads the task attachments, validated spans and logs of runs ({ runId, file }, such as the runs of a summary’s series). Calls onWarning for malformed logs and sequence gaps. |
renderReport(summary, format?, trials?) |
Renders html (default) or markdown; trials add trace and log evidence. |
compareRuns(base, head, options?) |
Builds a comparison. CompareOptions: by (['model']), gateFlips (['regression']), gateRate (true), and maxCost (off). |
renderComparison(comparison, format?) |
Renders text (default) or markdown; serialize the returned models with JSON.stringify for JSON. |
compareRuns validates gate options and rejects an ambiguous series. It does not select configs; filter the HistoryRun[] first. Comparison has by, resolved gate, evals, notCompared, and regressed. A compared eval exposes runs, differences, rates, tags, flips, checks, optional usage and metric changes, and regression. A notCompared series can carry a failed cost limit, headStoppedBy, or headPartial; regressed includes these and incomplete head series.
Exported types: RunPolicy, StoppedBy, RateEstimate, CaseRecord, CheckRecord, JudgeField, JudgeFields, TrialScore, TrialRecord, Trials, ScorerRecord, HistoryRun, History, SummaryOptions, HistorySummary, EvalSummary, SeriesSummary, SeriesRun, PolicyChange, LabelChange, HistogramBin, ScoreDistribution, Flips, Flip, SideState, CompareOptions, GateRule, RunRef, DifferenceKind, Difference, RateChange, PassRates, TagRates, FlipKind, CaseFlip, CheckFlip, Regression, EvalComparison, NotCompared, and Comparison.