Skip to content

trialkit

trialkit report turns recorded eval JSON Lines into HTML, Markdown, or JSON. trialkit compare compares a base history with a head history and can gate regressions. For outcome rules, see @trialkit/core outcomes.

Terminal window
npm install --save-dev trialkit @trialkit/core

Node.js 22.23.4 or later; ESM only. trialkit --version writes the version and exits 0.

Terminal window
trialkit report eval-history
trialkit compare eval-history eval-results --fail-on-regression
  • Record one JSON Lines file per run with jsonlFile, labels, and case tags.
  • The default series is one model and prompt pair. Tag cases that must not get worse with regression.
  • Give files or directories, not a shell glob. Directories are scanned recursively for .jsonl files.
import { runEval } from '@trialkit/core';
import { jsonlFile } from '@trialkit/core/node';
import { classifyEval } from './classify.eval';
await runEval(classifyEval, {
reporters: [jsonlFile({ dir: 'eval-results' })],
labels: { model: 'openai:gpt-5-mini', prompt: 'triage@3', commit: 'abc1234' },
config: { model: 'gpt-5-mini', prompt: 'triage@3' },
});

labels are string-valued. model and prompt are conventional names, not requirements. config is a plain JSON object recorded with its canonical hash; use it for settings that the run tested. See the config of a run.

Terminal window
trialkit report eval-history
trialkit report eval-history --by model --last 30
trialkit report eval-history --format markdown --out summary.md

A report groups runs by eval and series, shows pass-rate trends, score and consistency distributions, usage and cost totals, recorded metric means, changes, the latest run, and recorded spans and logs when available. HTML and Markdown show the same tables from one model: the latest completed run of each series, usage totals (one table per eval, without columns that are unknown for every series), metric means, flips and incomplete runs. HTML adds charts, a list of every run, and a page per run with its usage, cases and trials. A trial lists the judge of each score: its template, model, prompt hash and settings hash. The template is faithfulness@2 for the built-in prompt of a metric, and faithfulness@2 · override sha256:0123456789ab… when the judge has a prompt that replaced it (the hash is the prompt hash, cut to 12 hex digits). A shown unknown value is unknown, not zero. It warns and skips unusable lines or evidence; incomplete runs are shown but do not contribute to trends, distributions, or flips.

HTML previews recorded values at 300 characters and includes at most 4 KiB per value. Trace and log fragments use at most 256 KiB per trial; evidence shares a 10 MiB report budget. If the complete HTML would exceed 10 MiB, it is replaced by a notice to filter the history and consult the source JSONL. Both formats show at most 20 flips per series, and the reason of a flip cut at 300 characters. Markdown puts eval tables before details and evidence, shows at most 20 spans and 20 logs per trial, and is capped at 64 KiB. It omits table rows, shortens very long eval names, and omits source lines and evidence when they do not fit. HTML and Markdown validate selected-run logs and warn about malformed logs and sequence gaps; JSON reads the summary only. Capture is opt-in; redact before recording. A collapsed value is not redacted.

Option Default Meaning
--by <names> model,prompt Comma-separated labels, config, or config.<path> that define a series. --by= makes one series per eval.
--mark <labels> prompt when --by omits prompt; otherwise none Comma-separated labels whose changes inside a series are marked. --mark= marks none.
--eval <name> All evals Keep one eval.
--since <date> No limit Keep runs at or after an ISO 8601 date or date-time. A date is UTC; a zone-less date-time is local time.
--last <n> No limit Keep the latest n runs per series after --since; n is an integer at least 1.
--format <format> html html, markdown, or json.
--out <file> trialkit-report.html for HTML; stdout otherwise Output file, or - for stdout.
-h, --help — Print usage and exit 0.
--version — Print the version and exit 0.

config means the run’s configHash; config.<path> reads object fields only. A missing value displays as -. --by config. is invalid. Report exit codes are 0 when written, 1 for no matching runs or an error, and 2 for invalid arguments.

Terminal window
trialkit compare main-results pr-results
trialkit compare main-results pr-results --format markdown --out comparison.md
trialkit compare main-results pr-results --by= --fail-on-regression

Each side is a JSON Lines file or a directory of them. For every eval and --by series, compare uses the latest completed run on each side. It compares only shared cases with the same recorded input hash (or a missing hash); rates use cases that are pass or fail in both runs. It lists changed inputs, uncounted cases, and series that were not compared. A config change is reported but is not itself a comparability warning.

A case in both runs whose outcome differs is a regression (a gated case from pass to fail or error), errored (any other case from pass or fail to error; it fails the gate too), flipped (any other pass to fail), an improvement (fail or error to pass) or a change. A change in dataset hash, policy, scorer version, judge template, prompt hash, settings hash, prompt source or model warns; a change in config or labels only informs. The prompt source of a judge is built-in or override (@trialkit/judge: a metricJudge with a prompt), and a history that records none is built-in. It has its own line, judge <scorer> prompt source: built-in → override, so an override that has the text of the built-in prompt is still a change. Long or multiline config text displays as a short hash.

Option Default Meaning
--by <names> model Comma-separated labels, config, or config.<path> that pair runs. --by= pairs the latest completed run of each eval.
--base-config <match> All base runs Repeatable base selector: a sha256: hash prefix or <field>=<value>.
--head-config <match> All head runs Repeatable head selector, with the same syntax.
--eval <name> All evals Keep one eval.
--format <format> text text, markdown, or json.
--out <file> stdout Output file, or - for stdout.
--fail-on-regression Off Exit 1 when a series meets the regression rule below.
--gate-flips <tags> regression Repeatable, comma-separated case tags for the flip gate; all and none stand alone.
--no-gate-rate Off Disable the trial-pass-rate condition in the regression rule.
--max-cost <usd> No cost gate Finite non-negative USD limit for the latest completed head run; unknown cost fails it.
--allow-missing-base Off Treat a missing base path or a base with no run as the first run, with a stderr notice.
-h, --help — Print usage and exit 0.
--version — Print the version and exit 0.

A selector’s field value is JSON when valid JSON, otherwise a string. Repeat selectors require every match. A config selector removes whole config from pairing because the sides intentionally differ; config.<path> still pairs. Ambiguous hash prefixes and malformed selectors exit 2.

Exit code 0 means the comparison was written and, with --fail-on-regression, no regression occurred. Exit code 1 means a selected side has no completed run, nothing was comparable, a series cannot choose a run, a gate failed, or an operational error occurred. Exit code 2 means invalid arguments. The command writes output before code 1 for a comparison or cost gate; it writes no output when it cannot select a completed run. --allow-missing-base applies only to the base; a missing head still fails, and an enabled cost gate still runs.

A series fails the gate when any applicable condition is true:

  1. A case covered by --gate-flips changes from pass to fail or error; a tag on either side covers it.
  2. A covered base pass or fail has no head result or becomes skip, xfail or xpass. A case with a different recorded input hash is not compared or gated.
  3. A base pass or fail ends in head error, regardless of tags.
  4. A set check changes from pass to fail or error, or a base pass or fail check is absent, skipped or errored in head. Checks present on only one side are listed.
  5. Unless --no-gate-rate is set, the 95% interval of the overall trial-pass-rate change is wholly below 0.
  6. The head has stoppedBy, has fewer case results than its recorded dataset size, or has runs but none completed.
  7. With --max-cost, the latest completed head run exceeds the limit or has unknown cost.

--max-cost also checks a head-only series or an allowed missing base, and fails with code 1 even without --fail-on-regression. Other pass to fail changes are listed as flips but do not gate the command. A series absent on one side is notCompared; it does not fail by itself, but the command exits 1 with no compared series unless --allow-missing-base allowed a missing base. A stopped, incomplete, or partial head still fails when no case can be compared, including a head-only series.

The package root exports these functions:

Export Contract
readHistory(paths) Reads files without attachments and returns { runs, warnings }; warnings cover invalid lines and spans.
summarizeHistory(runs, options?) Builds the report model. SummaryOptions: by (['model', 'prompt']), mark (prompt when appropriate), evalName, since, and last.
readTrials(runs, onWarning?) Reads the task attachments, validated spans and logs of runs ({ runId, file }, such as the runs of a summary’s series). Calls onWarning for malformed logs and sequence gaps.
renderReport(summary, format?, trials?) Renders html (default) or markdown; trials add trace and log evidence.
compareRuns(base, head, options?) Builds a comparison. CompareOptions: by (['model']), gateFlips (['regression']), gateRate (true), and maxCost (off).
renderComparison(comparison, format?) Renders text (default) or markdown; serialize the returned models with JSON.stringify for JSON.

compareRuns validates gate options and rejects an ambiguous series. It does not select configs; filter the HistoryRun[] first. Comparison has by, resolved gate, evals, notCompared, and regressed. A compared eval exposes runs, differences, rates, tags, flips, checks, optional usage and metric changes, and regression. A notCompared series can carry a failed cost limit, headStoppedBy, or headPartial; regressed includes these and incomplete head series.

Exported types: RunPolicy, StoppedBy, RateEstimate, CaseRecord, CheckRecord, JudgeField, JudgeFields, TrialScore, TrialRecord, Trials, ScorerRecord, HistoryRun, History, SummaryOptions, HistorySummary, EvalSummary, SeriesSummary, SeriesRun, PolicyChange, LabelChange, HistogramBin, ScoreDistribution, Flips, Flip, SideState, CompareOptions, GateRule, RunRef, DifferenceKind, Difference, RateChange, PassRates, TagRates, FlipKind, CaseFlip, CheckFlip, Regression, EvalComparison, NotCompared, and Comparison.