Contributing to trialkit
For changes to trialkit itself. Read AGENTS.md for the commands, repository rules, API quality, recurring pitfalls and required documentation changes. Each area has an AGENTS.md of its own for its pitfalls; AGENTS.md lists them. Read docs/design.md for the design.
Layout
Section titled “Layout”| Path | Contents |
|---|---|
packages/ |
TypeScript packages, published as @trialkit/* and trialkit (pnpm workspace) |
python/ |
Python distributions (trialkit-*, imported as trialkit.*, and pytest-trialkit), including scorers in python/scorers/ and judges in python/judge/ (uv workspace) |
spec/ |
Language-neutral contracts: event JSON Schema, judge templates and shared fixtures |
fixtures/ |
Private workspaces that run the runner-adapter contract suite per runner version |
examples/ |
Example apps that use trialkit as an app would: focused workflows, support-triage, which combines them, and the model and config comparison examples |
scripts/ |
Build, package verification, smoke, contract, docs, changed-test and template generation scripts |
docs/ |
Design, guide, migration notes and the comparison data (comparison.json) |
site/ |
Astro/Starlight docs site; its pages are generated at build time from the repository Markdown and docs/comparison.json, with example HTML reports |
.github/ |
CI, secret scan, site, paid-eval and Dependabot workflows, the security policy and the pull-request template |
Development
Section titled “Development”Need Node.js 22.23.4 or later with Corepack (corepack enable pnpm) and uv. Bun and Deno are optional locally; CI requires them.
The commands for setup, the gate, formatting and the narrower checks are in AGENTS.md.
CI (.github/workflows/check.yml) runs the same gate, pnpm check, on Linux and Windows, at supported floors and current runtimes. Tool versions are pinned; change them deliberately and keep uv within each uv_build range.
Pull-request eval gate
Section titled “Pull-request eval gate”The Linux Node.js 26 job replays the example evals for the head and base commits, then runs trialkit compare --fail-on-regression. Preparation failures, no base run, no comparison, a regression, or a case that loses its verdict to error fail the gate. New head series are notCompared; at least one series must compare. The comment job publishes one updated PR comment.
Checks
Section titled “Checks”| Step | Command | What it checks |
|---|---|---|
| lint | pnpm lint |
Biome for TypeScript, JSON, Markdown and YAML; ruff for Python; includes formatting |
| typecheck | pnpm typecheck |
Workspace mappings, TypeScript packages/examples, and strict mypy for Python packages and pytest quickstart |
| build | pnpm build |
ESM output for each package and CommonJS where a tsconfig.cjs.json exists |
| test | pnpm test |
Package tests, including SDK and case parser floors; scripts, judge template assembly, examples, pytest (under xdist), shared spec scenarios and MLflow fixtures |
| coverage | pnpm coverage |
Vitest (@vitest/coverage-v8) and pytest (pytest-cov) reports, with no threshold. This step is outside pnpm check. |
| verify:packages | pnpm verify:packages |
Package policy, metadata, exports, README links, shared version, publint, attw, Python requirements, declarations, strict packed consumers, and the packed Studio files and bin |
| smoke | pnpm smoke |
Built runtime-neutral entries in Node, Bun and Deno; JSON Lines readback, suite, scorers, judges, replay and reporters, and the YAML and TOML file readers |
| parity | pnpm parity |
Fixed-seed TypeScript/Python parity for values, events, errors, datasets, readers and scripted runs |
| test:contract | pnpm test:contract |
Adapter contracts on Vitest 4.1.12/5.0.3/current 5.x, Jest 29/30 floors and current releases, and node:test |
| check:docs | pnpm check:docs |
Documentation names, links and headings in the Markdown that the site builds, the TypeScript blocks of the READMEs, guide, migration notes and design (compiled) and the Python blocks of the guide and the Python READMEs (mypy --strict), the README comparison block and generated report blocks, local paths in JSON data, and the size, names and CLAUDE.md imports of the AGENTS.md files |
| check:changed | pnpm check:changed |
Expected vitest run --changed selection for support-triage changes |
The root scripts run pnpm and uv workspaces directly; there is no task orchestrator. pnpm check:docs needs a build and uv sync.
On the lowest-runtime jobs, CI also runs pytest python/cases with ruamel.yaml 0.17.11 and pytest python/pytest with pytest 9.0.3, then the OpenTelemetry tests (python/core/tests/test_otel_sdk.py) and mypy with opentelemetry-sdk 1.45.1, these two even after an earlier failure unless the job is cancelled. The main gate tests the OpenTelemetry JavaScript API 1.0.0, the SDK 2.8.0 and 2.12.0, and Python SDK 1.45.2. CI fails, and does not skip, when Bun, Deno or a required Python test dependency is missing. The pytest plugin wheel test builds offline. Runner images (-latest) float, and the log of each job names its image. The comment job does not run for a pull request from a fork, whose token cannot write.
Test layers
Section titled “Test layers”Each layer checks a different boundary, from inferred types to an app that uses the published packages.
| Layer | What it checks | Where |
|---|---|---|
| Core and types | Core uses fake tasks and model clients. Type tests use defineEval without generics for core, scorers and judge; core also checks that createTestSuite infers run and score from task. Replay’s returned functions fit an SDK client’s fetch option. tsc checks these tests. |
core, scorers, judge, replay |
| Judges and models | Fake models include unreadable replies, and judge templates must match spec/judge-templates/. The model adapters run real openai clients of every supported major (4 to 7, from the first supported release) on a fake fetch, and ai 6 and 7 with mock language models, without network calls. |
template tests, OpenAI tests, AI SDK suite |
| Runner adapters | The contract fails if a set check passes or fails over a subset, runner timeouts remain on, an empty suite is not rejected at declaration, a budget-stopped run does not fail its suite, or cross-realm JSON is refused as non-plain. The test form also checks trial counts, fail locations and error text. Fake-runner unit tests cover what the contract cannot drive; pytester covers both Python forms. | contract definitions, expected results, pytest test form |
| Shared adapters and readers | createSuite tests validation, one execution per test, the random-order guard and close() failures for reporter errors and stoppedBy. readRuns tests damaged jsonlFile output and checks required fields against the schema. Core and CLI tests also pass cross-realm values through plain-object and error handling, labels and the CLI error path. Python mirrors the adapter and reader tests, including real-run cases. |
adapter, reader, Python adapter, Python reader |
| Reporters | Scripted evals check JSON Lines, JUnit XML and OpenTelemetry spans via an in-memory exporter; every JSON Lines event is checked against the schema. The MLflow reporter and uploader use a fake that enforces batch limits, key rules, duplicate or changed params and missing traces. Python also uses shared request fixtures, tests ported from TypeScript and an in-process HTTP server for its default transport. | MLflow fake, Python request fixtures |
| Packaging | Smoke tests load core and its /node, /util and /adapter entries, scorers, judge and /openai, replay and MLflow, and the YAML and TOML file readers (@trialkit/cases/yaml, /toml, @trialkit/judge/yaml) through import and require in Node.js, Bun and Deno. Packed tarballs install in scratch projects and compile against supported TypeScript versions and module resolutions. |
smoke, packed consumers |
| Examples | Minimal replays a test-form eval with exact checks and a rubric judge. Support-triage tests units, a component eval with deterministic scorers and a rubric judge, judge calibration against human labels and a black-box HTTP eval. CI uses a fake backend and replays the component recordings for the in-process service. The changed-test check asserts Vitest’s selection. | minimal, support-triage, changed-test check |
Property tests pin properties of trial verdicts and comparisons, not selected examples. For each verdict rule and every pass/fail sequence up to five trials, early stop gives the full-run verdict. Wilson intervals contain the observed rate and narrow as the sample grows, for every sample size up to 200. Removing error steps changes a case outcome only to or from error; an error trial never becomes a fail. fast-check and Hypothesis run policies of up to eight trials and three error retries with every kind of scripted step. Early stop preserves the outcome and stops at the deciding trial, errors within the retry limit change no count and no outcome, verdicts depend on the pass count, and more passes never lose a pass. CLI properties compare each history with itself through compareRuns and trialkit compare: no flips or differences, and no regression unless a run stopped early. Fixed seeds and run counts make a failure reproducible, and each property has been checked against a change that breaks its rule.
Secrets and CI rights
Section titled “Secrets and CI rights”- Start each workflow with
permissions: contents: read; give extra rights only to the job that needs them. Keeppersist-credentials: falseon non-pushing checkouts. - Pin every
uses:to a full commit SHA. Pass workflow inputs throughenv:, never directly intorun:. secrets.ymlscans all PR andmainhistory with gitleaks. Revoke a committed key; removing the commit is not enough..envand.env.*are ignored;.env.exampleis not.- Only the
evaljob inevals.ymlreadsOPENROUTER_API_KEYfrom theevalsenvironment.commentcan write PRs;publishcan write contents;deploycan write Pages and use OIDC. - Repository owners enable secret scanning, push protection, full-SHA Actions, approval for fork workflows, a PR rule on
main, and theevalsenvironment.