Skip to content

5. Architecture

TypeScript and Python packages provide the same core contracts, with separate integrations for runners, model clients and backends. An application installs only the integrations that it uses, and files in spec/ keep the two implementations aligned.

Each integration covers one technology and all of its supported majors. Every package takes core as a peer (Python: a bounded requirement).

Package Role
@trialkit/core Engine and model port; /node, /util and /adapter subpaths
@trialkit/scorers Deterministic scorers
@trialkit/judge Judges, templates, calibration; optional peers: SDKs and cases
@trialkit/vitest Vitest adapter
@trialkit/jest Jest adapter
@trialkit/node-test node:test adapter
@trialkit/replay Record and replay HTTP
@trialkit/cases YAML and TOML case files
@trialkit/report-junit JUnit XML reporter
@trialkit/report-otel OpenTelemetry reporter
@trialkit/report-mlflow MLflow reporter and uploader
trialkit CLI: report and compare
@trialkit/studio The local page of §15
trialkit-core Python engine, readers and reporters
trialkit-scorers Python scorers
trialkit-cases Python case files
trialkit-judge Python judges and an OpenAI-compatible client
pytest-trialkit pytest plugin
trialkit-report-mlflow Python MLflow reporter

Not built: @trialkit/judge/anthropic (the AI SDK client reaches Anthropic); @trialkit/report-allure and @trialkit/report-datadog (JUnit and OpenTelemetry cover CI reports and Datadog traces); one trialkit-model-* or trialkit-report-* distribution for each other provider or backend.

Packages declare integrations without bringing a second copy of an app’s technology.

  • TypeScript packages have no runtime, optional or bundled dependencies: SDKs, test frameworks and other integrations are peers. Python has no peers: a requirement or an extra names a trialkit distribution or its integration, with bounds. Why: peer-only integrations avoid duplicate packages.
  • A peer range lists every tested major, with no exact pin and no open range such as >=4. It starts at the lowest version in its major with the APIs we use and no known security advisory.
  • An adapter imports no other adapter and re-exports no peer, except a core type that its API returns (scorers and judge export Scorer). Why: pnpm links core only into the package that lists it, so an app emits declarations for a returned core type only through that package.
  • The one import of an adapter by another is @trialkit/judge/yaml, which reads a template file or a prompt file through readYamlMapping of @trialkit/cases, an optional peer of judge (Python: the extra trialkit-judge[yaml]). An app that reads neither loads neither. Why: the YAML rules stay in one reader that the case files, the template files and the prompt files share, and core cannot hold the parser, because core has no peers.
  • Packages ship no native modules, binaries, install scripts, HTTP server, bundler or TypeScript loader. The one exception is @trialkit/studio, an opt-in development tool that no other package imports.

ESM is built for every package and CommonJS where tsconfig.cjs.json exists; @trialkit/vitest and the CLI are ESM only. Why: Jest has only experimental ESM support, so a CommonJS Jest project needs CommonJS dependencies. scripts/install-test.mjs checks each TypeScript resolution mode. Both builds can load in one process, so classification uses value signals and run state belongs to a run.

Runtime-neutral packages (core, scorers, judge, replay, report-mlflow, report-otel) use only ECMAScript and shared web APIs (Node.js, Bun, Deno, browsers, edge workers); Node.js code lives in src/node/ subpaths. No package reads an environment variable, patches a global or registers a tracer provider: the app passes settings and supplies its own provider.

The Node.js floor (requirements) is the same in every package, in the root manifest and on the lowest CI leg. It does not come from a library need: the docs site’s Astro 7 requires it, and CI runs the whole workspace on one lowest Node.js. Vitest 5, the strictest runner peer, accepts 22.12, and a lower library floor is untested (§14). The package verifier checks that engines.node exists, not its value.

The npm packages and Python distributions release together under one version, from 0.1.0. The core peer range accepts 0.1.x only, so a new core minor needs a release of every package. verify-packages checks the metadata.

CONTRIBUTING.md lists the steps of the gate, and the workflow holds versions and limits. Tools have exact versions, so a release cannot change a run that nobody touched, and CI fails, not skips, when a tool is missing. Packed consumers install the adapters with strict peer checks, because --legacy-peer-deps proves declarations only. No test, gate script or unattended job calls a paid API.

Both languages implement the contracts in spec/ instead of sharing code: change a contract in spec/ first, then both implementations in the same commit. The parity tool sends generated inputs to the built TypeScript core and a Python oracle, and a difference fails with the seed and the smallest case. Why: a behavior that only one core has is a bug, and fixtures cannot list every case.