Skip to content

Primitives for testing LLM code. No extra dependencies. No platform.

trialkit adds what a test runner lacks for code that answers differently on each run: repeated trials with a pass rule, an error outcome for outages, scorers and LLM judges, and results as JSON Lines files. Use it in Vitest, Jest, node:test or pytest, with your own model client.

Read the guide See the examples

Install Pre-release

npm install --save-dev @trialkit/core @trialkit/vitest vitest

Pre-release: not published to npm or PyPI yet.

Example

classify.eval.test.ts
import { describeEval } from '@trialkit/vitest';
import { expect } from 'vitest';
import { client } from './openrouter.ts';
import { classify } from './src/classify.ts';
describeEval(
'classify',
{
task: (ticket: string, { signal }) => classify(client, 'z-ai/glm-5.3-flash', ticket, signal),
trials: 3,
verdict: 'majority',
},
(it) => {
it('double-charge', async ({ run }) => {
expect(await run('I was charged twice.')).toMatchObject({ category: 'billing' });
});
},
);

Run with vitest run. The excerpt uses the replay client of examples/minimal, so it needs no API key:

pnpm test
✓ classify.eval.test.ts > classify > double-charge
❯ trials
↳ 2/2 trials passed

Why trialkit

  1. No extra dependenciesA trialkit package depends only on trialkit and, for an integration, the tool it plugs into: your test runner, model client, OpenTelemetry API or YAML/TOML parser. That tool is a peer (npm) or a wide version range (Python), so your installed version is used, never a second copy. See Packages.
  2. No platformNo account, hosted service or telemetry. Results are files in your repository; reporters for MLflow, OpenTelemetry and JUnit are optional packages.
  3. Primitives, not a frameworkTrials and verdicts, error apart from fail, scorers, judges, HTTP replay and a pull-request gate that ignores random flips. Use one without the others; core also runs without a test runner.

Start here

  • First eval without an API key: Vitest or pytest. The other examples cover replay, a pull-request gate, an HTTP service, model comparison, drafted cases, configs from a database and all three levels (support-triage).
  • Guide: scoring, judges, trial policies, replay and CI.
  • Leaving OpenAI Evals, promptfoo or vitest-evals: migration notes.