Primitives for testing LLM code. No extra dependencies. No platform.
trialkit adds what a test runner lacks for code that answers differently on each run: repeated trials with a pass rule, an error outcome for outages, scorers and LLM judges, and results as JSON Lines files. Use it in Vitest, Jest, node:test or pytest, with your own model client.
Install Pre-release
npm install --save-dev @trialkit/core @trialkit/vitest vitestPre-release: not published to npm or PyPI yet.
Example
import { describeEval } from '@trialkit/vitest';import { expect } from 'vitest';import { client } from './openrouter.ts';import { classify } from './src/classify.ts';
describeEval( 'classify', { task: (ticket: string, { signal }) => classify(client, 'z-ai/glm-5.3-flash', ticket, signal), trials: 3, verdict: 'majority', }, (it) => { it('double-charge', async ({ run }) => { expect(await run('I was charged twice.')).toMatchObject({ category: 'billing' }); }); },);Run with vitest run. The excerpt uses the replay client of examples/minimal, so it needs no API key:
✓ classify.eval.test.ts > classify > double-charge
❯ trials ↳ 2/2 trials passedWhy trialkit
- No extra dependenciesA trialkit package depends only on trialkit and, for an integration, the tool it plugs into: your test runner, model client, OpenTelemetry API or YAML/TOML parser. That tool is a peer (npm) or a wide version range (Python), so your installed version is used, never a second copy. See Packages.
- No platformNo account, hosted service or telemetry. Results are files in your repository; reporters for MLflow, OpenTelemetry and JUnit are optional packages.
- Primitives, not a frameworkTrials and verdicts,
errorapart fromfail, scorers, judges, HTTP replay and a pull-request gate that ignores random flips. Use one without the others; core also runs without a test runner.
Start here
- First eval without an API key: Vitest or pytest. The other examples cover replay, a pull-request gate, an HTTP service, model comparison, drafted cases, configs from a database and all three levels (support-triage).
- Guide: scoring, judges, trial policies, replay and CI.
- Leaving OpenAI Evals, promptfoo or vitest-evals: migration notes.