Guide: unit tests, component evals and black-box evals
trialkit adds what a test runner lacks for LLM code: repeated trials with a verdict rule, an error outcome for outages, scorers, judges and JSON Lines results. It runs in Vitest, Jest, node:test or pytest, with your own model client and no platform. Versions and install commands: Requirements.
npm install --save-dev @trialkit/core @trialkit/vitest vitest # TypeScriptpip install pytest-trialkit # PythonStart with one production call and one fact:
import { describeEval } from '@trialkit/vitest';import { expect } from 'vitest';import { classify } from './classify.ts';
describeEval( 'classify', { task: (text: string, { signal }) => classify(text, { signal }), trials: 3, verdict: 'majority', }, (it) => { it('refund', async ({ run }) => { expect(await run('How do I get a refund?')).toBe('billing'); }); },);The same eval in pytest:
from classify import classifyfrom trialkit.pytest import RunFixture, describe_eval
pytestmark = describe_eval("classify", task=classify, trials=3, verdict="majority")
def test_refund(run: RunFixture) -> None: assert run("How do I get a refund?") == "billing"- Use deterministic scorers first. Add a calibrated judge only for a rule that code cannot decide.
- Record real replies once; replay them in required CI.
examples/minimal is a complete small eval. examples/support-triage has all three levels. migrating.md covers a move from another tool, and design.md says why trialkit works this way.
Pick a section:
- Choose a test: the levels, Level 1, Level 2, Level 3.
- Load cases: from a file, from a database or API, drafted by a model.
- Score output: scorers, judges, agent sessions.
- Handle noise and outages: errors, trial policies, time limits, set checks.
- Run in CI: CI layout, record and replay, history and reports.
- Use pytest: Python.
Sections
Section titled “Sections”- The levels
- Choose the level
- Level 1: a unit test
- Level 2: a component eval
- Agent sessions
- Level 3: a black-box eval
- Scorers: deterministic first
- Judges for free text, with a calibration
- Errors and failures
- Trial policies
- Time limits
- Set checks and thresholds
- CI layout
- Record and replay model replies
- History and reports
- Python