Skip to content

Guide: unit tests, component evals and black-box evals

trialkit adds what a test runner lacks for LLM code: repeated trials with a verdict rule, an error outcome for outages, scorers, judges and JSON Lines results. It runs in Vitest, Jest, node:test or pytest, with your own model client and no platform. Versions and install commands: Requirements.

Terminal window
npm install --save-dev @trialkit/core @trialkit/vitest vitest # TypeScript
pip install pytest-trialkit # Python

Start with one production call and one fact:

import { describeEval } from '@trialkit/vitest';
import { expect } from 'vitest';
import { classify } from './classify.ts';
describeEval(
'classify',
{
task: (text: string, { signal }) => classify(text, { signal }),
trials: 3,
verdict: 'majority',
},
(it) => {
it('refund', async ({ run }) => {
expect(await run('How do I get a refund?')).toBe('billing');
});
},
);

The same eval in pytest:

from classify import classify
from trialkit.pytest import RunFixture, describe_eval
pytestmark = describe_eval("classify", task=classify, trials=3, verdict="majority")
def test_refund(run: RunFixture) -> None:
assert run("How do I get a refund?") == "billing"
  • Use deterministic scorers first. Add a calibrated judge only for a rule that code cannot decide.
  • Record real replies once; replay them in required CI.

examples/minimal is a complete small eval. examples/support-triage has all three levels. migrating.md covers a move from another tool, and design.md says why trialkit works this way.

Pick a section: