Skip to content

Migrating to trialkit

Move the eval into the test runner that already tests your app. The task calls the production path, every case can run repeated trials, and the run writes JSON Lines instead of platform data.

Start with the guide for levels, policies, errors, replay and reports. In Python, use trialkit-core with pytest-trialkit: a data row becomes a parametrized test, and an assertion stays an assert.

Use this when you export an OpenAI Evals dataset and graders. The OpenAI Evals platform shuts down on 2026-11-30; export what you need first. trialkit does not import old runs, so its history starts with the first trialkit run.

OpenAI Evals trialkit
Eval, data source and criteria A describeEval test, or defineEval plus evalSuite when a script also runs it
JSONL item An it case or a row of it.for
input_messages, model and sampling parameters The application function called by task
string_check expect, exact or includes
label_model / score_model classifierJudge or rubricJudge
Dashboard and report_url JSON Lines, trialkit report and trialkit compare

Before, the platform owns the request and run:

await openai.evals.runs.create(evalId, {
data_source: {
type: 'completions',
model: 'gpt-5-mini',
input_messages: template,
source: { type: 'file_id', id: 'file-abc123' },
},
});

After, the test calls the application and scores its output (client is the OpenAI client of your app):

import { classifierJudge } from '@trialkit/judge';
import { openaiModel } from '@trialkit/judge/openai';
import { describeEval } from '@trialkit/vitest';
import { expect } from 'vitest';
import { triageTicket } from '../src/triage.ts';
const judge = openaiModel({ client, model: 'gpt-5-mini' });
const summary = classifierJudge({
name: 'summary',
model: judge,
rubric: 'Is the summary faithful to the ticket?',
choices: { faithful: 1, invents_facts: 0 },
output: (triage: { summary: string }) => triage.summary,
});
describeEval(
'triage',
{
task: async (ticket: string, { signal }) =>
(await triageTicket(ticket, { client, model: 'gpt-5-mini', signal })).triage,
trials: 3,
verdict: 'majority',
},
(it) => {
it('double-charge', async ({ run, score }) => {
const triage = await run('I was charged twice for my March invoice.');
expect(triage.category).toBe('billing');
await score(summary, triage);
});
},
);

A trial ends in error for an outage or unreadable judge reply; a wrong answer, assertion, parser error or scorer failure is fail. See Errors and failures. Calibrate a replacement judge on human-labeled outputs and treat its first run as a new baseline.

Use this when your eval already has describeEval, a harness and toSatisfyJudge.

vitest-evals trialkit
harness / createHarness task, which calls the production function
result.output await run(input)
toSatisfyJudge or suite judges await score(judge, output) or definition scorers
judgeHarness Each judge’s ModelClient
toolReplay @trialkit/replay at the HTTP boundary
Reporter, serve, GitHub Action JSON Lines, trialkit report and trialkit compare

Before:

const result = await run(ticket);
await expect(result).toSatisfyJudge(summaryJudge, { threshold: 0.8 });

After:

const triage = await run(ticket);
await score(summaryJudge, triage);

A judge decides its own pass result, so there is no matcher threshold. Put model variance in the eval policy (trials, verdict, earlyStop, errorRetries), not in a runner retry. Record one real HTTP run and replay it in CI.

Use this when a YAML config holds prompts, providers, tests and assertions. Move the prompt and request builder into the application first, then let the task call it.

promptfoo trialkit
promptfooconfig.yaml A test file with describeEval; use a definition for script-driven runs
prompts and providers The production function and its client
tests / vars it or it.for cases; definition dataset.cases
equals, contains, regex, JSON assertions expect, or exact, includes, regex, schema
llm-rubric rubricJudge and await score
--repeat trials plus a verdict
promptfoo view trialkit report and trialkit compare

Before:

prompts:
- file://triage-prompt.txt
providers:
- openai:gpt-5-mini
tests:
- vars:
ticket: I was charged twice for my March invoice.
assert:
- type: contains
value: billing
- type: llm-rubric
value: The summary is faithful to the ticket.

After:

import { rubricJudge } from '@trialkit/judge';
import { openaiModel } from '@trialkit/judge/openai';
import { describeEval } from '@trialkit/vitest';
import { expect } from 'vitest';
import { triageTicket } from '../src/triage.ts';
const judge = openaiModel({ client, model: 'gpt-5-mini' });
const summary = rubricJudge({
name: 'summary',
model: judge,
rubric: 'The summary is faithful to the ticket.',
output: (triage: { summary: string }) => triage.summary,
});
describeEval(
'triage',
{
task: async (ticket: string, ctx) =>
(await triageTicket(ticket, { client, model: 'gpt-5-mini', signal: ctx.signal })).triage,
trials: 3,
verdict: 'majority',
},
(it) => {
it('double-charge', async ({ run, score }) => {
const triage = await run('I was charged twice for my March invoice.');
expect(triage.category).toBe('billing');
await score(summary, triage);
});
},
);

There are no weighted assertions or response cache. A trial passes only when all assertions and scores pass. Put partial-score thresholds in a scorer; use a set check for a rate gate. Each trial calls the model again; use a recording, not a cache, for cheap CI replays.

Use this when a prompt, model call and JSON.parse decide a score. Replace the parser and retry loop with a versioned, strict judge.

Hand-written judge trialkit
Prompt string rubric, plus a versioned template
Interpolated input/output Narrow input and output pickers
JSON.parse / boolean coercion Strict template reply parsing
Retry loop errorRetries; SDK maxRetries for transport retries
Hand-counted tokens Usage attached to the trial and token budget
Numeric scale or labels classifierJudge({ choices, pass })

Before:

const reply = await client.chat.completions.create({ model: 'gpt-5-mini', messages });
const verdict = JSON.parse(reply.choices[0]?.message.content ?? '{}');
return Boolean(verdict.pass);

After:

import { rubricJudge } from '@trialkit/judge';
import { openaiModel } from '@trialkit/judge/openai';
const summaryJudge = rubricJudge({
name: 'summary',
model: openaiModel({ client, model: 'gpt-5-mini' }),
rubric: 'The summary is faithful to the ticket and invents nothing.',
input: (ticket: { text: string }) => ticket.text,
output: (triage: { summary: string }) => triage.summary,
});

Run it with await score(summaryJudge, output). A malformed reply or failed model call is error, never an invented verdict. Label real outputs, run calibrate, inspect agreement, Cohen’s kappa, errors and disagreements, then retain held-out cases. A changed rubric, template or judge model needs a new calibration and baseline.