Migrating to trialkit
Move the eval into the test runner that already tests your app. The task calls the production path, every case can run repeated trials, and the run writes JSON Lines instead of platform data.
Start with the guide for levels, policies, errors, replay and reports. In Python, use trialkit-core with pytest-trialkit: a data row becomes a parametrized test, and an assertion stays an assert.
From OpenAI Evals
Section titled “From OpenAI Evals”Use this when you export an OpenAI Evals dataset and graders. The OpenAI Evals platform shuts down on 2026-11-30; export what you need first. trialkit does not import old runs, so its history starts with the first trialkit run.
| OpenAI Evals | trialkit |
|---|---|
| Eval, data source and criteria | A describeEval test, or defineEval plus evalSuite when a script also runs it |
| JSONL item | An it case or a row of it.for |
input_messages, model and sampling parameters |
The application function called by task |
string_check |
expect, exact or includes |
label_model / score_model |
classifierJudge or rubricJudge |
Dashboard and report_url |
JSON Lines, trialkit report and trialkit compare |
Before, the platform owns the request and run:
await openai.evals.runs.create(evalId, { data_source: { type: 'completions', model: 'gpt-5-mini', input_messages: template, source: { type: 'file_id', id: 'file-abc123' }, },});After, the test calls the application and scores its output (client is the OpenAI client of your app):
import { classifierJudge } from '@trialkit/judge';import { openaiModel } from '@trialkit/judge/openai';import { describeEval } from '@trialkit/vitest';import { expect } from 'vitest';import { triageTicket } from '../src/triage.ts';
const judge = openaiModel({ client, model: 'gpt-5-mini' });const summary = classifierJudge({ name: 'summary', model: judge, rubric: 'Is the summary faithful to the ticket?', choices: { faithful: 1, invents_facts: 0 }, output: (triage: { summary: string }) => triage.summary,});
describeEval( 'triage', { task: async (ticket: string, { signal }) => (await triageTicket(ticket, { client, model: 'gpt-5-mini', signal })).triage, trials: 3, verdict: 'majority', }, (it) => { it('double-charge', async ({ run, score }) => { const triage = await run('I was charged twice for my March invoice.'); expect(triage.category).toBe('billing'); await score(summary, triage); }); },);A trial ends in error for an outage or unreadable judge reply; a wrong answer, assertion, parser error or scorer failure is fail. See Errors and failures. Calibrate a replacement judge on human-labeled outputs and treat its first run as a new baseline.
From vitest-evals
Section titled “From vitest-evals”Use this when your eval already has describeEval, a harness and toSatisfyJudge.
| vitest-evals | trialkit |
|---|---|
harness / createHarness |
task, which calls the production function |
result.output |
await run(input) |
toSatisfyJudge or suite judges |
await score(judge, output) or definition scorers |
judgeHarness |
Each judge’s ModelClient |
toolReplay |
@trialkit/replay at the HTTP boundary |
Reporter, serve, GitHub Action |
JSON Lines, trialkit report and trialkit compare |
Before:
const result = await run(ticket);await expect(result).toSatisfyJudge(summaryJudge, { threshold: 0.8 });After:
const triage = await run(ticket);await score(summaryJudge, triage);A judge decides its own pass result, so there is no matcher threshold. Put model variance in the eval policy (trials, verdict, earlyStop, errorRetries), not in a runner retry. Record one real HTTP run and replay it in CI.
From promptfoo
Section titled “From promptfoo”Use this when a YAML config holds prompts, providers, tests and assertions. Move the prompt and request builder into the application first, then let the task call it.
| promptfoo | trialkit |
|---|---|
promptfooconfig.yaml |
A test file with describeEval; use a definition for script-driven runs |
prompts and providers |
The production function and its client |
tests / vars |
it or it.for cases; definition dataset.cases |
equals, contains, regex, JSON assertions |
expect, or exact, includes, regex, schema |
llm-rubric |
rubricJudge and await score |
--repeat |
trials plus a verdict |
promptfoo view |
trialkit report and trialkit compare |
Before:
prompts: - file://triage-prompt.txtproviders: - openai:gpt-5-minitests: - vars: ticket: I was charged twice for my March invoice. assert: - type: contains value: billing - type: llm-rubric value: The summary is faithful to the ticket.After:
import { rubricJudge } from '@trialkit/judge';import { openaiModel } from '@trialkit/judge/openai';import { describeEval } from '@trialkit/vitest';import { expect } from 'vitest';import { triageTicket } from '../src/triage.ts';
const judge = openaiModel({ client, model: 'gpt-5-mini' });const summary = rubricJudge({ name: 'summary', model: judge, rubric: 'The summary is faithful to the ticket.', output: (triage: { summary: string }) => triage.summary,});
describeEval( 'triage', { task: async (ticket: string, ctx) => (await triageTicket(ticket, { client, model: 'gpt-5-mini', signal: ctx.signal })).triage, trials: 3, verdict: 'majority', }, (it) => { it('double-charge', async ({ run, score }) => { const triage = await run('I was charged twice for my March invoice.'); expect(triage.category).toBe('billing'); await score(summary, triage); }); },);There are no weighted assertions or response cache. A trial passes only when all assertions and scores pass. Put partial-score thresholds in a scorer; use a set check for a rate gate. Each trial calls the model again; use a recording, not a cache, for cheap CI replays.
From a hand-written judge
Section titled “From a hand-written judge”Use this when a prompt, model call and JSON.parse decide a score. Replace the parser and retry loop with a versioned, strict judge.
| Hand-written judge | trialkit |
|---|---|
| Prompt string | rubric, plus a versioned template |
| Interpolated input/output | Narrow input and output pickers |
JSON.parse / boolean coercion |
Strict template reply parsing |
| Retry loop | errorRetries; SDK maxRetries for transport retries |
| Hand-counted tokens | Usage attached to the trial and token budget |
| Numeric scale or labels | classifierJudge({ choices, pass }) |
Before:
const reply = await client.chat.completions.create({ model: 'gpt-5-mini', messages });const verdict = JSON.parse(reply.choices[0]?.message.content ?? '{}');return Boolean(verdict.pass);After:
import { rubricJudge } from '@trialkit/judge';import { openaiModel } from '@trialkit/judge/openai';
const summaryJudge = rubricJudge({ name: 'summary', model: openaiModel({ client, model: 'gpt-5-mini' }), rubric: 'The summary is faithful to the ticket and invents nothing.', input: (ticket: { text: string }) => ticket.text, output: (triage: { summary: string }) => triage.summary,});Run it with await score(summaryJudge, output). A malformed reply or failed model call is error, never an invented verdict. Label real outputs, run calibrate, inspect agreement, Cohen’s kappa, errors and disagreements, then retain held-out cases. A changed rubric, template or judge model needs a new calibration and baseline.