Skip to content

Judges for free text, with a calibration

Use a judge only when deterministic checks cannot decide whether output meets a free-text rule.

import { rubricJudge } from '@trialkit/judge';
export const summary = rubricJudge({
name: 'summary',
model,
rubric: 'The summary is faithful to the ticket and invents no fact.',
output: (triage: { summary: string }) => triage.summary,
});
  • Use it as any scorer: await score(summary, triage) in a test, or in the scorers of a definition. A judge sees the case input and picked output. It sees expected only when it is a string or the expected picker returns a string.
  • Strict reply parsing, a failed model call and an unreadable reply are error; errorRetries retries them. There is no task or judge response cache. No temperature is sent unless you set it.
  • Templates, rendered-prompt hashes and judge models are recorded. Do not edit a released template; use another model family than the model under test when possible.

The judge reference documents rubric, classifier and trajectory judges, model clients, templates and defaults.

Use calibration before a judge affects an engineering decision.

import { calibrate } from '@trialkit/judge';
import { expect } from 'vitest';
import { summary } from './judges.ts';
const labeledOutputs = [
{
evalCase: { id: 'double-charge', input: 'I was charged twice.' },
output: { summary: 'Charged twice.' },
label: true,
},
];
const report = await calibrate(summary, labeledOutputs);
expect(report.errors).toEqual([]);
expect(report.agreementInterval.low).toBeGreaterThanOrEqual(0.7);
  • Labels are human decisions: boolean for a rubric judge and a string for a classifier. They are not case expected values.
  • Read agreement with Cohen’s kappa. A judge that always passes can agree often and still have kappa 0.
  • Read agreementInterval, the 95% interval of agreement, next to it: 6 of 6 agree gives 0.61 to 1, so a small set cannot show 100%. A gate on its low bound fails when few cases ran, and when every case errored.
  • report.judge holds the template, model, promptHash, settingsHash and promptSource of the judge that gave the verdicts (built-in, or override for a metricJudge with a prompt). Compare it with data.judge of the scores of your eval, to check that they came from the calibrated configuration. calibrate rejects verdicts that come from more than one judge, also a built-in prompt mixed with an override.
  • Label real outputs with the judge hidden, include boundary cases, reserve a holdout set and calibrate again after a rubric, template or judge-model change.

Use recorded judge replies in CI. The judge reference explains calibration reports and classifier labels.

Choose a built-in metric when code cannot decide entailment, relevance, observed completion or a conversational obligation. A metric fixes the prompt, and you give typed evidence. Keep each scorer separate; these are trialkit definitions, not DeepEval-compatible numbers.

import { metricJudge } from '@trialkit/judge';
import { faithfulnessV2 } from '@trialkit/judge/metrics';
export const faithful = metricJudge(faithfulnessV2, { model });
  • Give the evidence as the second argument of score (await score(faithful, { request, answer, sources })), or build it from the case and the output with evidence in a definition. Python: metric_judge(faithfulness_v2, model=model), with faithfulness_v2 from trialkit.judge.metrics.
  • The judge reference lists every metric with its evidence fields and rules. The template contract defines the prompts.
  • Calibrate a metric on your own labeled data before it gates anything. The person-run example has the paid calibration commands for both languages; no command runs in CI.