Judges for free text, with a calibration
Use a judge only when deterministic checks cannot decide whether output meets a free-text rule.
import { rubricJudge } from '@trialkit/judge';
export const summary = rubricJudge({ name: 'summary', model, rubric: 'The summary is faithful to the ticket and invents no fact.', output: (triage: { summary: string }) => triage.summary,});- Use it as any scorer:
await score(summary, triage)in a test, or in thescorersof a definition. A judge sees the case input and picked output. It sees expected only when it is a string or theexpectedpicker returns a string. - Strict reply parsing, a failed model call and an unreadable reply are
error;errorRetriesretries them. There is no task or judge response cache. Notemperatureis sent unless you set it. - Templates, rendered-prompt hashes and judge models are recorded. Do not edit a released template; use another model family than the model under test when possible.
The judge reference documents rubric, classifier and trajectory judges, model clients, templates and defaults.
Calibrate before you trust a judge
Section titled “Calibrate before you trust a judge”Use calibration before a judge affects an engineering decision.
import { calibrate } from '@trialkit/judge';import { expect } from 'vitest';import { summary } from './judges.ts';
const labeledOutputs = [ { evalCase: { id: 'double-charge', input: 'I was charged twice.' }, output: { summary: 'Charged twice.' }, label: true, },];
const report = await calibrate(summary, labeledOutputs);expect(report.errors).toEqual([]);expect(report.agreementInterval.low).toBeGreaterThanOrEqual(0.7);- Labels are human decisions: boolean for a rubric judge and a string for a classifier. They are not case
expectedvalues. - Read agreement with Cohen’s kappa. A judge that always passes can agree often and still have kappa 0.
- Read
agreementInterval, the 95% interval ofagreement, next to it: 6 of 6 agree gives 0.61 to 1, so a small set cannot show 100%. A gate on itslowbound fails when few cases ran, and when every case errored. report.judgeholds thetemplate,model,promptHash,settingsHashandpromptSourceof the judge that gave the verdicts (built-in, oroverridefor ametricJudgewith aprompt). Compare it withdata.judgeof the scores of your eval, to check that they came from the calibrated configuration.calibraterejects verdicts that come from more than one judge, also a built-in prompt mixed with an override.- Label real outputs with the judge hidden, include boundary cases, reserve a holdout set and calibrate again after a rubric, template or judge-model change.
Use recorded judge replies in CI. The judge reference explains calibration reports and classifier labels.
Optional metric recipes
Section titled “Optional metric recipes”Choose a built-in metric when code cannot decide entailment, relevance, observed completion or a conversational obligation. A metric fixes the prompt, and you give typed evidence. Keep each scorer separate; these are trialkit definitions, not DeepEval-compatible numbers.
import { metricJudge } from '@trialkit/judge';import { faithfulnessV2 } from '@trialkit/judge/metrics';
export const faithful = metricJudge(faithfulnessV2, { model });- Give the evidence as the second argument of
score(await score(faithful, { request, answer, sources })), or build it from the case and the output withevidencein a definition. Python:metric_judge(faithfulness_v2, model=model), withfaithfulness_v2fromtrialkit.judge.metrics. - The judge reference lists every metric with its evidence fields and rules. The template contract defines the prompts.
- Calibrate a metric on your own labeled data before it gates anything. The person-run example has the paid calibration commands for both languages; no command runs in CI.