Skip to content

@trialkit/judge

Model-graded scorers for rubrics, fixed labels, and recorded agent trajectories. Each score records the judge model, template, and system-prompt hash so comparisons can identify a changed judge.

Terminal window
npm install --save-dev @trialkit/judge @trialkit/core

@trialkit/core is a peer. The matching adapter uses optional peers openai ^4.55.0 || ^5.0.0 || ^6.0.0 || ^7.0.0 or ai ^6.0.214 || ^7.0.4. No other dependencies: your installed SDK is used. Tests cover the floor of each supported major. Node.js 22.23.4+. The package runs where crypto.subtle is available. readTemplate and readPrompt of @trialkit/judge/yaml also need the optional peer @trialkit/cases (and its parser yaml): see Template files and Replace the prompt.

import { fromFunction } from '@trialkit/core';
import { rubricJudge } from '@trialkit/judge';
const model = fromFunction('fake:judge', async () => ({
text: '{"reason":"Names the 30 day limit.","pass":true}',
model: 'fake-1',
}));
const refundPolicy = rubricJudge({
model,
rubric: 'Pass only when the answer says a refund is available within 30 days.',
});
  • metricJudge grades one built-in metric, such as faithfulness, from typed evidence; rubricJudge gives a pass/fail score for your own rubric; classifierJudge selects a label and score; trajectoryJudge grades recorded spans.
  • Each trial calls the model; there is no cache. Reply usage is attributed to scorers.
  • An unreadable reply or failed model call is an error, not a quality failure. See core outcomes and errors.
Entry Exports
@trialkit/judge rubricJudge, classifierJudge, trajectoryJudge, metricJudge, calibrate, loadTemplate, retrievalTurns, toolDecisions, rubricV1, rubricV2, classifierV1, classifierV2, trajectoryV1; types Conversation, ConversationTurn, RetrievalTurn, ReferencedRetrievalTurn, References, ToolDecision, Scorer, Span, Interval, JudgeTemplate, TemplateOutput, Metric, PromptText, Tool, JudgeOptions, ClassifierJudgeOptions, TrajectoryJudgeOptions, MetricJudgeOptions, CalibrateOptions, CalibrationReport, CalibrationResult, CalibrationError, JudgeIdentity, Disagreement, LabeledOutput.
@trialkit/judge/metrics The metric constants, the first argument of metricJudge: faithfulnessV2, citationGroundingV2, contextAlignmentV2, contextRelevanceV2, contextRankingV2, referenceCoverageV2, contextSufficiencyV2, answerRelevanceV2, summaryQualityV2, instructionFollowingV2, taskCompletionV2, toolAppropriatenessV2, stepEfficiencyV2, planQualityV2, planAdherenceV2, knowledgeRetentionV2, roleAdherenceV2, topicAdherenceV2, turnRelevanceV2, completenessV2 and goalAccuracyV2. They have their own entry so that the root stays small as the set grows, and an app that uses only rubricJudge does not bundle them.
@trialkit/judge/openai openaiModel plus OpenAIModelOptions, OpenAIChatClient, OpenAIChatBody, OpenAIChatReply, OpenAIRequestOptions, and MaxOutputTokensParam.
@trialkit/judge/ai-sdk aiSdkModel, AiSdkModelOptions, and AiSdkLanguageModel.
@trialkit/judge/generate generateCases plus GenerateCasesOptions, GeneratedCases, CaseSource, SourceDocument, and the type exports EvalCase and TokenUsage.
@trialkit/judge/yaml readTemplate, readPrompt.

rubricJudge(options) passes only when the rubric is met. classifierJudge(options) asks for one label in choices; its score is that label’s value. Both return a core Scorer with version <template id>@<template version>.

Option Default Meaning
model Required A core ModelClient.
rubric Required Non-empty criteria; it is part of the system prompt.
name rubric or classifier Scorer name; scorer names within one eval must differ.
template rubricV1/classifierV1; V2 with context A JudgeTemplate with the matching output kind. A metric template is refused: use metricJudge.
input Case input; non-string becomes canonical JSON (input) => string | undefined; what the judge sees.
output Task output; non-string becomes canonical JSON (output) => string | undefined; what the judge grades.
context None (evalCase, output) => string | undefined; background in a separate <context> tag. Its presence must exactly match a {{context}} template placeholder.
expected String evalCase.expected, otherwise (none) A fixed string or (expected) => string | undefined; non-string case values are not sent unless selected.
temperature Omitted Finite number from 0 through 2. Omitted means no request field.
choices Required for classifierJudge Non-empty { label: score }; each score is 0 through 1. Labels keep insertion order; unsigned integer-like labels such as "1" are refused.
pass Label score is at least 0.5 (label) => boolean, asked once for every label when the classifier is made, after every other check.

Options are checked when the judge is made. A picker that returns a non-string is misuse and ends the trial in error; a thrown picker is a quality fail. A throwing or non-boolean pass rule is rejected at creation. A label that core cannot write as JSON is refused too. Without a picker, null, undefined and values that core cannot write as canonical JSON are misuse: add an explicit input or output picker. The score has name, version, value, pass, reason, and data.judge (model, provider replyModel, template, promptHash, promptSource, temperature, evidence, settingsHash); classifiers also record the choices scores, passByLabel (whether each label passes), and data.label. promptSource is built-in for these judges: only metricJudge takes a prompt that replaces the text of its template. Unset temperature is recorded as null. evidence is canonical-json: how a non-string input or output without a picker becomes text.

The judge accepts a reply object in reply.json, or reply.text containing exactly one JSON object, optionally inside one json fence. A rubric reply must be exactly { reason: string, pass: boolean }; a classifier reply must be exactly { reason: string, label: one of choices }. Other text, extra fields, wrong types, or an unknown label produce an infrastructure error. Every rejected model.generate call is wrapped as an infrastructure error, including a plain custom-client error.

trajectoryJudge(options) uses model, rubric, name (default trajectory), template (default trajectoryV1), input, expected, and temperature; it has no output or context picker. Its options type is TrajectoryJudgeOptions<E, I>: the expected type first, then the input type. input has the same default and picker rules as the other judges. A template must have verdict output and no context placeholder. It requires recorded spans and grades canonical JSON in record order. The transcript retains span ids, parent ids, status, and the supported GenAI operation, model, message, and tool-call attributes only. Missing or unrenderable spans end the trial in error before a model call. Captured span content goes to the judge model, so redact secrets before ctx.span.

metricJudge(metric, options) grades one built-in property, for example whether every claim of an answer follows from its sources. The metric fixes the evidence fields, how the prompt shows them and the reply, so you write no prompt and no input, output or context picker. It returns a core Scorer with version <template id>@<template version>.

import { defineEval, type EvalCase } from '@trialkit/core';
import { describeEval } from '@trialkit/vitest';
import { metricJudge } from '@trialkit/judge';
import { contextRankingV2, faithfulnessV2 } from '@trialkit/judge/metrics';
interface Reply {
text: string;
passages: string[];
}
type Case = EvalCase<{ question: string }, { referenceAnswer: string }>;
// Test form: the second argument of `score` is the evidence, built from the result of `run`.
describeEval('support', { task: ask }, (it) => {
it('phone support', async ({ run, score }) => {
const question = 'Which plan includes phone support?';
const reply = await run({ question });
await score(metricJudge(faithfulnessV2, { model }), {
request: question,
answer: reply.text,
sources: reply.passages,
});
});
});
// Definition form: `evidence` builds the evidence from the case and the output of the task.
export default defineEval({
name: 'support',
dataset: {
cases: [
{
id: 'phone-support',
input: { question: 'Which plan includes phone support?' },
expected: { referenceAnswer: 'The Plus plan includes phone support.' },
},
],
},
task: ask,
scorers: [
metricJudge(contextRankingV2, {
model,
evidence: ({ input, expected }: Case, output: Reply) => ({
request: input.question,
sources: output.passages,
reference: expected?.referenceAnswer,
}),
}),
],
});

Write no generic argument: TypeScript reads the evidence type from the metric. Annotate the parameters of evidence, as above. In defineEval, a picker without annotations gets unknown parameters, because defineEval types its scorers before it reads the task. A definition that has its own type (const definition: EvalDefinition<Question, Reply, Reference> = { ... }) needs no annotation on the picker. Without evidence, the output is the evidence: the argument of score in a test, or the output of the task in a definition. score takes the output type of its scorer, so the argument in a test can differ from the output of the task.

Option Default Meaning
model Required A core ModelClient.
evidence The output is the evidence (evalCase, output) => evidence: an object with the fields that the metric reads. A case without expected is misuse when the metric needs a reference, and the function is not called for it.
rubric None; the prompt shows (none) Application criteria in the system prompt: more constraints, not a replacement for the metric. A blank rubric is refused. A metric that requires one (instructionFollowingV2, roleAdherenceV2, topicAdherenceV2) refuses a judge made without it.
name The template id Scorer name; scorer names within one eval must differ.
temperature Omitted Finite number from 0 through 2. Omitted means no request field.
pass Label score is at least 0.5 For a label metric only (contextSufficiencyV2): (label) => boolean, asked once for every label when the judge is made, after every other check. A verdict metric refuses it.
prompt The text of the template { system, user }: your text replaces the system and user text of the metric, and the rest stays the metric’s. A prompt that breaks the placeholder rule is refused when the judge is made. See Replace the prompt.
Field Valid value The prompt shows
request A nonblank string The text: what the user asked.
answer A string; an empty one is valid The text. An empty answer is evidence that the metric fails.
sources An array of strings; [] is valid Canonical JSON [{"id":1,"text":"..."}] in order. Blank and duplicate entries keep their own IDs.
reference A nonblank string The text: the factual reference answer, not a human label.
tools An array of tools: each has a nonblank string name, a string description and an object inputSchema (JSON); [] is valid Canonical JSON [{"description":"...","inputSchema":{...},"name":"..."}] in order. No other key of a tool is sent.
successCriteria A nonempty array of nonblank strings A numbered list, one line for each criterion: 1. first.
plan A string; an empty one is valid The text: the public plan. An empty plan is evidence that the metric fails.
planSpanId A nonblank string The text: the ID of the span that holds the public plan. The judge does not look for the span; the model finds that it is missing.
conversation A conversation: { turns }, as Conversation metrics defines it Canonical JSON of the turns: each has its number, id, role, content and the calls and results that it has. No menu, snapshot or span ID.
target Optional: the ID of an assistant turn of conversation. undefined, null and absent mean none The ID as a JSON string, or null. The conversation ends at the target turn, and only that reply is graded.

The judge checks its options when it is made, and the evidence before the model call, field by field in the order of the template. A bad field ends the trial in error, not in fail, with a message that names the metric, the field, the case and the value: metricJudge: faithfulness@2: evidence.sources must be an array of strings for case phone, got "Pro.". These are misuse: evidence that is not an object, a field that is absent, undefined, null or of another type (but target, which is optional), a blank request, reference or planSpanId, a sources item that is not a string or has a lone surrogate, an empty successCriteria or a criterion that is blank, a tool without a nonblank name, a string description or a JSON object inputSchema (evidence.tools[1].inputSchema must be a JSON object), a conversation that is not valid, a target that is not the ID of an assistant turn of the conversation, and a case without expected when evidence is set and the metric needs a reference. Other keys of the evidence, and other keys of a tool, are never sent. The judge does not trim, deduplicate or limit the evidence: refuse oversized evidence before the call. A picker that throws is a quality fail, as for the other judges. A reply that is not exactly { reason, pass } or { reason, label }, and a failed model call, are infrastructure errors.

A score has the fields of the other judges, and data.judge.evidence is fields@1: the name of the rules that turn the evidence into text. A change of the rules shows in settingsHash. A label metric also records choices, passByLabel and data.label. data.judge.promptSource is built-in, or override when the judge has a prompt, which also records userTemplateHash.

Constant Evidence Judges
faithfulnessV2 request, sources, answer Every factual claim of the answer follows from the sources. An explicit abstention can pass; an empty answer fails.
citationGroundingV2 request, sources, answer Every factual claim and every attached [N] citation is supported by the sources.
contextAlignmentV2 request, sources, answer The answer does not contradict the sources, which are curated ideal contexts. Unsupported claims can pass.
summaryQualityV2 request, sources, answer Every claim of the summary follows from the source text and every material point in scope is covered. A blank summary fails.
contextRelevanceV2 request, sources Every substantive statement of the sources is relevant to the request or necessary background.
contextRankingV2 request, reference, sources At least one source is relevant, and every relevant source comes before every irrelevant one.
referenceCoverageV2 request, reference, sources The sources together support every fact of the reference without an unresolved contradiction.
contextSufficiencyV2 request, sources One label: sufficient (score 1), insufficient (0) or conflicting (0). A material conflict takes precedence.
answerRelevanceV2 request, answer The answer addresses the request and every material subquestion. Truth is not assessed. An empty answer fails.
instructionFollowingV2 request, answer The answer meets every applicable numbered instruction of the rubric, which is required.
taskCompletionV2 request, successCriteria, spans Every success criterion has observed evidence in the final message or the tool results. A success claim alone proves no external effect.
toolAppropriatenessV2 request, tools, spans The decisions, calls, arguments and results fit the tools and the task. Justified execution without tools can pass.
stepEfficiencyV2 request, tools, spans Every step is justified, without avoidable repetition. Justified retries can pass. Completion is separate.
planQualityV2 request, tools, plan The plan is actionable, complete, ordered and possible with the tools. A blank plan fails; a one-step plan can pass.
planAdherenceV2 request, tools, planSpanId, spans The run follows the plan in the span that planSpanId designates. Every applicable commitment has proof.
knowledgeRetentionV2 conversation, target Every graded reply respects the relevant user facts and commitments of its prefix. A contradiction and a needless re-ask fail; a necessary clarification can pass.
roleAdherenceV2 conversation, target; rubric Every graded reply meets the trusted role obligations of the rubric, despite requests in the conversation. An in-role refusal can pass; role drift fails.
topicAdherenceV2 conversation, target; rubric Every graded reply answers a permitted-topic request and declines an off-topic one, under the trusted topic rules of the rubric. Truth is separate.
turnRelevanceV2 conversation, target Every graded reply addresses the current user intent of its prefix. A needed clarification and a user-led topic change can pass. Truth is separate.
completenessV2 conversation Every uncancelled user request has a substantive conversational resolution by the end. A promise without delivery fails; a justified refusal can close a request.
goalAccuracyV2 conversation Every active user goal has visible proof of achievement by the end. A success claim alone proves no external action, and a justified refusal does not achieve the goal.

Each constant is a Metric: a frozen { template } that carries the type of its evidence (Metric<{ request: string; answer: string }>) and, for a label metric, of its labels. faithfulnessV2.template is the template. To use a template of your own, write its assembled JSON file (see the template contract) and give it its evidence type: const mine: Metric<{ request: string; answer: string }> = { template: loadTemplate(data) }. rubricJudge, classifierJudge and trajectoryJudge refuse a metric template when they are made.

The agent-process metrics grade a recorded run. “Spans” in the table are the spans that the trial recorded (ctx.span, spansOfAiSdk, or spansOf of OpenTelemetry): the metric reads them as trajectoryJudge does, so the evidence has no field for them, and a spans key in it is not read. The test form, a definition and calibrate (which replays LabeledOutput.spans through ctx.span) all give them to the judge. A trial that recorded none ends in error before the model call, and a calibration case without spans is listed in errors. planQualityV2 grades plan alone and needs none.

import { defineEval, type EvalCase } from '@trialkit/core';
import { metricJudge, type Tool } from '@trialkit/judge';
import { taskCompletionV2, toolAppropriatenessV2 } from '@trialkit/judge/metrics';
interface Job {
task: string;
tools: Tool[];
successCriteria: string[];
}
const bookRoom: Tool = {
name: 'book_room',
description: 'Books a meeting room.',
inputSchema: { type: 'object', properties: { room: { type: 'string' } } },
};
// The task records its spans with `ctx.span`, for example from the result of the AI SDK (`spansOfAiSdk`).
export default defineEval({
name: 'agent',
dataset: {
cases: [
{
id: 'book-iris',
input: {
task: 'Book room Iris.',
tools: [bookRoom],
successCriteria: ['The booking service confirms room Iris.'],
},
},
],
},
task: runAgent,
scorers: [
metricJudge(toolAppropriatenessV2, {
model,
evidence: ({ input }: EvalCase<Job>) => ({ request: input.task, tools: input.tools }),
}),
metricJudge(taskCompletionV2, {
model,
evidence: ({ input }: EvalCase<Job>) => ({
request: input.task,
successCriteria: input.successCriteria,
}),
}),
],
});

Rules of the metrics:

  • Ranking and reference coverage read the factual reference answer, never the human label. Ranking fails when any relevant source comes after an irrelevant one; there is no top-k option.
  • Alignment reads curated ideal contexts as sources, not retrieved passages. Citation grounding accepts standalone [N] markers and supported uncited claims, and every attached citation must support its claim. Normalize other citation formats before you build the evidence, and keep claim attachment and target identity.
  • Summary quality needs every claim to follow from the source (one item of sources) and every material point within scope to be covered. The scope is the whole source unless the request narrows it, and the rubric defines materiality.
  • Instruction following uses numbered atomic instructions in the trusted rubric. State precedence and conditions there, and resolve contradictory policy before you label. The request and quoted text cannot replace policy. Use deterministic scorers for exact formatting.
  • Empty sources are valid insufficient evidence for contextSufficiencyV2, and they fail relevance, ranking, coverage and alignment. An empty answer fails, unless an applicable instruction of instructionFollowingV2 requires silence.
  • The judge never trims, deduplicates or renumbers sources, and it has no size limit: refuse a list over your own limit before the call.
  • The transcript of the spans has no tool definitions, so tools is evidence that you supply: the name, description and input schema of each tool that the agent could use. Canonical JSON counts the depth of a tool from the tool, so a schema of 499 levels is valid.
  • planAdherenceV2: choose planSpanId before execution and record the public planner response in the gen_ai.output.messages of that span. Use the same text as plan when you also score quality. A missing designated span, a span without a plan and a retrospective plan fail. Equivalent implementations and branches that the plan allows can pass; replanning has no automatic exemption; a bad plan that the run follows can pass.
  • stepEfficiencyV2 judges justified steps. A model-only run can pass, and so can a retry that the evidence justifies; do not count copied history, or a model message that describes a tool call, as another action. Record order shows the order of recording, not elapsed time or causality. taskCompletionV2 needs observed evidence for each criterion: a recovered tool error can pass.
  • The twenty-one metrics replace the twenty-three version 1 recipes of the same properties: the two single-reply conversation recipes are the target of knowledgeRetentionV2 and roleAdherenceV2. Six ids lose their rag- prefix, five their agent- prefix and six their conversation- prefix, and the prompts and hashes are new, so treat the first run as a new baseline. They are binary verdicts, not DeepEval’s fractions, its minimum of alignment and coverage, or its extracted plans.

prompt: { system, user } replaces the system and user text of the template of the metric. Use it when the built-in wording does not fit your domain and a rubric is not enough. Everything else stays the library’s: the evidence fields, their checks and their rendering, the reply schema, the strict reading of the reply, error against fail, the rubric (a required rubric is still required) and, for a label metric, the labels and their scores.

import { metricJudge } from '@trialkit/judge';
import { faithfulnessV2 } from '@trialkit/judge/metrics';
const faithfulness = metricJudge(faithfulnessV2, {
model,
rubric: 'Treat a price as a factual claim.',
prompt: {
system: 'You grade whether a support answer follows from its sources.\n\nRubric:\n{{rubric}}',
user: '<question>\n{{request}}\n</question>\n\n<sources>\n{{sources}}\n</sources>\n\n<answer>\n{{answer}}\n</answer>\n\nReply with the JSON object only.',
},
});

The judge checks prompt when it is made, with the placeholder rule that loadTemplate applies to a metric template:

  • system has {{rubric}}, and {{choices}} for a label metric. user has the {{field}} of each evidence field of the metric (faithfulnessV2: request, sources and answer). No other placeholder may appear. A placeholder may repeat, and the order is yours.
  • system and user are strings that are not blank, and the object has no other key. null and undefined mean no prompt.
  • A bad prompt throws a TypeError that names the function, the metric and the problem, for example metricJudge: faithfulness@2: prompt.user has no {{sources}} placeholder. The metric evidence contract lists every message.

For a long prompt, write the text in a YAML file and read it with readPrompt(path) of @trialkit/judge/yaml. It returns PromptText, the type that prompt takes, so the call needs no cast. This file, evals/faithfulness-prompt.yaml, holds the prompt of the example above:

system: |-
You grade whether a support answer follows from its sources.
Rubric:
{{rubric}}
user: |-
<question>
{{request}}
</question>
<sources>
{{sources}}
</sources>
<answer>
{{answer}}
</answer>
Reply with the JSON object only.
import { metricJudge } from '@trialkit/judge';
import { faithfulnessV2 } from '@trialkit/judge/metrics';
import { readPrompt } from '@trialkit/judge/yaml';
export const faithfulness = metricJudge(faithfulnessV2, {
model,
rubric: 'Treat a price as a factual claim.',
prompt: readPrompt('evals/faithfulness-prompt.yaml'),
});
  • Write each text as a block scalar and use |-: the text ends at its last character. | adds a line break at the end, and the prompt hash changes with it. Indent with spaces: a tab character anywhere in the file is a YAML problem.
  • The file has the keys system and user, and no other. Both are strings that are not blank. A key that is neither is a TypeError that names it (the first one in UTF-16 order), so a misspelled usr is an error, and so is a whole template file. readPrompt does not check the placeholders, because they depend on the metric: metricJudge checks them when it makes the judge.
  • A prompt that is not valid is a TypeError with <path>: before its message, for example evals/faithfulness-prompt.yaml: prompt.user must be a nonblank string, got undefined. A YAML problem is a TypeError with <path>:<line>: <reason>, by the rules of the template files. readPrompt reads the file with readYamlMapping of @trialkit/cases, so install both: npm install --save-dev @trialkit/cases yaml.

The prompt is a new measurement:

  • data.judge.template and the scorer version stay the metric (faithfulness@2), and data.judge.promptSource is 'override'. data.judge.promptHash is the hash of the system prompt that was sent, as for every judge. It does not cover user, so an override also records data.judge.userTemplateHash, the hash of its user text, as one of the settings: settingsHash covers it, and two prompts that differ in user alone are two judges. A built-in judge records none. settingsHash does not cover the source. A prompt that has the text of the template has the hash of the built-in judge, and another source and settings hash.
  • Calibrate the judge that has your prompt: the calibration of the built-in prompt says nothing about it. calibrate reports judge.promptSource ('override' for your prompt), and it refuses a mix of built-in and override judges.
  • trialkit compare warns when the source differs (judge faithfulness prompt source: built-in → override), and trialkit report shows the judge as faithfulness@2 · override sha256:…. A history from before promptSource is built-in.

readTemplate(path) of @trialkit/judge/yaml reads a template from a YAML file and returns what loadTemplate makes of it. It reads the file with readYamlMapping of @trialkit/cases, so install both: npm install --save-dev @trialkit/cases yaml.

A metric of your own, in evals/polite-answer.yaml:

id: polite-answer
version: "1"
description: Checks that the answer is polite.
output: verdict
evidence: [request, answer]
rubric: optional
system: |-
You are an impartial grader of politeness. You do not write or improve the evidence.
Metric definition:
The answer is polite. Rude or dismissive wording fails.
Evidence:
- <request>: what the user asked.
- <answer>: the answer to assess.
Application rubric:
{{rubric}}
Reply with a JSON object containing exactly "reason" (a string) and "pass" (a boolean).
user: |-
<request>
{{request}}
</request>
<answer>
{{answer}}
</answer>
Assess the evidence. Reply with the JSON object only.
schema:
type: object
properties:
reason: { type: string }
pass: { type: boolean }
required: [reason, pass]
additionalProperties: false
import { type Metric, metricJudge } from '@trialkit/judge';
import { readTemplate } from '@trialkit/judge/yaml';
const polite: Metric<{ request: string; answer: string }> = {
template: readTemplate('evals/polite-answer.yaml'),
};
export const politeness = metricJudge(polite, { model });

To keep a built-in metric and change only its text, write a prompt file, which needs none of the template fields, and read it with readPrompt: Replace the prompt.

  • A YAML problem is a TypeError with <path>:<line>: <reason>: the rules are those of the case files (the 1.2 core schema, one document, unique keys, no aliases, tags or timestamps, at most 64 levels). A template that loadTemplate refuses is a TypeError with <path>: before its message. That includes a key that is not a template field, such as a misspelled lables.
  • Write version: "1" in quotes. YAML reads 1 as a number and 2025-11-15 as a timestamp, and both fail. Use |- for text that ends at its last character: | adds a line break at the end, and the prompt hash changes with it.

JudgeTemplate has id, version, description, system, user, output ('verdict' or 'choice'), and a reply schema. Optional requires: ['expected'] requires a nonblank selected reference; optional labels fixes the classifier labels and their order. Both default to no extra constraint. loadTemplate(data) validates these contracts, the exact reply schema, the placeholders and the keys: a key that is not a field of the template, such as a misspelled lables, is refused with a message that names it. Templates may use {{input}}, {{output}}, {{expected}}, {{rubric}}, {{context}}, and, for choice templates, {{choices}}; unknown placeholders fail validation. An expected option for a template without {{expected}} is refused, as an unused context option is.

A metric template (for example faithfulnessV2.template) also has evidence (its fields in prompt order, from request, answer, sources, reference, tools, successCriteria, plan, planSpanId, conversation, target and spans; target comes right after conversation), rubric ('optional' or 'required') and, for choice output, choices (the score of each label). It has exactly its declared placeholders: {{<field>}} for each evidence field in user, and {{rubric}} (and {{choices}}) in system. The template contract gives the rendering of each field.

The built-ins are deeply frozen, versioned prompt data. Do not change released prompt text; add a new id or version. Compare judge verdicts only when scorer version, data.judge.template, data.judge.promptHash, data.judge.settingsHash, data.judge.promptSource and data.judge.model agree. The prompt hash covers the rendered system prompt, including rubric and labels, but not a case’s evidence. The settings hash covers canonical JSON of temperature, evidence and, for a classifier, choices and passByLabel, and for an override userTemplateHash, the hash of its user text (the prompt hash covers the system prompt only); it does not cover the prompt source. Runs recorded before evidence was recorded are not comparable with later runs: they wrote a non-string input or output as JSON in insertion order, and their settings hash is missing or does not cover evidence. Treat the first run after the change as a new baseline. A run recorded before promptSource has the source built-in.

knowledgeRetentionV2, roleAdherenceV2, topicAdherenceV2, turnRelevanceV2, completenessV2 and goalAccuracyV2 are metric judges of a conversation. Their evidence is a conversation and, for the first four, an optional target. They are trialkit definitions, not DeepEval-compatible scores.

Conversation is { turns: ConversationTurn[] }. Every turn has a unique nonblank id, role ('user', 'assistant' or 'tool') and string content. Assistant decisions may include ordered retrievedContexts, toolCalls: [{ id, name, arguments }] and availableTools: [{ name, description, inputSchema }]; a tool result needs callId for an earlier call and status: 'ok' | 'error'. Any turn may link spanIds. Call IDs are unique separately from turn IDs. Menus and sources are snapshots, including empty lists; no history is pooled automatically. The judge refuses unknown fields, JSON that canonical JSON cannot write, an empty transcript, no assistant decision, and an assistant decision without preceding user context, as misuse that names the cause: metricJudge: goal-accuracy@2: evidence.conversation must be a valid conversation: turn ids must be unique for case c1, got {"turns":[.... It shows each turn with its 1-based number, id, role, content and the calls and results that it has, and no menu, snapshot or span ID.

import { metricJudge } from '@trialkit/judge';
import { completenessV2, knowledgeRetentionV2 } from '@trialkit/judge/metrics';
import { describeEval } from '@trialkit/vitest';
const completeness = metricJudge(completenessV2, { model });
const retention = metricJudge(knowledgeRetentionV2, { model });
describeEval('agent', { task: converse }, (it) => {
it('books a room', async ({ run, score }) => {
// The conversation that the agent ended is the evidence.
const result = await run({ conversation: seed });
await score(completeness, { conversation: result.conversation });
// One reply of it: the conversation ends at the target turn, and only that reply is graded.
await score(retention, { conversation: result.conversation, target: 'a2' });
});
});

Put the seed prefix in input.conversation, return the complete transcript in output.conversation, and preserve the prefix exactly. Without a target, knowledgeRetentionV2, roleAdherenceV2, topicAdherenceV2 and turnRelevanceV2 grade every assistant reply against its full preceding prefix, and a later correction cannot erase an earlier violation. With a target, the conversation ends at that turn and only that reply is graded: every earlier turn, also an earlier assistant reply, is context, and no later turn reaches the model. A target that no turn has, or that names a user or tool turn, is misuse, and so is a bad turn after the target. completenessV2 and goalAccuracyV2 judge the conversation through its end and read no target; a target key in their evidence is ignored.

A version 1 single-reply recipe took a history, the current user turn and the reply. Give the same conversation instead: the history, the user turn and the reply as an assistant turn, with that turn as the target. knowledgeRetentionV2 fails a needless re-ask of a fact that the user gave, for a targeted reply too.

Empty answers fail: an assistant turn that only calls tools is a decision, not an empty answer, and any other empty assistant reply fails. Tool turns are evidence, not user goals or policy. A topic change does not cancel a request; only an explicit cancellation removes the obligation. Promises and pending clarification are incomplete unless resolved by the end; a justified refusal can close a request without proving goal achievement. Executing an explicitly cancelled action afterward fails completeness, and an acknowledged stop request with observed cessation can be resolved. Policy belongs in the trusted rubric, not in the transcript. Goal accuracy is separate from completeness: a success claim does not prove an external action, and a justified refusal does not achieve the goal. Explicit corrections replace affected goals. Cancellation removes a goal but forbids executing the cancelled action later; a stop can succeed through acknowledgement and observed cessation. Set a size limit of your own and refuse oversized evidence without trimming.

retrievalTurns(conversation, references?) and toolDecisions(conversation, spans) make the evidence of each assistant decision of a conversation. Give each result to a metric as it is, and keep each verdict apart: do not average them, or combine a decision with plan quality or adherence.

Function Returns Refuses (misuse)
retrievalTurns(conversation, references?) One RetrievalTurn for each assistant turn that has a retrievedContexts snapshot, also an empty one: turnId, request (the dialogue before the turn), answer (its content) and sources (its own snapshot). With references, a reference too. An invalid conversation, no turn with a snapshot, references that is not an object, and a retrieval turn whose reference is not a nonblank own key of references.
toolDecisions(conversation, spans) One ToolDecision for each assistant turn, also one without a call: turnId, request (the dialogue before the turn), tools (its availableTools) and spans (the spans that the turn and the results of its calls map, in record order). An invalid conversation, spans that are not a nonempty valid list, a turn without its menu, mapping or spans, a span that two turns map, and a call whose span is no execute_tool span of the same tool with the same arguments.

references is the factual reference answer of each turn, by turn ID, not a human label. A turn without a snapshot is not a retrieval turn. No earlier snapshot is pooled and no later turn is shown: the evidence of a decision depends on its prefix alone. A span keeps its parent only when the parent is one of the decision’s spans. Arguments of a call and its span compare as canonical JSON, so key order does not matter and 1 is not true; arguments that canonical JSON cannot write, such as a string with a lone surrogate, are no match. The conversation projection contract has every message.

import { calibrate, metricJudge, retrievalTurns, toolDecisions } from '@trialkit/judge';
import { faithfulnessV2, toolAppropriatenessV2 } from '@trialkit/judge/metrics';
import { describeEval } from '@trialkit/vitest';
// One faithfulness verdict for each retrieval turn, in a test body.
const faithful = metricJudge(faithfulnessV2, { model });
describeEval('retrieval', { task: converse }, (it) => {
it('grades each retrieval turn', async ({ score }) => {
for (const turn of retrievalTurns(conversation)) await score(faithful, turn);
});
});
// One calibration item for each decision: the output is the evidence, and `calibrate` replays its spans.
const labeled = toolDecisions(conversation, spans).map((decision) => ({
evalCase: { id: decision.turnId, input: decision.request },
output: decision,
label: humanLabels[decision.turnId] ?? false,
spans: decision.spans,
}));
const report = await calibrate(metricJudge(toolAppropriatenessV2, { model }), labeled);

The ranking and coverage metrics read reference: pass references to retrievalTurns. A trial that grades a decision with toolAppropriatenessV2 must record decision.spans with ctx.span, as the task of a trial does. The calibration example shows both.

The person-run calibration example has opt-in paid commands for TypeScript and Python. Freeze before holdout; require full n and zero errors, and inspect every disagreement. Its tiny fictional MIT corpus is smoke calibration, not universal accuracy. Calibrate on application-held-out data before gating. Select only the stated text fields of the evidence, not a whole application object, and redact sources, history and spans before sending them to a model. Reports can contain judge reasons: review them before sharing.

calibrate(judge, labeled, options?) runs a scorer over human-labeled outputs. LabeledOutput is { evalCase, output, label, spans? }; label is boolean for a rubric score or a classifier label string. spans defaults to none. When supplied, each span is replayed through ctx.span in record order before scoring, with core validation and snapshots. Missing spans work as before for non-trajectory judges. The list must be non-empty and have valid, unique eval cases.

CalibrateOptions option Default Meaning
concurrency 4 Integer at least 1, or Infinity.
signal None Aborts running calls, prevents new calls, and rejects calibration.
trialTimeoutMs 120000 Per-call integer millisecond limit from 1 through 2147483647. Timed-out calls enter errors.

CalibrationReport contains judge, n, agreement, agreementInterval, Cohen’s kappa, confusion, results, disagreements, errors, and aggregate usage. results lists every scored case, in the order of labeled, as a CalibrationResult (caseId, and the human and judge labels: the judge of a result is a label, not the judge of the report). Disagreement is a result where the labels differ, with an optional reason. CalibrationError has caseId and message. agreement is NaN for no scored cases; kappa is also NaN when it is undefined. Keep calibration data separate from cases used to tune the rubric.

agreementInterval is { low, high }: the 95% Wilson interval of agreement over the n scored cases, from wilson of @trialkit/core. 6 of 6 agree gives 0.61 to 1, so a small set cannot show 100%. A case in errors is in neither n nor the interval. With no scored case the interval is 0 to 1, so a gate on agreementInterval.low fails when every case errored.

judge is { template, model, promptHash, settingsHash, promptSource } (JudgeIdentity), as the scores of the n verdicts record it in data.judge. promptSource is built-in, or override for a metricJudge with a prompt. A change of the rubric, label rule, temperature, template, prompt or model changes it. The model that replied does not. Compare it with data.judge of the scores of an eval to check that they come from the calibrated configuration. judge is null when n is 0, and when the scorer records no judge. A scorer of your own gets the four string fields reported when it puts them in data.judge; its promptSource is built-in unless it records override.

import { calibrate } from '@trialkit/judge';
import { expect } from 'vitest';
import { summary } from './judges.ts';
const report = await calibrate(summary, labeledOutputs);
expect(report.errors).toEqual([]);
expect(report.agreementInterval.low).toBeGreaterThanOrEqual(0.7);
const calibrated = report.judge; // keep it with the labels, and compare it with the eval's scores

calibrate refuses a mix of judges: it rejects with a TypeError when the scores of the verdicts differ in template, model, promptHash, settingsHash or promptSource, or when only some of them record data.judge. The message names the first field that differs and the two cases. A case with no verdict is not compared. The check follows the judge calls, so a rejected calibration has used the model.

generateCases(options) from @trialkit/judge/generate asks a model once for new eval cases and returns them as rows for a person to review. It runs in every runtime, like the judges. writeCases saves the rows as a draft file; copy the approved rows into the dataset. The references that a model writes are not ground truth.

import { writeCases } from '@trialkit/cases/write';
import { readCases } from '@trialkit/core/node';
import { generateCases } from '@trialkit/judge/generate';
const result = await generateCases({
model,
source: { cases: await readCases('seeds.jsonl') },
count: 12,
seed: 7,
});
await writeCases('draft.yaml', result.rows);
Option Default Meaning
model Required A core ModelClient. Each row records its id.
source Required Exactly one of { cases } (seed cases), { documents: [{ id, text }] } (unique ids) or { description }. Each is non-empty. Only seed input and expected, document ids and texts, or the description are sent.
count Required Integer from 1 to 100: the number of candidates to ask for. The reply must have exactly this many.
existing None Cases that the rows must not repeat, for example the dataset that the draft joins. Checked here, never sent. Seed cases count as existing.
seed 0 Integer from 0 to 4294967295 that the prompt shows. It is a nonce for the prompt, not a seed for the model’s random numbers.
maxOutputTokens 4096 Integer from 1 to 32768, sent to the model. The SDK or the provider enforces it.
temperature Omitted Number from 0 through 2. Omitted means no request field.
requestTimeoutMs 120000 Number above 0 and at most 2147483647. At the deadline the call is aborted (signal), and a late reply is ignored.

The result has rows, discarded and usage. Each row is a core EvalCase with the id generated-<sha256 of the canonical JSON of its input>, the tag generated, and the metadata generated, generation.model, generation.replyModel, generation.template (cases@1), generation.seed, generation.sourceHash and generation.promptHash. Keep them after the review: they record the origin, not an approval. count asks for candidates: a candidate that repeats an existing case, a seed case or an earlier candidate (same id or same canonical input; no trimming or case folding) is dropped and counted in discarded, and nothing refills it. usage is the usage of the reply, and absent when the reply has none.

The model is called once, with no retry and no cache. Options that are not valid throw a TypeError or a RangeError before the call; a prompt of more than 64 KiB throws a RangeError there too. A failed call, a reply that is not exactly { "cases": [{ "input", "expected" }] } with count valid cases (one invalid candidate rejects all), a reply text or a structured output of more than 64 KiB (each is checked on its own), a case nested more than a case file allows (64 levels), and a reply with no new case throw an infrastructure error, and no row is returned. The reply schema allows text, numbers, booleans, null and arrays of these, because strict structured output cannot express a free-form object. The limits are not a hard cost cap, and a failed call can cost money. The template contract has the prompt and the provenance.

openaiModel({ client, model, id?, maxOutputTokensParam? }) adapts a structural Chat Completions client. client and non-empty model are required; id defaults to <server>:<model> and maxOutputTokensParam defaults to max_completion_tokens (max_tokens is the alternative). It forwards messages, supplied temperature, max-output tokens, JSON schema, and the trial signal. OpenAIChatClient defines the required chat.completions.create shape; OpenAIChatBody, OpenAIChatReply, and OpenAIRequestOptions are its structural request/reply types.

aiSdkModel({ model, id? }) adapts an AI SDK language model or non-empty model-id string (AiSdkLanguageModel). id defaults to <provider>:<model> or gateway:<model> for a string. It sends the same request settings through generateText, enables system messages for both supported majors, and requests structured output when core supplies a JSON schema. Both adapters leave retries to their SDK, do not stream, preserve reply usage, and pass provider errors to the judge wrapper.

An OpenAI-compatible reply must name its model and have text content. Empty replies name a refusal or finish reason when present. IP-literal servers use the whole address and explicit port in the default id, for example 127.0.0.1:8000:m.

openaiModel names the server in the default id from the host of the baseURL of the client: openai for api.openai.com (and for a client with no baseURL), openrouter for openrouter.ai, databricks for *.databricks.com and *.azuredatabricks.net, and otherwise the first label of the host, for example localhost. Set id to name the server yourself. aiSdkModel takes the provider from the part of the provider of the model before the first dot, so anthropic('claude-haiku-5.5') gets anthropic:claude-haiku-5.5. Set id to name another route, for example bedrock:<model>; the adapter reaches every provider that the AI SDK supports. Both adapters throw a TypeError when you create them with a model or an id that is empty or not a string. openaiModel also throws one when the baseURL of the client has no host and no id is set.

A judge can call a governed model service, so judge calls get the gateway’s permissions, rate limits, guardrails and usage tracking. The gateway speaks the OpenAI-compatible Chat Completions API, so openaiModel reaches it with an openai client whose baseURL is the gateway:

const client = new OpenAI({ apiKey: token, baseURL: `${host}/ai-gateway/mlflow/v1` });
const judgeModel = openaiModel({
client,
model: 'system.ai.claude-sonnet-4-5',
maxOutputTokensParam: 'max_tokens',
});
  • The id is databricks:<model service>, for example databricks:system.ai.claude-sonnet-4-5. A judge score records it, so a change of the model service behind a judge shows in compare.
  • Your app gives the token to the openai client. The adapter reads no DATABRICKS_TOKEN and no Databricks config file.
  • Databricks documents max_tokens for this API, so the example sets maxOutputTokensParam: 'max_tokens'.
  • The usage records of the gateway (system.ai_gateway.usage) and the trial usage events are separate. A reporter can join them on the request time and the model service.
  • Python’s openai_model takes the base_url and the token of the gateway in the same way.