Skip to content

8. Judges

A judge is a scorer that asks a ModelClient (§9) to grade an output. An unreadable reply or a failed model call ends the trial in error, never in a verdict. Options and examples are in the TypeScript and Python references.

Each judge is a core Scorer. It calls the model once, returns one verdict or label, and takes reason from the reply. rubricJudge, classifierJudge and trajectoryJudge take your rubric; metricJudge asks one fixed, built-in question about typed evidence, and its prompt option replaces the text of that question. A judge decides pass itself, so it has no threshold argument. Judges are optional and for free text only: expected values and reference-based scorers come first, because a reference-free metric cannot see a wrong total that the context supports.

A prompt is versioned data that both languages load (template contract). A released version never changes. Why: a changed prompt changes verdicts, so runs that used the old text must stay comparable. Both packages ship copies of the files, and a test fails when a copy differs from spec/. Not built: a template that matches promptfoo’s llm-rubric prompt, for teams that leave promptfoo. It waits for the license question (§14), because a template that copies another tool’s prompt keeps the source’s license notice.

An app writes its own template as a YAML file, and readTemplate (read_template) reads it and checks it with loadTemplate. To change only the text of a built-in metric, the app writes a file with the keys system and user, and readPrompt (read_prompt) reads it for the prompt option. A template file, a prompt file and a case file refuse the same things: all go through readYamlMapping (read_yaml_mapping), so a copy of the YAML rules cannot drift. Template files and Replace the prompt in the judge reference have the examples.

A judge sends only what you name, because it calls a paid model: a picker chooses each of input, output, expected and context. The context picker puts text such as retrieved sources in a <context> tag apart from the output, so the judge does not grade it. The default temperature is none. Why: a fixed temperature lowers the variance of verdicts, but some models reject one (the OpenAI gpt-5 models answer temperature: 0 with HTTP 400). Set it only for a model that accepts it, and calibrate and compare with the same setting.

A judge accepts only a reply that matches the template exactly, and it never turns a failure into a verdict. An unreadable reply and a rejected model call are infrastructure errors: the trial ends in error, and errorRetries retries it. Why: a lenient parser can turn an unreadable reply into a pass, and a model that is down must never lower a pass rate (core would count a plain Error as a fail). There is no cache: each trial calls the model again, so repeated trials also measure the variance of the judge, and a cache would hide it.

A score records which judge made it (template and version, model, prompt hash, prompt source, temperature, evidence mode and settings hash), so a changed judge cannot pass as the same measurement (what a judge records). A change of template, model, promptHash, settingsHash or promptSource makes verdicts not comparable, and trialkit compare warns.

A metricJudge with a prompt records promptSource: 'override', and template stays the metric (rules). Why: the metric still decides the evidence and the reply, so the score keeps its name and version, and the prompt hash and the source tell the text apart. The source is its own field and not part of settingsHash: the settings hash of every existing history stays valid, and a prompt that has the text of the template still differs from the built-in judge by its source. The prompt hash covers the system prompt only, and a built-in template fixes its user text, so an override also records the hash of its user text (userTemplateHash) in the settings: two overrides that differ in user alone have another settingsHash, and compare and calibrate tell them apart. A built-in judge records none, so no existing hash changes. A history from before the field is built-in.

The trajectory judge grades the spans that a trial recorded, not the task’s output. It sends canonical JSON in record order: span names, ids and status and selected GenAI attributes, without times or other attributes. It treats the transcript as untrusted data. Missing spans are misuse: the trial ends in error before any model call. Privacy: captured messages and tool results go to the judge model, so redact before recording spans.

generateCases asks a model once for new cases and returns rows; a person reviews a draft file before the cases join a dataset (options). It makes one call, with no retry, batch, cache or critic, and it drops a candidate that repeats an existing case. Why: a refill loop hides how much the model repeats, and a retry spends money again. One invalid candidate rejects the whole reply. Why: a repaired reply would put text in a dataset that nobody wrote. Each row has the tag generated and metadata that record its origin, not its approval, and writeCases never overwrites a file. A generated reference is not ground truth. Not built: evolutions, critic loops, embeddings or chunking, PDF parsing, conversation simulation, red-team inputs and hosted datasets.

The twenty-one grounding, output-quality, agent-process and conversation metrics are version 2 Metric constants for metricJudge (reference, contract). Each selects one independent semantic rule, not a composite quality score, and none is enabled by default. Definitions stay in the system prompt, and the named evidence goes in the user prompt as tagged untrusted data. The caller bounds the evidence, and trialkit never trims, deduplicates or renumbers it. Each assistant decision gets its own verdict from its own prefix, so do not average them.

Calibration measures how well a judge agrees with people; treat a judge with no calibration result as advisory (reference, guide). agreementInterval is the 95% Wilson interval of agreement: 6 of 6 agree gives 0.61 to 1. Why: 6 of 6 is a small-sample result, and a gate on the interval’s low bound passes only when enough cases ran. The report’s judge identity comes from the scores of the verdicts. Why: calibrate takes any scorer, and the scores are the only record of the configuration that gave the verdicts. calibrate rejects a mix of configurations, because a report over two of them calibrates neither. A judge that threw gave no verdict: it is in errors, not in n.