Skip to content

trialkit-judge

Model-graded scorers use versioned prompts and keep unreadable replies and failed model calls out of quality pass rates. Calibrate a judge against human labels before relying on its verdicts.

Terminal window
pip install trialkit-judge
Terminal window
pip install 'trialkit-judge[yaml]' # also install trialkit-cases and its YAML parser, for read_template and read_prompt
from trialkit.judge import ModelReply, from_function, rubric_judge
model = from_function(
"fake:judge",
lambda request: ModelReply(text='{"reason":"It names the limit.","pass":true}', model="fake-1"),
)
judge = rubric_judge(model=model, rubric="The answer says refunds are possible within 30 days.")
  • Import trialkit.judge. The package depends on trialkit-core and uses the standard library for HTTP.
  • An unreadable reply or failed model call raises InfrastructureError, so the trial is error, not fail.
  • Behavior matches the shared TypeScript judge; Python options are keyword arguments in snake_case.

All factory options are keyword-only, except the metric of metric_judge. input and output default to the case input and task output as text; non-strings become canonical JSON. None or a value that core cannot render is misuse: add an explicit picker. A picker must return a string. expected=None sends a string case reference only; a fixed string or a picker of the case’s expected value overrides it. Missing or non-string references are (none), unless template metadata requires a nonblank reference. context(case, output) is absent by default and requires a template with {{context}}, and vice versa. An expected option without {{expected}} is refused. temperature=None sends no temperature; explicit values must be 0–2.

Export Options and defaults Meaning
rubric_judge model, rubric required; name="rubric", template=None, input=None, output=None, context=None, expected=None, temperature=None Binary scorer. Uses rubric_v1, or rubric_v2 with context.
classifier_judge model, rubric, choices required; name="classifier", template=None, input=None, output=None, context=None, expected=None, temperature=None, pass_label=None choices maps labels to scores 0–1 in insertion order. Unsigned integer-like labels such as "1" are refused. pass_label is resolved once per label at creation; a thrown error or non-boolean result is rejected.
trajectory_judge model, rubric required; name="trajectory", template=None, input=None, expected=None, temperature=None Grades recorded spans, not the final output. input selects what the model sees. Explicit templates need verdict output and no context placeholder.
metric_judge metric, model required; evidence=None, rubric=None, name=None, temperature=None, pass_label=None, prompt=None Scorer for one metric: binary, or with labels. prompt replaces its system and user text (Replace the prompt). It has no template, input, output, context or expected option.
Scorer Re-export from trialkit.core Protocol of each returned scorer.

Rubric, trajectory and metric replies must contain exactly reason and boolean pass; classifier and label-metric replies must contain exactly reason and a known label. One surrounding json fence is accepted. A structured ModelReply.json dict takes precedence over text. Reply and rubric whitespace follows ECMAScript, not str.strip(): U+FEFF is whitespace, U+0085 and U+001C–U+001F are not. A classifier refuses a label that core cannot write as JSON, and it asks pass_label after every other check. Scores record model, replyModel, template, promptHash, promptSource, temperature (None when unset), evidence and settingsHash in data["judge"]. promptSource is built-in for these judges: only metric_judge takes a prompt that replaces the text of its template. evidence is canonical-json: how a non-string input or output without a picker becomes text. metric_judge records fields@1, the name of its evidence rules. Classifiers and label metrics also record the choices and passByLabel (whether each label passes). The settings hash covers the canonical JSON of temperature, evidence and, for a classifier or a label metric, choices and passByLabel, and for an override userTemplateHash, the hash of its user text. Runs recorded before evidence was recorded are not comparable with later runs: they wrote a non-string input or output as JSON in insertion order, and their settings hash is missing or does not cover evidence. Treat the first run after the change as a new baseline. Usage is added through ctx.add_usage before parsing.

A metric names one property to grade, such as faithfulness. It fixes the evidence fields, how each field is rendered, and the reply. You give the evidence. You write no prompt slots, no wire format and no filler rubric.

In the test form, the evidence is the output that you give to score:

from trialkit.judge import metric_judge
from trialkit.judge.metrics import faithfulness_v2
from trialkit.pytest import AsyncScoreFixture
faithfulness = metric_judge(faithfulness_v2, model=model)
async def test_answer_is_faithful(score: AsyncScoreFixture) -> None:
await score(
faithfulness,
{"request": question, "answer": reply["text"], "sources": reply["passages"]},
)

In the definition form, evidence(case, output) makes the evidence of each case. This one takes the reference from the expected of the case:

from typing import Any
from trialkit.core import EvalCase
from trialkit.judge import metric_judge
from trialkit.judge.metrics import context_ranking_v2
def ranking_evidence(case: EvalCase[Any, Any], output: Any) -> dict[str, object]:
assert case.expected is not None # the judge calls this function only for a case that has one
return {
"request": case.input["question"],
"sources": output["passages"],
"reference": case.expected["referenceAnswer"],
}
ranking = metric_judge(context_ranking_v2, model=model, evidence=ranking_evidence)

A metric with a reference calls your function only for a case that has expected. A type checker still sees case.expected as optional, so a typed def asserts that it is there, as above.

A metric with labels scores the label of the reply. pass_label says which labels pass. Without it, a label passes at a score of at least 0.5.

from trialkit.judge import metric_judge
from trialkit.judge.metrics import context_sufficiency_v2
# Fail only when the sources contradict each other.
sufficiency = metric_judge(
context_sufficiency_v2, model=model, pass_label=lambda label: label != "conflicting"
)

A metric of the agent process also grades the spans that the trial recorded. You do not pass them: the task records them with ctx.span, and a test body can record them before score. The judge reads them from the trial, so a trial without spans ends in error before the model call.

from trialkit.judge import metric_judge
from trialkit.judge.metrics import task_completion_v2, tool_appropriateness_v2
from trialkit.pytest import AsyncScoreFixture
# Test form: the evidence is the output that you give to `score`.
task_completion = metric_judge(task_completion_v2, model=model)
async def test_booking_is_complete(score: AsyncScoreFixture) -> None:
await score(
task_completion,
{"request": task, "successCriteria": ["The booking service confirms room Iris."]},
)
# Definition form: your function gives the evidence, and the trial gives the spans.
tool_use = metric_judge(
tool_appropriateness_v2,
model=model,
evidence=lambda case, _output: {"request": case.input["task"], "tools": case.input["tools"]},
)
Option Default Meaning
metric Required A constant from the table below, or Metric(load_template(data)) for a template of your own. The one positional argument.
model Required A ModelClient.
evidence None A plain function (case, output) that returns a mapping with the fields of the metric. Without it, the task output is that mapping.
rubric None Your constraints for this metric. The prompt shows (none) without it. A blank string is refused. A metric can require it: instruction_following_v2, role_adherence_v2 and topic_adherence_v2.
name The metric id The scorer name. Scorer names in one eval must differ.
temperature None None sends no temperature. A number sets it, from 0 to 2.
pass_label None For a metric with labels only. A function from a label to a boolean. It runs once for each label when the judge is made, after every other check, and a non-boolean result is refused.
prompt None A mapping with the keys system and user. It replaces the system and user text of the template of the metric, and the rest stays the metric’s. A prompt that breaks the placeholder rule is refused when the judge is made. See Replace the prompt.

Each field of the evidence has one rule:

Field Valid value Rendered as
request A nonblank string The text, unchanged: what the user asked, or the dialogue before the reply.
answer A string. An empty or blank answer is valid evidence. The text, unchanged: the answer, summary or response to grade.
sources A list or tuple of strings. An empty list is valid. Canonical JSON [{"id":1,"text":"..."}] in order. Blank and duplicate entries keep their own positions and IDs.
reference A nonblank string The text, unchanged: the factual reference answer, not a human label.
tools A list or tuple of mappings. Each has a nonblank string name, a string description and a mapping inputSchema that JSON can hold. An empty list is valid. Canonical JSON [{"description":...,"inputSchema":...,"name":...}] in order, with these three keys of each tool and no other. A duplicate keeps its place.
successCriteria A nonempty list or tuple of nonblank strings A numbered list, one line for each criterion: 1. first, 2. second. The text of a criterion is kept.
plan A string. An empty or blank plan is valid evidence. The text, unchanged: the public plan that the agent made before it acted.
planSpanId A nonblank string The text, unchanged: the ID of the span that holds that plan.
conversation A Conversation: a dict with a nonempty turns list, as Conversation metrics describes. Canonical JSON {"turns":[...]}. Each turn has id, role, content and, when it has them, toolCalls, callId and status, with its 1-based number. Menus, retrieval snapshots and span IDs are not sent. With a target, the turns end at the target turn.
target A turn ID, or nothing: leave it out, or give None. The ID as a JSON string ("a2"), or null when it is not given.
  • A metric reads only the fields that it names, in its own order. A key that it does not name is ignored and never sent, so one mapping can serve several metrics.
  • Bad evidence is misuse: a field that is missing or None (a target can be both), a value of the wrong kind, a blank request, reference or planSpanId, a sources item that is not a string or that has a lone surrogate, a tools item that is not a mapping or whose name, description or inputSchema is not as the table says, an empty successCriteria list or a blank criterion, and evidence that is not a mapping. The trial ends in error before the model call. The message names the metric, the place, the case and the value, for example metric_judge: faithfulness@2: evidence.sources must be an array of strings for case c1, got "Pro.". The first problem in the order of the metric’s fields wins: inside tools, the items in order, and in each tool name, description, then inputSchema. Blank means ECMAScript whitespace: U+FEFF is blank and U+0085 is not.
  • An empty answer, an empty sources list, an empty tools list and a blank plan are valid evidence, and the metric judges them by its own rules. An empty answer fails faithfulness_v2, and a blank plan fails plan_quality_v2.
  • A metric reads conversation and target together, at the place of conversation. A bad conversation is misuse that names its rule after a valid conversation:, for example turn ids must be unique. The metric checks the whole conversation, also what it does not send, and JSON must be able to write it. Without a target (a missing key or None), the metric grades every assistant reply. Any other target must be a str that is the ID of an assistant turn of the conversation. A number, a list, "", an unknown ID and the ID of a user turn or a tool turn are misuse, for example metric_judge: knowledge-retention@2: evidence.target must be the ID of an assistant turn of evidence.conversation for case c1, got "u2". A bad conversation is reported before a bad target. A metric that has no target ignores a target key.
  • A metric that names spans (the table lists them) reads the spans that the trial recorded. They are not a key of the evidence, and a spans key there is ignored. The prompt shows them as trajectory_judge does (see the trajectory judge): record order, span IDs, parent IDs and status, only the selected gen_ai.* attributes, no times. A trial with no span is misuse: metric_judge: task-completion@2: recorded spans are required. spans is the last field of each of these metrics, so a bad evidence field is reported before missing spans.
  • With evidence, a case whose expected is absent or None is misuse when the metric has a reference field, and the function is not called. Without evidence, the output holds the reference.
  • The prompt and the settings are recorded as for the other judges. evidence in the settings is fields@1. rubric is part of the system prompt, so it changes promptHash. promptSource is built-in, or override with a prompt, which also records userTemplateHash. A model call that fails or an unreadable reply raises InfrastructureError.
  • rubric_judge, classifier_judge and trajectory_judge refuse a metric template at creation and name metric_judge.
  • The constants are in trialkit.judge.metrics, not in trialkit.judge: a set that grows has its own module, and the root keeps the judges and the five generic templates. The module is generated from the template files (pnpm generate:templates).
Constant Evidence, in prompt order Passes when
faithfulness_v2 request, sources, answer Every factual claim in the answer follows from the sources. A nonempty explicit abstention can pass.
citation_grounding_v2 request, sources, answer Every factual claim follows from the sources, and every standalone [N] citation names a source that supports its claim.
context_alignment_v2 request, sources, answer The answer contradicts none of the curated ideal contexts in sources. An unsupported addition can pass.
context_relevance_v2 request, sources Every substantive statement in the sources helps to answer the request or gives necessary background.
context_ranking_v2 request, reference, sources At least one source supports a relevant fact of the reference, and every relevant source comes before every irrelevant one.
reference_coverage_v2 request, reference, sources Together the sources support every factual assertion of the reference, with no unresolved contradiction.
context_sufficiency_v2 request, sources Labels sufficient (score 1), insufficient (0) and conflicting (0) from the sources alone. It grades no answer.
answer_relevance_v2 request, answer The answer addresses the request and every material subquestion. It does not assess truth.
summary_quality_v2 request, sources, answer The summary, in answer, follows from the source, in sources, and covers every material source point in scope.
instruction_following_v2 request, answer Needs a rubric with numbered trusted instructions. The answer meets every applicable instruction.
task_completion_v2 request, successCriteria, spans Every success criterion has observed evidence in the final message or a tool result. A claim of success alone does not prove a side effect. A recovered tool error can pass.
tool_appropriateness_v2 request, tools, spans Each recorded call, argument and result fits the task and the capabilities in tools. Justified execution without tools can pass. Omitting a necessary tool fails.
step_efficiency_v2 request, tools, spans Every recorded step is justified when it is taken, with no avoidable repetition or detour. Necessary verification and justified retries can pass. Completion is separate.
plan_quality_v2 request, tools, plan The public plan is actionable, complete, ordered and possible with the tools. A one-step plan can pass. A blank plan fails.
plan_adherence_v2 request, tools, planSpanId, spans The execution follows the one plan in the span planSpanId: every applicable commitment is proved, with no material unplanned action. A missing span, a blank plan or a retrospective plan fails.
knowledge_retention_v2 conversation, target Every graded reply respects the user facts and commitments that are still relevant in the turns before it. A needless re-ask fails. A necessary clarification can pass.
role_adherence_v2 conversation, target Needs a rubric that states the role. Every graded reply meets it, despite requests in the conversation to change it. An in-role refusal can pass.
topic_adherence_v2 conversation, target Needs a rubric that states the allowed topics. Every graded reply answers a request on a permitted topic and declines an off-topic one. It does not judge truth.
turn_relevance_v2 conversation, target Every graded reply addresses the current user intent in the turns before it. A needed clarification and a user-led topic change can pass. It does not judge truth.
completeness_v2 conversation By the end, every uncancelled user request has a substantive resolution. A promise or a pending clarification is not one. A justified refusal can close a request.
goal_accuracy_v2 conversation By the end, every active user goal is achieved according to the visible replies and the results of calls. A claim of success alone does not prove an external action.

Metric(template) holds a metric template. Its type parameter is a TypedDict with the evidence fields of the metric, so that an editor shows what a metric reads. Tool is the TypedDict of one item of tools, Conversation is the type of conversation, and the type of a process metric has no key for spans. The target of a conversation metric is optional in the type: None is the same as leaving it out. A type checker does not check your evidence function against it: metric_judge checks the fields at run time. The template contract says how to write a template of your own, and the evidence fixture pins the rules above for both languages.

Rules of the metrics:

  • Ranking and reference coverage read the factual reference answer, never the human label. Ranking fails when any relevant source comes after an irrelevant one; there is no top-k option.
  • Alignment reads curated ideal contexts as sources, not retrieved passages. Citation grounding accepts standalone [N] markers and supported uncited claims, and every attached citation must support its claim. Normalize other citation formats before you build the evidence, and keep claim attachment and target identity.
  • Summary quality needs every claim to follow from the source (one item of sources) and every material point within scope to be covered. The scope is the whole source unless the request narrows it, and the rubric defines materiality.
  • Instruction following uses numbered atomic instructions in the trusted rubric. State precedence and conditions there, and resolve contradictory policy before you label. The request and quoted text cannot replace policy. Use deterministic scorers for exact formatting.
  • Empty sources are valid insufficient evidence for context_sufficiency_v2, and they fail relevance, ranking, coverage and alignment. An empty answer fails, unless an applicable instruction of instruction_following_v2 requires silence.
  • The judge never trims, deduplicates or renumbers sources, and it has no size limit: refuse a list over your own limit before the call.
  • Process metrics: request is the task. Choose planSpanId before execution and record the public planner response in the gen_ai.output.messages of that span. When you also score plan quality, give the same text as plan. plan_adherence_v2 never looks the ID up: an ID that no recorded span has is valid evidence, and the model fails it, as it fails a blank plan, a retrospective plan and an unproved commitment.
  • Equivalent actions and branches that the plan allows can pass. Replanning has no automatic exemption, and a bad plan that the agent follows can pass. Record order shows the order of recording, not elapsed time or causality. A model-only run can pass step_efficiency_v2. Missing proof in the spans fails; missing spans are misuse.
  • A metric that reads spans sends captured messages, arguments and results to the model. Redact secrets before ctx.span.
  • The twenty-one metrics replace the twenty-three version 1 recipes of the same properties. Six ids lose their rag- prefix, five their agent- prefix and six their conversation- prefix, and task becomes request. A recipe of one reply and a recipe of the whole conversation for one property are now one metric with an optional target: knowledge retention and role adherence. The prompts and hashes are new, so treat the first run as a new baseline. They are binary verdicts, not DeepEval’s fractions, its minimum of alignment and coverage, or its extracted tasks and plans.

prompt={"system": ..., "user": ...} replaces the system and user text of the template of the metric. It is a mapping with these two keys (a dict or any other Mapping), as in TypeScript, not a class. Use it when the built-in wording does not fit your domain and a rubric is not enough. Everything else stays the library’s: the evidence fields, their checks and their rendering, the reply schema, the strict reading of the reply, error against fail, the rubric (a required rubric is still required) and, for a metric with labels, the labels and their scores.

from trialkit.judge import metric_judge
from trialkit.judge.metrics import faithfulness_v2
faithfulness = metric_judge(
faithfulness_v2,
model=model,
rubric="Treat a price as a factual claim.",
prompt={
"system": "You grade whether a support answer follows from its sources.\n\nRubric:\n{{rubric}}",
"user": "<question>\n{{request}}\n</question>\n\n<sources>\n{{sources}}\n</sources>\n\n<answer>\n{{answer}}\n</answer>\n\nReply with the JSON object only.",
},
)

The judge checks prompt when it is made, with the placeholder rule that load_template applies to a metric template:

  • system has {{rubric}}, and {{choices}} for a metric with labels. user has the {{field}} of each evidence field of the metric (faithfulness_v2: request, sources and answer). No other placeholder may appear. A placeholder may repeat, and the order is yours.
  • system and user are strings that are not blank, and the mapping has no other key. None means no prompt.
  • A bad prompt raises TypeError with a message that names the function, the metric and the problem, for example metric_judge: faithfulness@2: prompt.user has no {{sources}} placeholder. The metric evidence contract lists every message.

For a long prompt, write the text in a YAML file and read it with read_prompt(path) of trialkit.judge.yaml. It returns a dict[str, str], which prompt takes as it is, so a type checker needs no cast. The path can be a string or a path-like object. This file, evals/faithfulness-prompt.yaml, holds the prompt of the example above:

system: |-
You grade whether a support answer follows from its sources.
Rubric:
{{rubric}}
user: |-
<question>
{{request}}
</question>
<sources>
{{sources}}
</sources>
<answer>
{{answer}}
</answer>
Reply with the JSON object only.
from trialkit.judge import metric_judge
from trialkit.judge.metrics import faithfulness_v2
from trialkit.judge.yaml import read_prompt
faithfulness = metric_judge(
faithfulness_v2,
model=model,
rubric="Treat a price as a factual claim.",
prompt=read_prompt("evals/faithfulness-prompt.yaml"),
)
  • Write each text as a block scalar and use |-: the text ends at its last character. | adds a line break at the end, and the prompt hash changes with it. Indent with spaces: a tab character anywhere in the file is a YAML problem.
  • The file has the keys system and user, and no other. Both are strings that are not blank. A key that is neither raises TypeError that names it (the first one in UTF-16 order), so a misspelled usr is an error, and so is a whole template file. read_prompt does not check the placeholders, because they depend on the metric: metric_judge checks them when it makes the judge.
  • A prompt that is not valid raises TypeError with <path>: before its message, for example evals/faithfulness-prompt.yaml: prompt.user must be a nonblank string, got undefined. A YAML problem raises TypeError with <path>:<line>: <reason>, by the rules of the template files. read_prompt reads the file with read_yaml_mapping of trialkit-cases, so install the extra: pip install 'trialkit-judge[yaml]'. A missing package raises ImportError with the install command.

The prompt is a new measurement:

  • data["judge"]["template"] and the scorer version stay the metric (faithfulness@2), and data["judge"]["promptSource"] is "override". data["judge"]["promptHash"] is the hash of the system prompt that was sent, as for every judge. It does not cover user, so an override also records data["judge"]["userTemplateHash"], the hash of its user text, as one of the settings: settingsHash covers it, and two prompts that differ in user alone are two judges. A built-in judge records none. settingsHash does not cover the source. A prompt that has the text of the template has the hash of the built-in judge, and another source and settings hash.
  • Calibrate the judge that has your prompt: the calibration of the built-in prompt says nothing about it. calibrate reports judge.prompt_source ("override" for your prompt), and it raises TypeError for a mix of built-in and override judges.
  • trialkit compare warns when the source differs (judge faithfulness prompt source: built-in → override), and trialkit report shows the judge as faithfulness@2 · override sha256:…. A history from before promptSource is built-in.

read_template(path) of trialkit.judge.yaml reads a template from a YAML file and returns what load_template makes of it. The path can be a string or a path-like object. It reads the file with read_yaml_mapping of trialkit-cases, so install the extra: pip install 'trialkit-judge[yaml]'. A missing package raises ImportError with the install command.

The file has the fields of JudgeTemplate, and it is the same file as in TypeScript: id, version, description, system, user, output and schema, and for a metric template evidence, rubric and, for choice, choices and labels. A metric of your own, in evals/polite-answer.yaml:

id: polite-answer
version: "1"
description: Checks that the answer is polite.
output: verdict
evidence: [request, answer]
rubric: optional
system: |-
You are an impartial grader of politeness. You do not write or improve the evidence.
Metric definition:
The answer is polite. Rude or dismissive wording fails.
Evidence:
- <request>: what the user asked.
- <answer>: the answer to assess.
Application rubric:
{{rubric}}
Reply with a JSON object containing exactly "reason" (a string) and "pass" (a boolean).
user: |-
<request>
{{request}}
</request>
<answer>
{{answer}}
</answer>
Assess the evidence. Reply with the JSON object only.
schema:
type: object
properties:
reason: { type: string }
pass: { type: boolean }
required: [reason, pass]
additionalProperties: false
from trialkit.judge import Metric, metric_judge
from trialkit.judge.yaml import read_template
polite = Metric(read_template("evals/polite-answer.yaml"))
politeness = metric_judge(polite, model=model)

To keep a built-in metric and change only its text, write a prompt file, which needs none of the template fields, and read it with read_prompt: Replace the prompt.

  • A YAML problem raises TypeError with <path>:<line>: <reason>: the rules are those of the case files (the 1.2 core schema, one document, unique keys, no aliases, tags or timestamps, at most 64 levels). A template that load_template refuses raises TypeError with <path>: before its message. That includes a key that is not a template field, such as a misspelled lables.
  • Write version: "1" in quotes. YAML reads 1 as a number and 2025-11-15 as a timestamp, and both fail. Use |- for text that ends at its last character: | adds a line break at the end, and the prompt hash changes with it.
Export Options and defaults Meaning
JudgeTemplate(id, version, description, system, user, output, schema, requires=(), labels=(), evidence=None, rubric=None, choices=None) First seven required A versioned template; optional requires=("expected",) requires a nonblank reference, and labels fixes classifier labels and order. A metric template also has evidence (its fields), rubric ("optional" or "required") and, for choice output, choices (label scores); see the template contract.
load_template(data) Required Validate a template mapping or JudgeTemplate. A key that is not a field of the template, such as a misspelled lables, is refused.
read_template(path) (trialkit.judge.yaml) Required Read a template from a YAML file and check it with load_template; needs the yaml extra. See Template files.
read_prompt(path) (trialkit.judge.yaml) Required Read the system and user text of a prompt from a YAML file, as a dict[str, str] for the prompt of metric_judge; needs the yaml extra. See Replace the prompt.
rubric_v1, rubric_v2, classifier_v1, classifier_v2, trajectory_v1 Built-in templates Shared specification versions.
Conversation, ConversationTurn Types only The wire shape of the evidence of a conversation metric; see Conversation metrics.
Metric(template), Tool; the twenty-one _v2 metric constants are in trialkit.judge.metrics template required A metric template for metric_judge; see Metric judges. Metric(load_template(data)) wraps a template of your own.
retrieval_turns(conversation, references=None), tool_decisions(conversation, spans), RetrievalTurn, ToolDecision conversation; spans for the second The evidence of each assistant decision of a conversation, for the metrics that grade one decision; see Projection helpers.
calibrate(judge, labeled, *, concurrency=4, trial_timeout_ms=120_000) judge and labeled required Async calibration against human labels; cancel the awaiting task to stop it.
LabeledOutput(eval_case, output, label, spans=None) First three required Optional Sequence[Span] evidence is replayed through ctx.span in record order before scoring, with core validation and snapshots.
CalibrationReport Returned by calibrate Has judge, n, agreement, agreement_interval, kappa, confusion, results, disagreements, errors and usage.
CalibrationResult(case_id, human, judge), Disagreement(case_id, human, judge, reason=None) As shown results has every scored case, in the order of labeled; a disagreement is a result where the labels differ, with the judge’s reason.
CalibrationError(case_id, message) As shown A case that the judge did not score.
JudgeIdentity(template, model, prompt_hash, settings_hash, prompt_source) As shown The judge that gave the verdicts, as their scores record it in data["judge"].

Built-in templates and their nested schemas are immutable. load_template checks that the schema describes exactly the reply fields of its output kind. Compare model, template, prompt hash, prompt source and settings hash before comparing verdicts.

agreement_interval is Interval(low, high): the 95% Wilson interval of agreement over the n scored cases, from wilson of trialkit.core. 6 of 6 agree gives 0.61 to 1, so a small set cannot show 100%. A case in errors is in neither n nor the interval. With no scored case the interval is 0 to 1, so a gate on report.agreement_interval.low fails when every case errored.

judge is a JudgeIdentity, as the scores of the n verdicts record it in data["judge"] (template, model, promptHash, settingsHash and promptSource). prompt_source is "built-in", or "override" for a metric_judge with a prompt. A change of the rubric, label rule, temperature, template, prompt or model changes it. The model that replied does not. Compare it with data["judge"] of the scores of an eval to check that they come from the calibrated configuration. judge is None when n is 0, and when the scorer records no judge. A scorer of your own gets the four string fields reported when it puts them in data["judge"]; its prompt_source is "built-in" unless it records "override". (The judge of a CalibrationResult is a label, not this.)

calibrate refuses a mix of judges: it raises TypeError when the scores of the verdicts differ in template, model, promptHash, settingsHash or promptSource, or when only some of them record data["judge"]. The message names the first field that differs and the two cases. A case with no verdict is not compared. The check follows the judge calls, so a rejected calibration has used the model.

knowledge_retention_v2, role_adherence_v2, topic_adherence_v2, turn_relevance_v2, completeness_v2 and goal_accuracy_v2 are metric judges of a conversation. Their evidence is a conversation and, for the first four, an optional target. They are trialkit definitions, not DeepEval-compatible scores.

Conversation and ConversationTurn are TypedDicts with the same wire keys as TypeScript. Every turn has a unique nonblank id, a role ("user", "assistant" or "tool") and string content. Assistant decisions can include ordered retrievedContexts, toolCalls (id, name, arguments) and availableTools (name, description, inputSchema). A tool result needs callId for an earlier call and status ("ok" or "error"). Any turn can link spanIds. Call IDs are unique, apart from turn IDs. Menus and sources are snapshots, also empty lists; nothing pools them. The judge refuses unknown fields, a value that is not JSON (a tuple, a key that is not a string) or that canonical JSON cannot write, an empty transcript, no assistant decision, and an assistant decision without a user turn before it, as misuse that names the cause: metric_judge: goal-accuracy@2: evidence.conversation must be a valid conversation: turn ids must be unique for case c1, got {"turns":[.... It shows each turn with its 1-based number, id, role, content and the calls and results that it has, and no menu, snapshot or span ID.

from trialkit.judge import metric_judge
from trialkit.judge.metrics import completeness_v2, knowledge_retention_v2
from trialkit.pytest import AsyncRunFixture, AsyncScoreFixture
completeness = metric_judge(completeness_v2, model=model)
retention = metric_judge(knowledge_retention_v2, model=model)
async def test_agent_conversation(run: AsyncRunFixture, score: AsyncScoreFixture) -> None:
# The conversation that the agent ended is the evidence.
result = await run({"conversation": seed})
await score(completeness, {"conversation": result["conversation"]})
# One reply of it: the conversation ends at the target turn, and only that reply is graded.
await score(retention, {"conversation": result["conversation"], "target": "a2"})

Put the seed prefix in input["conversation"] and return the complete transcript in output["conversation"]. Without a target, knowledge_retention_v2, role_adherence_v2, topic_adherence_v2 and turn_relevance_v2 grade every assistant reply against its full preceding turns, and a later correction cannot erase an earlier violation. With a target, the conversation ends at that turn and only that reply is graded: every earlier turn, also an earlier assistant reply, is context, and no later turn reaches the model. A target that no turn has, or that names a user or tool turn, is misuse, and so is a bad turn after the target. completeness_v2 and goal_accuracy_v2 judge the conversation through its end and read no target; a target key in their evidence is ignored. In the definition form, your function gives the target of each case:

from trialkit.judge import metric_judge
from trialkit.judge.metrics import knowledge_retention_v2
retention = metric_judge(
knowledge_retention_v2,
model=model,
evidence=lambda case, output: {
"conversation": output["conversation"],
"target": case.input["targetTurnId"], # None grades every reply
},
)

A version 1 single-reply recipe took a history, the current user turn and the reply. Give the same conversation instead: the history, the user turn and the reply as an assistant turn, with that turn as the target. knowledge_retention_v2 fails a needless re-ask of a fact that the user gave, for a targeted reply too.

Empty answers fail: an assistant turn that only calls tools is a decision, not an empty answer, and any other empty assistant reply fails. Tool turns are evidence, not user goals or policy. A topic change does not cancel a request; only an explicit cancellation removes the obligation. Promises and pending clarification are incomplete unless resolved by the end; a justified refusal can close a request without proving goal achievement. Executing an explicitly cancelled action afterward fails completeness, and an acknowledged stop request with observed cessation can be resolved. Policy belongs in the trusted rubric, not in the transcript. Goal accuracy is separate from completeness: a success claim does not prove an external action, and a justified refusal does not achieve the goal. Explicit corrections replace affected goals. Cancellation removes a goal but forbids executing the cancelled action later; a stop can succeed through acknowledgement and observed cessation. Set a size limit of your own and refuse oversized evidence without trimming.

retrieval_turns(conversation, references=None) and tool_decisions(conversation, spans) make the evidence of each assistant decision of a conversation. Give each result to a metric as it is, and keep each verdict apart: do not average them, or combine a decision with plan quality or adherence.

Function Returns Refuses (misuse)
retrieval_turns(conversation, references=None) One RetrievalTurn (a dict) for each assistant turn that has a retrievedContexts snapshot, also an empty one: turnId, request (the dialogue before the turn), answer (its content) and sources (its own snapshot). With references, a reference too. An invalid conversation, no turn with a snapshot, references that is not a mapping with string keys, and a retrieval turn whose reference is not a nonblank string in references.
tool_decisions(conversation, spans) One ToolDecision (a dict) for each assistant turn, also one without a call: turnId, request (the dialogue before the turn), tools (its availableTools) and spans (the spans that the turn and the results of its calls map, in record order). An invalid conversation, spans that are not a nonempty valid list, a turn without its menu, mapping or spans, a span that two turns map, and a call whose span is no execute_tool span of the same tool with the same arguments.

references is the factual reference answer of each turn, by turn ID, not a human label. A turn without a snapshot is not a retrieval turn. No earlier snapshot is pooled and no later turn is shown: the evidence of a decision depends on its prefix alone. A span keeps its parent only when the parent is one of the decision’s spans. Arguments of a call and its span compare as canonical JSON, so key order does not matter and 1 is not true; arguments that canonical JSON cannot write, such as a string with a lone surrogate, are no match. The conversation projection contract has every message.

The ranking and coverage metrics read reference: pass references to retrieval_turns. tool_appropriateness_v2 reads request and tools, and it grades the spans of its trial, not the spans key of the evidence. Grade each decision in a trial of its own: in one trial, the spans of the other decisions would be graded too. The task records decision["spans"] with ctx.span, and calibrate replays them from LabeledOutput(..., spans=decision["spans"]).

from trialkit.core import Dataset, EvalCase, EvalDefinition, TrialContext
from trialkit.judge import ToolDecision, metric_judge, retrieval_turns, tool_decisions
from trialkit.judge.metrics import faithfulness_v2, tool_appropriateness_v2
# One faithfulness verdict for each retrieval turn: the output of the task is the evidence.
turn_faithfulness = EvalDefinition(
name="turn-faithfulness",
dataset=Dataset(
cases=[EvalCase(turn["turnId"], turn) for turn in retrieval_turns(conversation)]
),
task=lambda turn: turn,
scorers=[metric_judge(faithfulness_v2, model=model)],
)
# One tool-use verdict for each decision: the task records the spans of its decision.
async def record_spans(decision: ToolDecision, ctx: TrialContext) -> ToolDecision:
for span in decision["spans"]:
ctx.span(span)
return decision
tool_use = EvalDefinition(
name="tool-use",
dataset=Dataset(cases=[EvalCase(d["turnId"], d) for d in tool_decisions(conversation, spans)]),
task=record_spans,
scorers=[metric_judge(tool_appropriateness_v2, model=model)],
)

The person-run calibration example has opt-in paid commands for TypeScript and Python. Freeze before holdout; require full n and zero errors, and inspect every disagreement. Its tiny fictional MIT corpus is smoke calibration, not universal accuracy. Calibrate on application-held-out data before gating. calibrate excludes errors from n, agreement, kappa and confusion; it does not count them as negative labels. Keep report.judge with a report: it has the template, the model, the prompt hash and the settings hash. Missing spans remain valid for judges that do not grade trajectories. Select only the stated text fields of the evidence, not a whole application object, and redact conversations, sources and spans before sending them to a model. Reports can contain judge reasons: review them before sharing.

Export Options and defaults Meaning
ModelClient id, async generate(request) Model port.
ModelMessage(role, content) Both required Message role is system, user or assistant.
ModelRequest(messages, json_schema=None, temperature=None, max_output_tokens=None) messages required Model request.
ModelReply(text, model, json=None, usage=None) text and model required Reply and optional input_tokens, output_tokens, cached_input_tokens, reasoning_tokens, cost usage.
from_function(id, generate) Both required Wrap a plain or async model function.
openai_model(*, base_url, api_key, model, id=None, max_output_tokens_param="max_completion_tokens", request_timeout_ms=120_000) base_url, api_key, model required OpenAI-compatible Chat Completions client. Default id is <provider>:<model> from the host; use max_output_tokens_param="max_tokens" when required.

The HTTP client follows no redirects, hides the API key from repr and uses urllib proxy settings. It retries connection errors and HTTP 408, 409, 429 and 5xx twice, like the OpenAI SDK default. It honors Retry-After (up to 60 seconds); otherwise it uses bounded exponential backoff. It never retries an unreadable reply or a quality verdict. The request deadline covers all attempts and delays, and cancellation stops retrying. Python cannot force-stop a blocked daemon worker; its socket timeout limits inactivity. Empty replies name a refusal or finish reason when present, and a missing model is an infrastructure error. IP-literal server ids keep the whole address and explicit port.

import asyncio
from trialkit.cases import write_cases
from trialkit.core import read_cases
from trialkit.judge.generate import generate_cases
result = asyncio.run(
generate_cases(model=model, source={"cases": read_cases("seeds.jsonl")}, count=12, seed=7)
)
write_cases("draft.yaml", result.rows)

generate_cases asks a model once for new eval cases and returns them for a person to review. It behaves as generateCases in TypeScript; read that page for the rules. All options are keywords.

Option Default Meaning
model Required A ModelClient; each row records its id.
source Required Exactly one of {"cases": [EvalCase, ...]}, {"documents": [{"id": ..., "text": ...}]} or {"description": "..."}. Each is non-empty.
count Required Integer from 1 to 100: the candidates to ask for. The reply must have exactly this many.
existing None EvalCase values or rows that the new rows must not repeat. Never sent. Seed cases count as existing.
seed 0 Integer from 0 to 4294967295 that the prompt shows: a nonce, not a seed for the model.
max_output_tokens 4096 Integer from 1 to 32768, sent to the model.
temperature None Number from 0 to 2. None sends no temperature.
request_timeout_ms 120_000 Number above 0 and at most 2147483647. At the deadline the call is cancelled, and a late reply is ignored.

It returns GeneratedCases(rows, discarded, usage): EvalCase rows with the id generated-<sha256>, the tag generated and the provenance metadata, the number of dropped candidates, and the usage of the reply (None when unknown, never zero). Invalid options raise TypeError or ValueError before the call; a prompt of more than 64 KiB raises ValueError there too. A failed call, an unreadable reply (a reply text or a structured output of more than 64 KiB is one; each is checked on its own), an invalid candidate and a reply with no new case raise InfrastructureError with the cause. Cancellation of the task, KeyboardInterrupt and SystemExit pass through. Generation calls the model once, with no reply repair or cache; the HTTP client owns transient transport retries.

from trialkit.judge import trajectory_judge
process = trajectory_judge(
model=model,
rubric="Look up the returns policy before answering. Use the captured result in the answer.",
)

The canonical JSON transcript preserves span order, ids, parent ids and status. It includes only gen_ai.operation.name, gen_ai.request.model, gen_ai.response.model, gen_ai.system_instructions, gen_ai.input.messages, gen_ai.output.messages, gen_ai.tool.name, gen_ai.tool.call.id, gen_ai.tool.call.arguments and gen_ai.tool.call.result. It omits times and unknown attributes. Missing or unrenderable spans are error before a model call; model-only spans are valid. Missing arguments differ from explicit None. Sequence is not proof the agent used a result.

This judge sends captured messages, arguments and results to the model. Redact secrets before ctx.span. Use tool_calls and tool_policy for deterministic requirements and limits.