trialkit-judge
Model-graded scorers use versioned prompts and keep unreadable replies and failed model calls out of quality pass rates. Calibrate a judge against human labels before relying on its verdicts.
Install
Section titled “Install”pip install trialkit-judgepip install 'trialkit-judge[yaml]' # also install trialkit-cases and its YAML parser, for read_template and read_promptExample
Section titled “Example”from trialkit.judge import ModelReply, from_function, rubric_judge
model = from_function( "fake:judge", lambda request: ModelReply(text='{"reason":"It names the limit.","pass":true}', model="fake-1"),)judge = rubric_judge(model=model, rubric="The answer says refunds are possible within 30 days.")- Import
trialkit.judge. The package depends ontrialkit-coreand uses the standard library for HTTP. - An unreadable reply or failed model call raises
InfrastructureError, so the trial iserror, notfail. - Behavior matches the shared TypeScript judge; Python options are keyword arguments in
snake_case.
Judges
Section titled “Judges”All factory options are keyword-only, except the metric of metric_judge. input and output default to the case input and task output as text; non-strings become canonical JSON. None or a value that core cannot render is misuse: add an explicit picker. A picker must return a string. expected=None sends a string case reference only; a fixed string or a picker of the case’s expected value overrides it. Missing or non-string references are (none), unless template metadata requires a nonblank reference. context(case, output) is absent by default and requires a template with {{context}}, and vice versa. An expected option without {{expected}} is refused. temperature=None sends no temperature; explicit values must be 0–2.
| Export | Options and defaults | Meaning |
|---|---|---|
rubric_judge |
model, rubric required; name="rubric", template=None, input=None, output=None, context=None, expected=None, temperature=None |
Binary scorer. Uses rubric_v1, or rubric_v2 with context. |
classifier_judge |
model, rubric, choices required; name="classifier", template=None, input=None, output=None, context=None, expected=None, temperature=None, pass_label=None |
choices maps labels to scores 0–1 in insertion order. Unsigned integer-like labels such as "1" are refused. pass_label is resolved once per label at creation; a thrown error or non-boolean result is rejected. |
trajectory_judge |
model, rubric required; name="trajectory", template=None, input=None, expected=None, temperature=None |
Grades recorded spans, not the final output. input selects what the model sees. Explicit templates need verdict output and no context placeholder. |
metric_judge |
metric, model required; evidence=None, rubric=None, name=None, temperature=None, pass_label=None, prompt=None |
Scorer for one metric: binary, or with labels. prompt replaces its system and user text (Replace the prompt). It has no template, input, output, context or expected option. |
Scorer |
Re-export from trialkit.core |
Protocol of each returned scorer. |
Rubric, trajectory and metric replies must contain exactly reason and boolean pass; classifier and label-metric replies must contain exactly reason and a known label. One surrounding json fence is accepted. A structured ModelReply.json dict takes precedence over text. Reply and rubric whitespace follows ECMAScript, not str.strip(): U+FEFF is whitespace, U+0085 and U+001C–U+001F are not. A classifier refuses a label that core cannot write as JSON, and it asks pass_label after every other check. Scores record model, replyModel, template, promptHash, promptSource, temperature (None when unset), evidence and settingsHash in data["judge"]. promptSource is built-in for these judges: only metric_judge takes a prompt that replaces the text of its template. evidence is canonical-json: how a non-string input or output without a picker becomes text. metric_judge records fields@1, the name of its evidence rules. Classifiers and label metrics also record the choices and passByLabel (whether each label passes). The settings hash covers the canonical JSON of temperature, evidence and, for a classifier or a label metric, choices and passByLabel, and for an override userTemplateHash, the hash of its user text. Runs recorded before evidence was recorded are not comparable with later runs: they wrote a non-string input or output as JSON in insertion order, and their settings hash is missing or does not cover evidence. Treat the first run after the change as a new baseline. Usage is added through ctx.add_usage before parsing.
Metric judges
Section titled “Metric judges”A metric names one property to grade, such as faithfulness. It fixes the evidence fields, how each field is rendered, and the reply. You give the evidence. You write no prompt slots, no wire format and no filler rubric.
In the test form, the evidence is the output that you give to score:
from trialkit.judge import metric_judgefrom trialkit.judge.metrics import faithfulness_v2from trialkit.pytest import AsyncScoreFixture
faithfulness = metric_judge(faithfulness_v2, model=model)
async def test_answer_is_faithful(score: AsyncScoreFixture) -> None: await score( faithfulness, {"request": question, "answer": reply["text"], "sources": reply["passages"]}, )In the definition form, evidence(case, output) makes the evidence of each case. This one takes the reference from the expected of the case:
from typing import Any
from trialkit.core import EvalCasefrom trialkit.judge import metric_judgefrom trialkit.judge.metrics import context_ranking_v2
def ranking_evidence(case: EvalCase[Any, Any], output: Any) -> dict[str, object]: assert case.expected is not None # the judge calls this function only for a case that has one return { "request": case.input["question"], "sources": output["passages"], "reference": case.expected["referenceAnswer"], }
ranking = metric_judge(context_ranking_v2, model=model, evidence=ranking_evidence)A metric with a reference calls your function only for a case that has expected. A type checker still sees case.expected as optional, so a typed def asserts that it is there, as above.
A metric with labels scores the label of the reply. pass_label says which labels pass. Without it, a label passes at a score of at least 0.5.
from trialkit.judge import metric_judgefrom trialkit.judge.metrics import context_sufficiency_v2
# Fail only when the sources contradict each other.sufficiency = metric_judge( context_sufficiency_v2, model=model, pass_label=lambda label: label != "conflicting")A metric of the agent process also grades the spans that the trial recorded. You do not pass them: the task records them with ctx.span, and a test body can record them before score. The judge reads them from the trial, so a trial without spans ends in error before the model call.
from trialkit.judge import metric_judgefrom trialkit.judge.metrics import task_completion_v2, tool_appropriateness_v2from trialkit.pytest import AsyncScoreFixture
# Test form: the evidence is the output that you give to `score`.task_completion = metric_judge(task_completion_v2, model=model)
async def test_booking_is_complete(score: AsyncScoreFixture) -> None: await score( task_completion, {"request": task, "successCriteria": ["The booking service confirms room Iris."]}, )
# Definition form: your function gives the evidence, and the trial gives the spans.tool_use = metric_judge( tool_appropriateness_v2, model=model, evidence=lambda case, _output: {"request": case.input["task"], "tools": case.input["tools"]},)| Option | Default | Meaning |
|---|---|---|
metric |
Required | A constant from the table below, or Metric(load_template(data)) for a template of your own. The one positional argument. |
model |
Required | A ModelClient. |
evidence |
None |
A plain function (case, output) that returns a mapping with the fields of the metric. Without it, the task output is that mapping. |
rubric |
None |
Your constraints for this metric. The prompt shows (none) without it. A blank string is refused. A metric can require it: instruction_following_v2, role_adherence_v2 and topic_adherence_v2. |
name |
The metric id | The scorer name. Scorer names in one eval must differ. |
temperature |
None |
None sends no temperature. A number sets it, from 0 to 2. |
pass_label |
None |
For a metric with labels only. A function from a label to a boolean. It runs once for each label when the judge is made, after every other check, and a non-boolean result is refused. |
prompt |
None |
A mapping with the keys system and user. It replaces the system and user text of the template of the metric, and the rest stays the metric’s. A prompt that breaks the placeholder rule is refused when the judge is made. See Replace the prompt. |
Each field of the evidence has one rule:
| Field | Valid value | Rendered as |
|---|---|---|
request |
A nonblank string | The text, unchanged: what the user asked, or the dialogue before the reply. |
answer |
A string. An empty or blank answer is valid evidence. | The text, unchanged: the answer, summary or response to grade. |
sources |
A list or tuple of strings. An empty list is valid. | Canonical JSON [{"id":1,"text":"..."}] in order. Blank and duplicate entries keep their own positions and IDs. |
reference |
A nonblank string | The text, unchanged: the factual reference answer, not a human label. |
tools |
A list or tuple of mappings. Each has a nonblank string name, a string description and a mapping inputSchema that JSON can hold. An empty list is valid. |
Canonical JSON [{"description":...,"inputSchema":...,"name":...}] in order, with these three keys of each tool and no other. A duplicate keeps its place. |
successCriteria |
A nonempty list or tuple of nonblank strings | A numbered list, one line for each criterion: 1. first, 2. second. The text of a criterion is kept. |
plan |
A string. An empty or blank plan is valid evidence. | The text, unchanged: the public plan that the agent made before it acted. |
planSpanId |
A nonblank string | The text, unchanged: the ID of the span that holds that plan. |
conversation |
A Conversation: a dict with a nonempty turns list, as Conversation metrics describes. |
Canonical JSON {"turns":[...]}. Each turn has id, role, content and, when it has them, toolCalls, callId and status, with its 1-based number. Menus, retrieval snapshots and span IDs are not sent. With a target, the turns end at the target turn. |
target |
A turn ID, or nothing: leave it out, or give None. |
The ID as a JSON string ("a2"), or null when it is not given. |
- A metric reads only the fields that it names, in its own order. A key that it does not name is ignored and never sent, so one mapping can serve several metrics.
- Bad evidence is misuse: a field that is missing or
None(atargetcan be both), a value of the wrong kind, a blankrequest,referenceorplanSpanId, asourcesitem that is not a string or that has a lone surrogate, atoolsitem that is not a mapping or whosename,descriptionorinputSchemais not as the table says, an emptysuccessCriterialist or a blank criterion, and evidence that is not a mapping. The trial ends inerrorbefore the model call. The message names the metric, the place, the case and the value, for examplemetric_judge: faithfulness@2: evidence.sources must be an array of strings for case c1, got "Pro.". The first problem in the order of the metric’s fields wins: insidetools, the items in order, and in each toolname,description, theninputSchema. Blank means ECMAScript whitespace: U+FEFF is blank and U+0085 is not. - An empty
answer, an emptysourceslist, an emptytoolslist and a blankplanare valid evidence, and the metric judges them by its own rules. An empty answer failsfaithfulness_v2, and a blank plan failsplan_quality_v2. - A metric reads
conversationandtargettogether, at the place ofconversation. A badconversationis misuse that names its rule aftera valid conversation:, for exampleturn ids must be unique. The metric checks the whole conversation, also what it does not send, and JSON must be able to write it. Without atarget(a missing key orNone), the metric grades every assistant reply. Any othertargetmust be astrthat is the ID of an assistant turn of the conversation. A number, a list,"", an unknown ID and the ID of a user turn or a tool turn are misuse, for examplemetric_judge: knowledge-retention@2: evidence.target must be the ID of an assistant turn of evidence.conversation for case c1, got "u2". A badconversationis reported before a badtarget. A metric that has notargetignores atargetkey. - A metric that names
spans(the table lists them) reads the spans that the trial recorded. They are not a key of the evidence, and aspanskey there is ignored. The prompt shows them astrajectory_judgedoes (see the trajectory judge): record order, span IDs, parent IDs and status, only the selectedgen_ai.*attributes, no times. A trial with no span is misuse:metric_judge: task-completion@2: recorded spans are required.spansis the last field of each of these metrics, so a bad evidence field is reported before missing spans. - With
evidence, a case whoseexpectedis absent orNoneis misuse when the metric has areferencefield, and the function is not called. Withoutevidence, the output holds thereference. - The prompt and the settings are recorded as for the other judges.
evidencein the settings isfields@1.rubricis part of the system prompt, so it changespromptHash.promptSourceisbuilt-in, oroverridewith aprompt, which also recordsuserTemplateHash. A model call that fails or an unreadable reply raisesInfrastructureError. rubric_judge,classifier_judgeandtrajectory_judgerefuse a metric template at creation and namemetric_judge.- The constants are in
trialkit.judge.metrics, not intrialkit.judge: a set that grows has its own module, and the root keeps the judges and the five generic templates. The module is generated from the template files (pnpm generate:templates).
| Constant | Evidence, in prompt order | Passes when |
|---|---|---|
faithfulness_v2 |
request, sources, answer |
Every factual claim in the answer follows from the sources. A nonempty explicit abstention can pass. |
citation_grounding_v2 |
request, sources, answer |
Every factual claim follows from the sources, and every standalone [N] citation names a source that supports its claim. |
context_alignment_v2 |
request, sources, answer |
The answer contradicts none of the curated ideal contexts in sources. An unsupported addition can pass. |
context_relevance_v2 |
request, sources |
Every substantive statement in the sources helps to answer the request or gives necessary background. |
context_ranking_v2 |
request, reference, sources |
At least one source supports a relevant fact of the reference, and every relevant source comes before every irrelevant one. |
reference_coverage_v2 |
request, reference, sources |
Together the sources support every factual assertion of the reference, with no unresolved contradiction. |
context_sufficiency_v2 |
request, sources |
Labels sufficient (score 1), insufficient (0) and conflicting (0) from the sources alone. It grades no answer. |
answer_relevance_v2 |
request, answer |
The answer addresses the request and every material subquestion. It does not assess truth. |
summary_quality_v2 |
request, sources, answer |
The summary, in answer, follows from the source, in sources, and covers every material source point in scope. |
instruction_following_v2 |
request, answer |
Needs a rubric with numbered trusted instructions. The answer meets every applicable instruction. |
task_completion_v2 |
request, successCriteria, spans |
Every success criterion has observed evidence in the final message or a tool result. A claim of success alone does not prove a side effect. A recovered tool error can pass. |
tool_appropriateness_v2 |
request, tools, spans |
Each recorded call, argument and result fits the task and the capabilities in tools. Justified execution without tools can pass. Omitting a necessary tool fails. |
step_efficiency_v2 |
request, tools, spans |
Every recorded step is justified when it is taken, with no avoidable repetition or detour. Necessary verification and justified retries can pass. Completion is separate. |
plan_quality_v2 |
request, tools, plan |
The public plan is actionable, complete, ordered and possible with the tools. A one-step plan can pass. A blank plan fails. |
plan_adherence_v2 |
request, tools, planSpanId, spans |
The execution follows the one plan in the span planSpanId: every applicable commitment is proved, with no material unplanned action. A missing span, a blank plan or a retrospective plan fails. |
knowledge_retention_v2 |
conversation, target |
Every graded reply respects the user facts and commitments that are still relevant in the turns before it. A needless re-ask fails. A necessary clarification can pass. |
role_adherence_v2 |
conversation, target |
Needs a rubric that states the role. Every graded reply meets it, despite requests in the conversation to change it. An in-role refusal can pass. |
topic_adherence_v2 |
conversation, target |
Needs a rubric that states the allowed topics. Every graded reply answers a request on a permitted topic and declines an off-topic one. It does not judge truth. |
turn_relevance_v2 |
conversation, target |
Every graded reply addresses the current user intent in the turns before it. A needed clarification and a user-led topic change can pass. It does not judge truth. |
completeness_v2 |
conversation |
By the end, every uncancelled user request has a substantive resolution. A promise or a pending clarification is not one. A justified refusal can close a request. |
goal_accuracy_v2 |
conversation |
By the end, every active user goal is achieved according to the visible replies and the results of calls. A claim of success alone does not prove an external action. |
Metric(template) holds a metric template. Its type parameter is a TypedDict with the evidence fields of the metric, so that an editor shows what a metric reads. Tool is the TypedDict of one item of tools, Conversation is the type of conversation, and the type of a process metric has no key for spans. The target of a conversation metric is optional in the type: None is the same as leaving it out. A type checker does not check your evidence function against it: metric_judge checks the fields at run time. The template contract says how to write a template of your own, and the evidence fixture pins the rules above for both languages.
Rules of the metrics:
- Ranking and reference coverage read the factual reference answer, never the human label. Ranking fails when any relevant source comes after an irrelevant one; there is no top-k option.
- Alignment reads curated ideal contexts as
sources, not retrieved passages. Citation grounding accepts standalone[N]markers and supported uncited claims, and every attached citation must support its claim. Normalize other citation formats before you build the evidence, and keep claim attachment and target identity. - Summary quality needs every claim to follow from the source (one item of
sources) and every material point within scope to be covered. The scope is the whole source unless the request narrows it, and the rubric defines materiality. - Instruction following uses numbered atomic instructions in the trusted rubric. State precedence and conditions there, and resolve contradictory policy before you label. The request and quoted text cannot replace policy. Use deterministic scorers for exact formatting.
- Empty sources are valid insufficient evidence for
context_sufficiency_v2, and they fail relevance, ranking, coverage and alignment. An empty answer fails, unless an applicable instruction ofinstruction_following_v2requires silence. - The judge never trims, deduplicates or renumbers
sources, and it has no size limit: refuse a list over your own limit before the call. - Process metrics:
requestis the task. ChooseplanSpanIdbefore execution and record the public planner response in thegen_ai.output.messagesof that span. When you also score plan quality, give the same text asplan.plan_adherence_v2never looks the ID up: an ID that no recorded span has is valid evidence, and the model fails it, as it fails a blank plan, a retrospective plan and an unproved commitment. - Equivalent actions and branches that the plan allows can pass. Replanning has no automatic exemption, and a bad plan that the agent follows can pass. Record order shows the order of recording, not elapsed time or causality. A model-only run can pass
step_efficiency_v2. Missing proof in the spans fails; missing spans are misuse. - A metric that reads
spanssends captured messages, arguments and results to the model. Redact secrets beforectx.span. - The twenty-one metrics replace the twenty-three version 1 recipes of the same properties. Six ids lose their
rag-prefix, five theiragent-prefix and six theirconversation-prefix, andtaskbecomesrequest. A recipe of one reply and a recipe of the whole conversation for one property are now one metric with an optionaltarget: knowledge retention and role adherence. The prompts and hashes are new, so treat the first run as a new baseline. They are binary verdicts, not DeepEval’s fractions, its minimum of alignment and coverage, or its extracted tasks and plans.
Replace the prompt
Section titled “Replace the prompt”prompt={"system": ..., "user": ...} replaces the system and user text of the template of the metric. It is a mapping with these two keys (a dict or any other Mapping), as in TypeScript, not a class. Use it when the built-in wording does not fit your domain and a rubric is not enough. Everything else stays the library’s: the evidence fields, their checks and their rendering, the reply schema, the strict reading of the reply, error against fail, the rubric (a required rubric is still required) and, for a metric with labels, the labels and their scores.
from trialkit.judge import metric_judgefrom trialkit.judge.metrics import faithfulness_v2
faithfulness = metric_judge( faithfulness_v2, model=model, rubric="Treat a price as a factual claim.", prompt={ "system": "You grade whether a support answer follows from its sources.\n\nRubric:\n{{rubric}}", "user": "<question>\n{{request}}\n</question>\n\n<sources>\n{{sources}}\n</sources>\n\n<answer>\n{{answer}}\n</answer>\n\nReply with the JSON object only.", },)The judge checks prompt when it is made, with the placeholder rule that load_template applies to a metric template:
systemhas{{rubric}}, and{{choices}}for a metric with labels.userhas the{{field}}of each evidence field of the metric (faithfulness_v2:request,sourcesandanswer). No other placeholder may appear. A placeholder may repeat, and the order is yours.systemanduserare strings that are not blank, and the mapping has no other key.Nonemeans no prompt.- A bad prompt raises
TypeErrorwith a message that names the function, the metric and the problem, for examplemetric_judge: faithfulness@2: prompt.user has no {{sources}} placeholder. The metric evidence contract lists every message.
For a long prompt, write the text in a YAML file and read it with read_prompt(path) of trialkit.judge.yaml. It returns a dict[str, str], which prompt takes as it is, so a type checker needs no cast. The path can be a string or a path-like object. This file, evals/faithfulness-prompt.yaml, holds the prompt of the example above:
system: |- You grade whether a support answer follows from its sources.
Rubric: {{rubric}}user: |- <question> {{request}} </question>
<sources> {{sources}} </sources>
<answer> {{answer}} </answer>
Reply with the JSON object only.from trialkit.judge import metric_judgefrom trialkit.judge.metrics import faithfulness_v2from trialkit.judge.yaml import read_prompt
faithfulness = metric_judge( faithfulness_v2, model=model, rubric="Treat a price as a factual claim.", prompt=read_prompt("evals/faithfulness-prompt.yaml"),)- Write each text as a block scalar and use
|-: the text ends at its last character.|adds a line break at the end, and the prompt hash changes with it. Indent with spaces: a tab character anywhere in the file is a YAML problem. - The file has the keys
systemanduser, and no other. Both are strings that are not blank. A key that is neither raisesTypeErrorthat names it (the first one in UTF-16 order), so a misspelledusris an error, and so is a whole template file.read_promptdoes not check the placeholders, because they depend on the metric:metric_judgechecks them when it makes the judge. - A prompt that is not valid raises
TypeErrorwith<path>:before its message, for exampleevals/faithfulness-prompt.yaml: prompt.user must be a nonblank string, got undefined. A YAML problem raisesTypeErrorwith<path>:<line>: <reason>, by the rules of the template files.read_promptreads the file withread_yaml_mappingoftrialkit-cases, so install the extra:pip install 'trialkit-judge[yaml]'. A missing package raisesImportErrorwith the install command.
The prompt is a new measurement:
data["judge"]["template"]and the scorerversionstay the metric (faithfulness@2), anddata["judge"]["promptSource"]is"override".data["judge"]["promptHash"]is the hash of the system prompt that was sent, as for every judge. It does not coveruser, so an override also recordsdata["judge"]["userTemplateHash"], the hash of itsusertext, as one of the settings:settingsHashcovers it, and two prompts that differ inuseralone are two judges. A built-in judge records none.settingsHashdoes not cover the source. A prompt that has the text of the template has the hash of the built-in judge, and another source and settings hash.- Calibrate the judge that has your prompt: the calibration of the built-in prompt says nothing about it.
calibratereportsjudge.prompt_source("override"for your prompt), and it raisesTypeErrorfor a mix of built-in and override judges. trialkit comparewarns when the source differs (judge faithfulness prompt source: built-in → override), andtrialkit reportshows the judge asfaithfulness@2 · override sha256:…. A history from beforepromptSourceisbuilt-in.
Template files
Section titled “Template files”read_template(path) of trialkit.judge.yaml reads a template from a YAML file and returns what load_template makes of it. The path can be a string or a path-like object. It reads the file with read_yaml_mapping of trialkit-cases, so install the extra: pip install 'trialkit-judge[yaml]'. A missing package raises ImportError with the install command.
The file has the fields of JudgeTemplate, and it is the same file as in TypeScript: id, version, description, system, user, output and schema, and for a metric template evidence, rubric and, for choice, choices and labels. A metric of your own, in evals/polite-answer.yaml:
id: polite-answerversion: "1"description: Checks that the answer is polite.output: verdictevidence: [request, answer]rubric: optionalsystem: |- You are an impartial grader of politeness. You do not write or improve the evidence.
Metric definition: The answer is polite. Rude or dismissive wording fails.
Evidence: - <request>: what the user asked. - <answer>: the answer to assess.
Application rubric: {{rubric}}
Reply with a JSON object containing exactly "reason" (a string) and "pass" (a boolean).user: |- <request> {{request}} </request>
<answer> {{answer}} </answer>
Assess the evidence. Reply with the JSON object only.schema: type: object properties: reason: { type: string } pass: { type: boolean } required: [reason, pass] additionalProperties: falsefrom trialkit.judge import Metric, metric_judgefrom trialkit.judge.yaml import read_template
polite = Metric(read_template("evals/polite-answer.yaml"))politeness = metric_judge(polite, model=model)To keep a built-in metric and change only its text, write a prompt file, which needs none of the template fields, and read it with read_prompt: Replace the prompt.
- A YAML problem raises
TypeErrorwith<path>:<line>: <reason>: the rules are those of the case files (the 1.2 core schema, one document, unique keys, no aliases, tags or timestamps, at most 64 levels). A template thatload_templaterefuses raisesTypeErrorwith<path>:before its message. That includes a key that is not a template field, such as a misspelledlables. - Write
version: "1"in quotes. YAML reads1as a number and2025-11-15as a timestamp, and both fail. Use|-for text that ends at its last character:|adds a line break at the end, and the prompt hash changes with it.
Templates and calibration
Section titled “Templates and calibration”| Export | Options and defaults | Meaning |
|---|---|---|
JudgeTemplate(id, version, description, system, user, output, schema, requires=(), labels=(), evidence=None, rubric=None, choices=None) |
First seven required | A versioned template; optional requires=("expected",) requires a nonblank reference, and labels fixes classifier labels and order. A metric template also has evidence (its fields), rubric ("optional" or "required") and, for choice output, choices (label scores); see the template contract. |
load_template(data) |
Required | Validate a template mapping or JudgeTemplate. A key that is not a field of the template, such as a misspelled lables, is refused. |
read_template(path) (trialkit.judge.yaml) |
Required | Read a template from a YAML file and check it with load_template; needs the yaml extra. See Template files. |
read_prompt(path) (trialkit.judge.yaml) |
Required | Read the system and user text of a prompt from a YAML file, as a dict[str, str] for the prompt of metric_judge; needs the yaml extra. See Replace the prompt. |
rubric_v1, rubric_v2, classifier_v1, classifier_v2, trajectory_v1 |
Built-in templates | Shared specification versions. |
Conversation, ConversationTurn |
Types only | The wire shape of the evidence of a conversation metric; see Conversation metrics. |
Metric(template), Tool; the twenty-one _v2 metric constants are in trialkit.judge.metrics |
template required |
A metric template for metric_judge; see Metric judges. Metric(load_template(data)) wraps a template of your own. |
retrieval_turns(conversation, references=None), tool_decisions(conversation, spans), RetrievalTurn, ToolDecision |
conversation; spans for the second |
The evidence of each assistant decision of a conversation, for the metrics that grade one decision; see Projection helpers. |
calibrate(judge, labeled, *, concurrency=4, trial_timeout_ms=120_000) |
judge and labeled required |
Async calibration against human labels; cancel the awaiting task to stop it. |
LabeledOutput(eval_case, output, label, spans=None) |
First three required | Optional Sequence[Span] evidence is replayed through ctx.span in record order before scoring, with core validation and snapshots. |
CalibrationReport |
Returned by calibrate |
Has judge, n, agreement, agreement_interval, kappa, confusion, results, disagreements, errors and usage. |
CalibrationResult(case_id, human, judge), Disagreement(case_id, human, judge, reason=None) |
As shown | results has every scored case, in the order of labeled; a disagreement is a result where the labels differ, with the judge’s reason. |
CalibrationError(case_id, message) |
As shown | A case that the judge did not score. |
JudgeIdentity(template, model, prompt_hash, settings_hash, prompt_source) |
As shown | The judge that gave the verdicts, as their scores record it in data["judge"]. |
Built-in templates and their nested schemas are immutable. load_template checks that the schema describes exactly the reply fields of its output kind. Compare model, template, prompt hash, prompt source and settings hash before comparing verdicts.
agreement_interval is Interval(low, high): the 95% Wilson interval of agreement over the n scored cases, from wilson of trialkit.core. 6 of 6 agree gives 0.61 to 1, so a small set cannot show 100%. A case in errors is in neither n nor the interval. With no scored case the interval is 0 to 1, so a gate on report.agreement_interval.low fails when every case errored.
judge is a JudgeIdentity, as the scores of the n verdicts record it in data["judge"] (template, model, promptHash, settingsHash and promptSource). prompt_source is "built-in", or "override" for a metric_judge with a prompt. A change of the rubric, label rule, temperature, template, prompt or model changes it. The model that replied does not. Compare it with data["judge"] of the scores of an eval to check that they come from the calibrated configuration. judge is None when n is 0, and when the scorer records no judge. A scorer of your own gets the four string fields reported when it puts them in data["judge"]; its prompt_source is "built-in" unless it records "override". (The judge of a CalibrationResult is a label, not this.)
calibrate refuses a mix of judges: it raises TypeError when the scores of the verdicts differ in template, model, promptHash, settingsHash or promptSource, or when only some of them record data["judge"]. The message names the first field that differs and the two cases. A case with no verdict is not compared. The check follows the judge calls, so a rejected calibration has used the model.
Conversation metrics
Section titled “Conversation metrics”knowledge_retention_v2, role_adherence_v2, topic_adherence_v2, turn_relevance_v2, completeness_v2 and goal_accuracy_v2 are metric judges of a conversation. Their evidence is a conversation and, for the first four, an optional target. They are trialkit definitions, not DeepEval-compatible scores.
Conversation and ConversationTurn are TypedDicts with the same wire keys as TypeScript. Every turn has a unique nonblank id, a role ("user", "assistant" or "tool") and string content. Assistant decisions can include ordered retrievedContexts, toolCalls (id, name, arguments) and availableTools (name, description, inputSchema). A tool result needs callId for an earlier call and status ("ok" or "error"). Any turn can link spanIds. Call IDs are unique, apart from turn IDs. Menus and sources are snapshots, also empty lists; nothing pools them. The judge refuses unknown fields, a value that is not JSON (a tuple, a key that is not a string) or that canonical JSON cannot write, an empty transcript, no assistant decision, and an assistant decision without a user turn before it, as misuse that names the cause: metric_judge: goal-accuracy@2: evidence.conversation must be a valid conversation: turn ids must be unique for case c1, got {"turns":[.... It shows each turn with its 1-based number, id, role, content and the calls and results that it has, and no menu, snapshot or span ID.
from trialkit.judge import metric_judgefrom trialkit.judge.metrics import completeness_v2, knowledge_retention_v2from trialkit.pytest import AsyncRunFixture, AsyncScoreFixture
completeness = metric_judge(completeness_v2, model=model)retention = metric_judge(knowledge_retention_v2, model=model)
async def test_agent_conversation(run: AsyncRunFixture, score: AsyncScoreFixture) -> None: # The conversation that the agent ended is the evidence. result = await run({"conversation": seed}) await score(completeness, {"conversation": result["conversation"]}) # One reply of it: the conversation ends at the target turn, and only that reply is graded. await score(retention, {"conversation": result["conversation"], "target": "a2"})Put the seed prefix in input["conversation"] and return the complete transcript in output["conversation"]. Without a target, knowledge_retention_v2, role_adherence_v2, topic_adherence_v2 and turn_relevance_v2 grade every assistant reply against its full preceding turns, and a later correction cannot erase an earlier violation. With a target, the conversation ends at that turn and only that reply is graded: every earlier turn, also an earlier assistant reply, is context, and no later turn reaches the model. A target that no turn has, or that names a user or tool turn, is misuse, and so is a bad turn after the target. completeness_v2 and goal_accuracy_v2 judge the conversation through its end and read no target; a target key in their evidence is ignored. In the definition form, your function gives the target of each case:
from trialkit.judge import metric_judgefrom trialkit.judge.metrics import knowledge_retention_v2
retention = metric_judge( knowledge_retention_v2, model=model, evidence=lambda case, output: { "conversation": output["conversation"], "target": case.input["targetTurnId"], # None grades every reply },)A version 1 single-reply recipe took a history, the current user turn and the reply. Give the same conversation instead: the history, the user turn and the reply as an assistant turn, with that turn as the target. knowledge_retention_v2 fails a needless re-ask of a fact that the user gave, for a targeted reply too.
Empty answers fail: an assistant turn that only calls tools is a decision, not an empty answer, and any other empty assistant reply fails. Tool turns are evidence, not user goals or policy. A topic change does not cancel a request; only an explicit cancellation removes the obligation. Promises and pending clarification are incomplete unless resolved by the end; a justified refusal can close a request without proving goal achievement. Executing an explicitly cancelled action afterward fails completeness, and an acknowledged stop request with observed cessation can be resolved. Policy belongs in the trusted rubric, not in the transcript. Goal accuracy is separate from completeness: a success claim does not prove an external action, and a justified refusal does not achieve the goal. Explicit corrections replace affected goals. Cancellation removes a goal but forbids executing the cancelled action later; a stop can succeed through acknowledgement and observed cessation. Set a size limit of your own and refuse oversized evidence without trimming.
Projection helpers
Section titled “Projection helpers”retrieval_turns(conversation, references=None) and tool_decisions(conversation, spans) make the evidence of each assistant decision of a conversation. Give each result to a metric as it is, and keep each verdict apart: do not average them, or combine a decision with plan quality or adherence.
| Function | Returns | Refuses (misuse) |
|---|---|---|
retrieval_turns(conversation, references=None) |
One RetrievalTurn (a dict) for each assistant turn that has a retrievedContexts snapshot, also an empty one: turnId, request (the dialogue before the turn), answer (its content) and sources (its own snapshot). With references, a reference too. |
An invalid conversation, no turn with a snapshot, references that is not a mapping with string keys, and a retrieval turn whose reference is not a nonblank string in references. |
tool_decisions(conversation, spans) |
One ToolDecision (a dict) for each assistant turn, also one without a call: turnId, request (the dialogue before the turn), tools (its availableTools) and spans (the spans that the turn and the results of its calls map, in record order). |
An invalid conversation, spans that are not a nonempty valid list, a turn without its menu, mapping or spans, a span that two turns map, and a call whose span is no execute_tool span of the same tool with the same arguments. |
references is the factual reference answer of each turn, by turn ID, not a human label. A turn without a snapshot is not a retrieval turn. No earlier snapshot is pooled and no later turn is shown: the evidence of a decision depends on its prefix alone. A span keeps its parent only when the parent is one of the decision’s spans. Arguments of a call and its span compare as canonical JSON, so key order does not matter and 1 is not true; arguments that canonical JSON cannot write, such as a string with a lone surrogate, are no match. The conversation projection contract has every message.
The ranking and coverage metrics read reference: pass references to retrieval_turns. tool_appropriateness_v2 reads request and tools, and it grades the spans of its trial, not the spans key of the evidence. Grade each decision in a trial of its own: in one trial, the spans of the other decisions would be graded too. The task records decision["spans"] with ctx.span, and calibrate replays them from LabeledOutput(..., spans=decision["spans"]).
from trialkit.core import Dataset, EvalCase, EvalDefinition, TrialContextfrom trialkit.judge import ToolDecision, metric_judge, retrieval_turns, tool_decisionsfrom trialkit.judge.metrics import faithfulness_v2, tool_appropriateness_v2
# One faithfulness verdict for each retrieval turn: the output of the task is the evidence.turn_faithfulness = EvalDefinition( name="turn-faithfulness", dataset=Dataset( cases=[EvalCase(turn["turnId"], turn) for turn in retrieval_turns(conversation)] ), task=lambda turn: turn, scorers=[metric_judge(faithfulness_v2, model=model)],)
# One tool-use verdict for each decision: the task records the spans of its decision.async def record_spans(decision: ToolDecision, ctx: TrialContext) -> ToolDecision: for span in decision["spans"]: ctx.span(span) return decision
tool_use = EvalDefinition( name="tool-use", dataset=Dataset(cases=[EvalCase(d["turnId"], d) for d in tool_decisions(conversation, spans)]), task=record_spans, scorers=[metric_judge(tool_appropriateness_v2, model=model)],)The person-run calibration example has opt-in paid commands for TypeScript and Python. Freeze before holdout; require full n and zero errors, and inspect every disagreement. Its tiny fictional MIT corpus is smoke calibration, not universal accuracy. Calibrate on application-held-out data before gating. calibrate excludes errors from n, agreement, kappa and confusion; it does not count them as negative labels. Keep report.judge with a report: it has the template, the model, the prompt hash and the settings hash. Missing spans remain valid for judges that do not grade trajectories. Select only the stated text fields of the evidence, not a whole application object, and redact conversations, sources and spans before sending them to a model. Reports can contain judge reasons: review them before sharing.
Model clients
Section titled “Model clients”| Export | Options and defaults | Meaning |
|---|---|---|
ModelClient |
id, async generate(request) |
Model port. |
ModelMessage(role, content) |
Both required | Message role is system, user or assistant. |
ModelRequest(messages, json_schema=None, temperature=None, max_output_tokens=None) |
messages required |
Model request. |
ModelReply(text, model, json=None, usage=None) |
text and model required |
Reply and optional input_tokens, output_tokens, cached_input_tokens, reasoning_tokens, cost usage. |
from_function(id, generate) |
Both required | Wrap a plain or async model function. |
openai_model(*, base_url, api_key, model, id=None, max_output_tokens_param="max_completion_tokens", request_timeout_ms=120_000) |
base_url, api_key, model required |
OpenAI-compatible Chat Completions client. Default id is <provider>:<model> from the host; use max_output_tokens_param="max_tokens" when required. |
The HTTP client follows no redirects, hides the API key from repr and uses urllib proxy settings. It retries connection errors and HTTP 408, 409, 429 and 5xx twice, like the OpenAI SDK default. It honors Retry-After (up to 60 seconds); otherwise it uses bounded exponential backoff. It never retries an unreadable reply or a quality verdict. The request deadline covers all attempts and delays, and cancellation stops retrying. Python cannot force-stop a blocked daemon worker; its socket timeout limits inactivity. Empty replies name a refusal or finish reason when present, and a missing model is an infrastructure error. IP-literal server ids keep the whole address and explicit port.
Generate cases
Section titled “Generate cases”import asyncio
from trialkit.cases import write_casesfrom trialkit.core import read_casesfrom trialkit.judge.generate import generate_cases
result = asyncio.run( generate_cases(model=model, source={"cases": read_cases("seeds.jsonl")}, count=12, seed=7))write_cases("draft.yaml", result.rows)generate_cases asks a model once for new eval cases and returns them for a person to review. It behaves as generateCases in TypeScript; read that page for the rules. All options are keywords.
| Option | Default | Meaning |
|---|---|---|
model |
Required | A ModelClient; each row records its id. |
source |
Required | Exactly one of {"cases": [EvalCase, ...]}, {"documents": [{"id": ..., "text": ...}]} or {"description": "..."}. Each is non-empty. |
count |
Required | Integer from 1 to 100: the candidates to ask for. The reply must have exactly this many. |
existing |
None |
EvalCase values or rows that the new rows must not repeat. Never sent. Seed cases count as existing. |
seed |
0 |
Integer from 0 to 4294967295 that the prompt shows: a nonce, not a seed for the model. |
max_output_tokens |
4096 |
Integer from 1 to 32768, sent to the model. |
temperature |
None |
Number from 0 to 2. None sends no temperature. |
request_timeout_ms |
120_000 |
Number above 0 and at most 2147483647. At the deadline the call is cancelled, and a late reply is ignored. |
It returns GeneratedCases(rows, discarded, usage): EvalCase rows with the id generated-<sha256>, the tag generated and the provenance metadata, the number of dropped candidates, and the usage of the reply (None when unknown, never zero). Invalid options raise TypeError or ValueError before the call; a prompt of more than 64 KiB raises ValueError there too. A failed call, an unreadable reply (a reply text or a structured output of more than 64 KiB is one; each is checked on its own), an invalid candidate and a reply with no new case raise InfrastructureError with the cause. Cancellation of the task, KeyboardInterrupt and SystemExit pass through. Generation calls the model once, with no reply repair or cache; the HTTP client owns transient transport retries.
Trajectory judge
Section titled “Trajectory judge”from trialkit.judge import trajectory_judge
process = trajectory_judge( model=model, rubric="Look up the returns policy before answering. Use the captured result in the answer.",)The canonical JSON transcript preserves span order, ids, parent ids and status. It includes only gen_ai.operation.name, gen_ai.request.model, gen_ai.response.model, gen_ai.system_instructions, gen_ai.input.messages, gen_ai.output.messages, gen_ai.tool.name, gen_ai.tool.call.id, gen_ai.tool.call.arguments and gen_ai.tool.call.result. It omits times and unknown attributes. Missing or unrenderable spans are error before a model call; model-only spans are valid. Missing arguments differ from explicit None. Sequence is not proof the agent used a result.
This judge sends captured messages, arguments and results to the model. Redact secrets before ctx.span. Use tool_calls and tool_policy for deterministic requirements and limits.