trialkit-scorers
Deterministic scorers check outputs without a model call. They return named, versioned scores for trialkit runs.
Install
Section titled “Install”pip install trialkit-scorersExample
Section titled “Example”from trialkit.core import Dataset, EvalCase, EvalDefinitionfrom trialkit.scorers import exact, numeric
def answer(question: str) -> dict[str, str | int]: return {"category": "billing", "priority": 2}
definition = EvalDefinition( name="triage", dataset=Dataset( cases=[EvalCase("invoice", "A duplicate charge", {"category": "billing", "priority": 2})] ), task=answer, scorers=[ exact( name="category", output=lambda result: result["category"], expected=lambda value: value["category"], ), numeric( name="priority", output=lambda result: result["priority"], expected=lambda value: value["priority"], tolerance=1, ), ],)- Import
trialkit.scorers. The only package dependency istrialkit-core. - Every factory option is keyword-only.
output=Noneselects the whole task output; an omittedexpectedselects the case’s expected value. A fixed value or picker can override it. - A missing or invalid case
expectediserror; invalid output fails the score.EvalCase.expected=Nonemeans no reference, but explicitexpected=Noneonexactcompares with null.Scoreris re-exported fromtrialkit.core.
Scorers
Section titled “Scorers”These follow the shared TypeScript scorer behavior, with Python keyword names. Core validates every returned ScoreValue.
| Export | Options and defaults | Meaning |
|---|---|---|
exact |
name="exact", ignore_case=False, trim=False, output=None, expected from case |
Iterative JSON equality, at most 500 container levels. String options apply to values, not keys. trim=True uses ECMAScript whitespace. Version 2. |
includes |
name="includes", mode="all", ignore_case=False, output=None, expected from case |
all scores the fraction of non-empty needles found; any passes if one is found. Version 2. |
regex |
pattern required (re.Pattern), name="regex", output=None |
Pass if the compiled pattern finds a match. Reasons show portable i, m and s flags. Version 2. |
numeric |
name="numeric", tolerance=0, scale=None, output=None, expected from case |
Pass within inclusive tolerance. Numeric inputs round to ECMAScript doubles. With a scale, value is max(0, 1 - difference / scale). Version 2. |
set_match |
name="set-match", metric="f1", threshold=1, output=None, expected from case |
Compare iterables of strings and finite doubles. Metrics: f1, jaccard, precision, recall. Two empty sets score 1. Version 2. |
edit_distance |
name="edit-distance", threshold=0.8, ignore_case=False, output=None, expected from case |
Bit-parallel Levenshtein similarity over code points; each string is limited to 10,000 code points. Version 2. |
tool_calls |
name="tool-calls", order="exact", args="equal", results="ignore", output=None, expected from case |
Order: exact, in_order or any; args and results: equal, subset or ignore. Each list has at most 2,000 calls, and at most 256 with the order any, whose search for an assignment grows with the cube of the lists; more raise ValueError. The matching blocks until it ends: about 2 s for two lists of 2,000 calls. Recorded spans take precedence over output. Version 3. |
tool_policy |
name="tool-policy", allowed=None, forbidden=(), max_calls=None, allow_repeated=True |
Binary restrictions on recorded tool calls; no output or expected picker. Version 2. |
schema |
schema required, name="schema", output=None |
A plain or async callable. It reads the result by its shape: a dict, or an object with properties. The result needs issues or value. Issues that are not None are a list of objects with a string message and an optional path, and they fail the score. Any other result is misuse (error). Version 2. |
Tool call results
Section titled “Tool call results”from trialkit.scorers import tool_calls
required_result = tool_calls( args="ignore", results="subset", expected=[{"name": "lookup", "result": {"found": True}}],)With results="equal" or "subset", every expected call needs a JSON result; None is valid. A missing expected result is a dataset error. The scorer reads gen_ai.tool.call.result from spans, or result from output calls. A missing result or an error-status span never matches. Strings compare as recorded, without JSON parsing. Subsets allow extra object fields at every depth; arrays must have the same length and order. The default "ignore" does not check results or span status.
Recorded tool policy
Section titled “Recorded tool policy”from trialkit.scorers import tool_calls, tool_policy
required = tool_calls(order="in_order", args="ignore", expected=[{"name": "lookup"}])limits = tool_policy( allowed=["lookup", "fetch"], forbidden=["delete"], max_calls=3, allow_repeated=False)- Set an active restriction.
allowedandforbiddencontain non-empty, case-sensitive names;allowed=Nonesets no allowlist, andallowed=[]permits no calls. A name cannot be both allowed and forbidden. The reason names the first disallowed call. max_callsis a finite non-negative integer or integral float;allow_repeatedis boolean. Creation validates options and snapshots names. A repeat has the same name and canonical JSON arguments; missing arguments differ fromNoneand{}.- Error-status tool spans count. Missing spans or an unnamed tool operation are
error; model-only spans contain zero calls and can pass.
Whole-string and JSON checks
Section titled “Whole-string and JSON checks”These checks need no extra option. Each snippet runs in tests/test_recipes.py.
import jsonimport refrom typing import Any
from trialkit.scorers import exact, regex, schema
# Exact match: ignore whitespace at both ends of each string.same = exact(trim=True)
# Whole-string pattern: the pattern must cover all of the output.code = regex(pattern=re.compile(r"\A(?:[A-Z]{3}-\d{4})\Z"))
# JSON correctness: pass your validator function, and parse the output text first.def validate_person(value: Any) -> dict[str, Any]: if isinstance(value, dict) and isinstance(value.get("name"), str): return {"value": value} return {"issues": [{"message": "expected an object with a string name"}]}
json_person = schema(schema=validate_person, output=json.loads)regexfinds a match anywhere in the output. Write\A(?:p)\Z, not^(?:p)$.$also matches before a final newline, so"ABC-1234\n"would pass. Withre.MULTILINE,^and$match at each line, so a match on one line of several would pass.\Aand\Zmatch only at the ends of the string, with or without that flag. The group keeps|inside the anchors.regexdoes not trim; to trim first, addoutput=lambda text: text.strip().validate_personis your own validator function. Ajson.loadserror is afailwith its message as the reason, not anerror. JSON that breaks the schema is afailtoo.
Numbers follow ECMAScript doubles, including Python integer inputs. Similarity reasons use ECMAScript toFixed(3) rounding. A custom scorer can raise caller misuse with misuse(TypeError(...)) from trialkit.core.adapter; an ordinary validator exception remains a quality failure unless it describes infrastructure.