Skip to content

trialkit-scorers

Deterministic scorers check outputs without a model call. They return named, versioned scores for trialkit runs.

Terminal window
pip install trialkit-scorers
from trialkit.core import Dataset, EvalCase, EvalDefinition
from trialkit.scorers import exact, numeric
def answer(question: str) -> dict[str, str | int]:
return {"category": "billing", "priority": 2}
definition = EvalDefinition(
name="triage",
dataset=Dataset(
cases=[EvalCase("invoice", "A duplicate charge", {"category": "billing", "priority": 2})]
),
task=answer,
scorers=[
exact(
name="category",
output=lambda result: result["category"],
expected=lambda value: value["category"],
),
numeric(
name="priority",
output=lambda result: result["priority"],
expected=lambda value: value["priority"],
tolerance=1,
),
],
)
  • Import trialkit.scorers. The only package dependency is trialkit-core.
  • Every factory option is keyword-only. output=None selects the whole task output; an omitted expected selects the case’s expected value. A fixed value or picker can override it.
  • A missing or invalid case expected is error; invalid output fails the score. EvalCase.expected=None means no reference, but explicit expected=None on exact compares with null. Scorer is re-exported from trialkit.core.

These follow the shared TypeScript scorer behavior, with Python keyword names. Core validates every returned ScoreValue.

Export Options and defaults Meaning
exact name="exact", ignore_case=False, trim=False, output=None, expected from case Iterative JSON equality, at most 500 container levels. String options apply to values, not keys. trim=True uses ECMAScript whitespace. Version 2.
includes name="includes", mode="all", ignore_case=False, output=None, expected from case all scores the fraction of non-empty needles found; any passes if one is found. Version 2.
regex pattern required (re.Pattern), name="regex", output=None Pass if the compiled pattern finds a match. Reasons show portable i, m and s flags. Version 2.
numeric name="numeric", tolerance=0, scale=None, output=None, expected from case Pass within inclusive tolerance. Numeric inputs round to ECMAScript doubles. With a scale, value is max(0, 1 - difference / scale). Version 2.
set_match name="set-match", metric="f1", threshold=1, output=None, expected from case Compare iterables of strings and finite doubles. Metrics: f1, jaccard, precision, recall. Two empty sets score 1. Version 2.
edit_distance name="edit-distance", threshold=0.8, ignore_case=False, output=None, expected from case Bit-parallel Levenshtein similarity over code points; each string is limited to 10,000 code points. Version 2.
tool_calls name="tool-calls", order="exact", args="equal", results="ignore", output=None, expected from case Order: exact, in_order or any; args and results: equal, subset or ignore. Each list has at most 2,000 calls, and at most 256 with the order any, whose search for an assignment grows with the cube of the lists; more raise ValueError. The matching blocks until it ends: about 2 s for two lists of 2,000 calls. Recorded spans take precedence over output. Version 3.
tool_policy name="tool-policy", allowed=None, forbidden=(), max_calls=None, allow_repeated=True Binary restrictions on recorded tool calls; no output or expected picker. Version 2.
schema schema required, name="schema", output=None A plain or async callable. It reads the result by its shape: a dict, or an object with properties. The result needs issues or value. Issues that are not None are a list of objects with a string message and an optional path, and they fail the score. Any other result is misuse (error). Version 2.
from trialkit.scorers import tool_calls
required_result = tool_calls(
args="ignore",
results="subset",
expected=[{"name": "lookup", "result": {"found": True}}],
)

With results="equal" or "subset", every expected call needs a JSON result; None is valid. A missing expected result is a dataset error. The scorer reads gen_ai.tool.call.result from spans, or result from output calls. A missing result or an error-status span never matches. Strings compare as recorded, without JSON parsing. Subsets allow extra object fields at every depth; arrays must have the same length and order. The default "ignore" does not check results or span status.

from trialkit.scorers import tool_calls, tool_policy
required = tool_calls(order="in_order", args="ignore", expected=[{"name": "lookup"}])
limits = tool_policy(
allowed=["lookup", "fetch"], forbidden=["delete"], max_calls=3, allow_repeated=False
)
  • Set an active restriction. allowed and forbidden contain non-empty, case-sensitive names; allowed=None sets no allowlist, and allowed=[] permits no calls. A name cannot be both allowed and forbidden. The reason names the first disallowed call.
  • max_calls is a finite non-negative integer or integral float; allow_repeated is boolean. Creation validates options and snapshots names. A repeat has the same name and canonical JSON arguments; missing arguments differ from None and {}.
  • Error-status tool spans count. Missing spans or an unnamed tool operation are error; model-only spans contain zero calls and can pass.

These checks need no extra option. Each snippet runs in tests/test_recipes.py.

import json
import re
from typing import Any
from trialkit.scorers import exact, regex, schema
# Exact match: ignore whitespace at both ends of each string.
same = exact(trim=True)
# Whole-string pattern: the pattern must cover all of the output.
code = regex(pattern=re.compile(r"\A(?:[A-Z]{3}-\d{4})\Z"))
# JSON correctness: pass your validator function, and parse the output text first.
def validate_person(value: Any) -> dict[str, Any]:
if isinstance(value, dict) and isinstance(value.get("name"), str):
return {"value": value}
return {"issues": [{"message": "expected an object with a string name"}]}
json_person = schema(schema=validate_person, output=json.loads)
  • regex finds a match anywhere in the output. Write \A(?:p)\Z, not ^(?:p)$. $ also matches before a final newline, so "ABC-1234\n" would pass. With re.MULTILINE, ^ and $ match at each line, so a match on one line of several would pass. \A and \Z match only at the ends of the string, with or without that flag. The group keeps | inside the anchors. regex does not trim; to trim first, add output=lambda text: text.strip().
  • validate_person is your own validator function. A json.loads error is a fail with its message as the reason, not an error. JSON that breaks the schema is a fail too.

Numbers follow ECMAScript doubles, including Python integer inputs. Similarity reasons use ECMAScript toFixed(3) rounding. A custom scorer can raise caller misuse with misuse(TypeError(...)) from trialkit.core.adapter; an ordinary validator exception remains a quality failure unless it describes infrastructure.