Skip to content

@trialkit/scorers

Deterministic scorers for values, strings, sets, tool calls, and schemas. They make no model calls.

Terminal window
npm install --save-dev @trialkit/scorers @trialkit/core

Peer: @trialkit/core; no other dependencies. Node.js 22.23.4+ and TypeScript 5.4+. Install your own Standard Schema v1 validator for schema.

import { defineEval } from '@trialkit/core';
import { exact, numeric } from '@trialkit/scorers';
const evaluation = defineEval({
name: 'answer',
dataset: { cases: [{ id: 'one', input: 'one', expected: 1 }] },
task: (input) => (input === 'one' ? 1 : 0),
scorers: [exact(), numeric({ name: 'near', tolerance: 0.1 })],
});
  • Give scorers unique names within an eval; in the test form use await score(scorer, output, expected?).
  • Invalid thresholds, modes, patterns, and most constant expected values throw at creation; exact checks JSON when scoring.
  • For model-based judges see @trialkit/judge.

Exports: exact, includes, regex, numeric, setMatch, toolCalls, toolPolicy, editDistance, schema, and the core Scorer type. Types: ExactOptions, IncludesOptions, RegexOptions, NumericOptions, SetMatchOptions, SetMetric, SetItem, ToolCallsOptions, ToolCallSpec, ToolCallMatch, ToolCallOrder, ToolPolicyOptions, EditDistanceOptions, SchemaOptions, StandardSchemaV1.

Each factory returns version '2', except toolCalls at '3'. All except toolPolicy accept an output picker (default: the output); all except regex, schema, and toolPolicy accept expected (default: the case’s expected value). expected may be a constant or a picker called with the case’s expected value, not the case; it is not called when that value is absent or null. A fixed exact({ expected: null }) still compares with null. Picker types can be inferred from an annotated parameter.

Option Default Meaning
name Factory name below Scorer name; override for uniqueness.
exact.ignoreCase, exact.trim false, false Apply to every string value at any depth; object keys compare as written.
includes.mode 'all' 'all' requires all expected strings; 'any' requires one.
includes.ignoreCase false Compare without case.
regex.pattern Required RegExp to match; a copy resets global/sticky state on each call.
numeric.tolerance 0 Maximum absolute difference; finite and >= 0.
numeric.scale None Finite positive number for partial credit max(0, 1 - difference / scale); otherwise value is 0 or 1.
setMatch.metric 'f1' SetMetric: 'f1', 'jaccard', 'precision', or 'recall'.
setMatch.threshold 1 Minimum metric, from 0 through 1.
toolCalls.order 'exact' ToolCallOrder: exact list, expected calls in order ('inOrder'), or in any order ('any').
toolCalls.args 'equal' ToolCallMatch: 'equal', recursive object 'subset' with equal arrays, or 'ignore'.
toolCalls.results 'ignore' 'equal' or recursive object 'subset' compares JSON results, independent of args; 'ignore' skips results.
toolPolicy.allowed No allowlist Case-sensitive non-empty names that may be called; [] permits no calls.
toolPolicy.forbidden [] Case-sensitive non-empty names that must not be called.
toolPolicy.maxCalls No limit Maximum calls, a non-negative integer.
toolPolicy.allowRepeated true Allow repeated tool names with identical canonical JSON arguments.
editDistance.threshold 0.8 Minimum similarity, from 0 through 1.
editDistance.ignoreCase false Compare without case.
schema.schema Required A StandardSchemaV1 validator, including async validators.
Scorer (default name) Result and limits
exact ('exact') Iterative equality of plain JSON values, at most 500 container levels. undefined output does not match.
includes ('includes') String output and one non-empty expected string or non-empty array of them. all returns the fraction found; any returns 0 or 1.
regex ('regex') String output; value is 1 on a match, otherwise 0.
numeric ('numeric') Finite numbers pass when absolute difference <= tolerance; scale changes the value, not the pass rule.
setMatch ('set-match') Compares sets of SetItem (strings or finite numbers); duplicates count once, bare strings are invalid, two empty sets score 1, and reasons list up to 10 missing and 10 extra items.
toolCalls ('tool-calls') Compares lists of ToolCallSpec, { name, args?, result? }: at most 2000 calls in each, and at most 256 with order: 'any', whose search for an assignment grows with the cube of the lists; more throw RangeError. Computation blocks timers: about 0.5 s for two lists of 2000 calls. Non-empty recorded spans take precedence over output; error spans match only when results are ignored.
toolPolicy ('tool-policy') Applies restrictions to recorded execute_tool spans, not output or expected values; value is 1 or 0.
editDistance ('edit-distance') Similarity is 1 - levenshtein / longest in code points, or 1 for two empty strings. No Unicode normalization; strings over 10000 code points throw RangeError. Computation blocks timers.
schema ('schema') Reads the result by its shape, so a class instance works, as ArkType’s does. It needs issues or value. Issues that are not null are an array of objects with a string message and a valid path, and they fail the score. Any other result is misuse (error).

toolCalls requires the same list with exact; inOrder and any allow extras and require a non-empty expected list. Repeated expected calls require repeated matches. The score is matched calls divided by the longer list for exact (using their longest common subsequence), or by expected calls otherwise. Missing args means {}; explicit null is refused unless arguments are ignored. Arguments must be JSON objects, not JSON-encoded strings. Bad expected calls are errors; bad output calls fail.

With results: 'equal' or 'subset', every expected call needs a JSON result; null is valid. A missing expected result is a dataset error. The scorer reads gen_ai.tool.call.result from spans, or result from output calls. A missing result or an error-status span never matches. Strings compare as recorded, without JSON parsing. Subsets use the same rules as args: object fields may have extras at every depth, but arrays must have the same length and order.

import { toolCalls } from '@trialkit/scorers';
const requiredResult = toolCalls({
args: 'ignore',
results: 'subset',
expected: [{ name: 'lookup', result: { found: true } }],
});

toolPolicy requires at least one active restriction and snapshots allowed and forbidden names. A name cannot be both allowed and forbidden. A call outside allowed fails and the reason names the first such call. Repetition distinguishes missing arguments from null and {}; named error spans count. Missing/empty spans or an unnamed tool operation are errors; a model-only trajectory has zero tool calls and can pass.

A missing, null or undefined case expectation, or one rejected by a scorer’s expected-value check, ends the trial in error. schema passes undefined output to its validator; other invalid output ordinarily fails the trial. Validator throws follow the shared error rule: infrastructure is error, other throws are fail. Reasons use safely described values, clipped to 200 characters. Numeric inputs follow ECMAScript doubles in both languages; similarity reasons use ECMAScript toFixed(3) rounding.

These checks need no extra option. Each snippet runs in test/recipes.test.ts.

import { exact, regex, schema } from '@trialkit/scorers';
// Exact match: ignore whitespace at both ends of each string.
const same = exact({ trim: true });
// Whole-string pattern: the pattern must cover all of the output.
const code = regex({ pattern: /^(?:[A-Z]{3}-\d{4})$/ });
// JSON correctness: parse the output text, then check the value against a schema.
const json = schema({ schema: person, output: (text: string) => JSON.parse(text) });
  • regex finds a match anywhere in the output. Write ^(?:p)$ and keep the m flag off: with m, a match on one line of several passes. The group keeps | inside the anchors. regex does not trim; to trim first, add output: (text: string) => text.trim().
  • person is any Standard Schema v1 validator. A JSON.parse error is a fail with its message as the reason, not an error. JSON that breaks the schema is a fail too.