Skip to content

Scorers: deterministic first

Use a deterministic scorer when code can decide the rule.

Output Scorer
Class, id or field exact
Number with tolerance numeric
Date, id or code regex or exact
Required phrase or citation includes
List setMatch
Agent calls toolCalls or toolPolicy
JSON shape schema with the app’s Standard Schema validator
import type { Scorer } from '@trialkit/core';
const maxWords: Scorer<string, { summary: string }> = {
name: 'summary-words',
version: '1',
score: ({ output }) => {
const count = output.summary.trim().split(/\s+/).length;
return { value: count <= 12 ? 1 : 0, pass: count <= 12, reason: `${count} words` };
},
};
  • A scorer has a unique name, version and score. Change the version when its logic changes, so comparison does not mix measurements.
  • In a test form, use await score(scorer, output, expected?). The optional toPass matcher needs await too: never discard its promise.

The scorers reference lists every scorer, option and default. For a whole-string pattern, an exact match without surrounding whitespace, or a JSON check, see the recipes for TypeScript and Python.