Metric recipe calibration
Run the 21 version 2 metrics of @trialkit/judge and trialkit-judge against original fictional MIT labels, with a model that you supply. The commands make paid model calls and never run in CI. The result is a smoke calibration, not a measure of accuracy.
recipes.ts and recipes.py are a list of the 28 recipes: a metric constant and the picker that selects its evidence from a fixture case. A turn recipe grades one decision of a conversation, with the evidence that retrievalTurns or toolDecisions makes. calibrate.ts and calibrate.py are the command. The tests use fake models only: pnpm --filter example-metric-recipes test.
Run the paid smoke calibration
Section titled “Run the paid smoke calibration”From the repository root, after pnpm install, uv sync and pnpm build. Git ignores eval-results/: keep the module, the freezes and the reports there.
-
Key. Make an OpenRouter key for this run only, with a credit limit of USD 1. Set
OPENROUTER_API_KEYin the shell that runs the commands (PowerShell:$env:OPENROUTER_API_KEY = 'sk-or-...'). The modules below read it; no package of the repository reads it. -
Model module. Export
modelandsettings.settingsis a flat JSON object of the non-secret provider settings; its optionaltemperature(nullmeans none) also goes to each judge. Use the model that you plan to judge with.eval-results/judge.mjs import { fromFunction } from '@trialkit/core';const MODEL = 'anthropic/claude-haiku-5.5';export const settings = { temperature: 0.2 };export const model = fromFunction(`openrouter:${MODEL}`, async (request, signal) => {const response = await fetch('https://openrouter.ai/api/v1/chat/completions', {method: 'POST',signal,headers: {authorization: `Bearer ${process.env.OPENROUTER_API_KEY}`,'content-type': 'application/json',},body: JSON.stringify({model: MODEL,messages: request.messages,temperature: request.temperature,response_format: { type: 'json_schema', json_schema: { ...request.jsonSchema, strict: true } },}),});if (!response.ok) throw new Error(`OpenRouter ${response.status}: ${await response.text()}`);const body = await response.json();return {text: body.choices[0].message.content,model: body.model,usage: { inputTokens: body.usage.prompt_tokens, outputTokens: body.usage.completion_tokens },};});eval-results/judge.py import osfrom trialkit.judge import openai_modelsettings = {"temperature": 0.2}model = openai_model(base_url="https://openrouter.ai/api/v1",api_key=os.environ["OPENROUTER_API_KEY"],model="anthropic/claude-haiku-5.5",) -
Commands. Run development, read the report, then run holdout. The report paths must be new. Run one language or both.
Terminal window mkdir eval-resultsnode examples/metric-recipes/calibrate.ts eval-results/judge.mjs development eval-results/metrics-freeze.json eval-results/metrics-development.jsonuv run --no-sync python examples/metric-recipes/calibrate.py eval-results/judge.py development eval-results/metrics-freeze-py.json eval-results/metrics-development-py.json# Freeze the recipes and settings before you inspect holdout results.node examples/metric-recipes/calibrate.ts eval-results/judge.mjs holdout eval-results/metrics-freeze.json eval-results/metrics-holdout.jsonuv run --no-sync python examples/metric-recipes/calibrate.py eval-results/judge.py holdout eval-results/metrics-freeze-py.json eval-results/metrics-holdout-py.json -
Report. Each command saves one JSON file: the report path that you gave it (
eval-results/metrics-development.jsonand the others). The freeze files areeval-results/metrics-freeze.jsonandeval-results/metrics-freeze-py.json. Inspect every disagreement before you trust a recipe.
Cost: an estimate from the request sizes, not a quote. Each split sends 168 requests (28 recipes with six cases each), about 555 KB of JSON: 551,003 bytes in development and 564,132 in holdout. At 4 bytes a token that is about 138,000 and 141,000 input tokens. A reply is a reason and a verdict, about 150 tokens, so about 25,000 output tokens a split. At the list price of anthropic/claude-haiku-5.5 on OpenRouter, USD 0.10 for a million input tokens and USD 0.50 for a million output tokens (price page), one split costs about 2.6 cents and both splits about 5 cents, in one language. The two languages together cost twice that. A model that thinks bills its thinking tokens as output: at 1,500 output tokens a call, one split costs about 14 cents. A tokenizer, a changed fixture or another price gives other numbers.
- Freeze. Development writes the freeze file before the first model call. Holdout refuses a changed model, settings, template, fixture bytes, or hash of the requests that the judges send (rubric, labels, selected evidence and temperature). Do not tune on holdout: revise a recipe with a new development freeze and report path. The holdout cases of
faithfulness,context-sufficiencyandconversation-tool-useguided the last change of their templates, so treat those three holdouts as development data: they no longer measure a template that never saw them. - Report. The command reserves a new report file after it loads the module and checks the settings, and before the first call. It refuses an existing entry, also a dangling symlink, and never unlinks it. A report holds, for each recipe,
judge(the template, model, prompt hash, settings hash and prompt source that the scores record indata.judge: compare it with the scores of an eval to check that they come from the calibrated judge), agreement and its 95% intervalagreementInterval(6 of 6 is 61% to 100%), kappa (nullwhen undefined), confusion, one result for each case, disagreements, errors and usage, with the models, settings and hashes of the freeze. A failed run writeserrorandcompletedthrough the reserved handle and exits non-zero. The fixture contract defines the report. - Limits. Each recipe has six cases in a split, so a run makes 168 calls, and it needs full
nand zero errors. Before any call, a local fake model sends every case of both splits through the real judge. No request may have more than 65,536 UTF-8 bytes, and nothing is trimmed.
Definitions and data contracts: judge templates, fixture contract, TypeScript API, Python API.