Skip to content

Metric recipe calibration

Run the 21 version 2 metrics of @trialkit/judge and trialkit-judge against original fictional MIT labels, with a model that you supply. The commands make paid model calls and never run in CI. The result is a smoke calibration, not a measure of accuracy.

recipes.ts and recipes.py are a list of the 28 recipes: a metric constant and the picker that selects its evidence from a fixture case. A turn recipe grades one decision of a conversation, with the evidence that retrievalTurns or toolDecisions makes. calibrate.ts and calibrate.py are the command. The tests use fake models only: pnpm --filter example-metric-recipes test.

From the repository root, after pnpm install, uv sync and pnpm build. Git ignores eval-results/: keep the module, the freezes and the reports there.

  1. Key. Make an OpenRouter key for this run only, with a credit limit of USD 1. Set OPENROUTER_API_KEY in the shell that runs the commands (PowerShell: $env:OPENROUTER_API_KEY = 'sk-or-...'). The modules below read it; no package of the repository reads it.

  2. Model module. Export model and settings. settings is a flat JSON object of the non-secret provider settings; its optional temperature (null means none) also goes to each judge. Use the model that you plan to judge with.

    eval-results/judge.mjs
    import { fromFunction } from '@trialkit/core';
    const MODEL = 'anthropic/claude-haiku-5.5';
    export const settings = { temperature: 0.2 };
    export const model = fromFunction(`openrouter:${MODEL}`, async (request, signal) => {
    const response = await fetch('https://openrouter.ai/api/v1/chat/completions', {
    method: 'POST',
    signal,
    headers: {
    authorization: `Bearer ${process.env.OPENROUTER_API_KEY}`,
    'content-type': 'application/json',
    },
    body: JSON.stringify({
    model: MODEL,
    messages: request.messages,
    temperature: request.temperature,
    response_format: { type: 'json_schema', json_schema: { ...request.jsonSchema, strict: true } },
    }),
    });
    if (!response.ok) throw new Error(`OpenRouter ${response.status}: ${await response.text()}`);
    const body = await response.json();
    return {
    text: body.choices[0].message.content,
    model: body.model,
    usage: { inputTokens: body.usage.prompt_tokens, outputTokens: body.usage.completion_tokens },
    };
    });
    eval-results/judge.py
    import os
    from trialkit.judge import openai_model
    settings = {"temperature": 0.2}
    model = openai_model(
    base_url="https://openrouter.ai/api/v1",
    api_key=os.environ["OPENROUTER_API_KEY"],
    model="anthropic/claude-haiku-5.5",
    )
  3. Commands. Run development, read the report, then run holdout. The report paths must be new. Run one language or both.

    Terminal window
    mkdir eval-results
    node examples/metric-recipes/calibrate.ts eval-results/judge.mjs development eval-results/metrics-freeze.json eval-results/metrics-development.json
    uv run --no-sync python examples/metric-recipes/calibrate.py eval-results/judge.py development eval-results/metrics-freeze-py.json eval-results/metrics-development-py.json
    # Freeze the recipes and settings before you inspect holdout results.
    node examples/metric-recipes/calibrate.ts eval-results/judge.mjs holdout eval-results/metrics-freeze.json eval-results/metrics-holdout.json
    uv run --no-sync python examples/metric-recipes/calibrate.py eval-results/judge.py holdout eval-results/metrics-freeze-py.json eval-results/metrics-holdout-py.json
  4. Report. Each command saves one JSON file: the report path that you gave it (eval-results/metrics-development.json and the others). The freeze files are eval-results/metrics-freeze.json and eval-results/metrics-freeze-py.json. Inspect every disagreement before you trust a recipe.

Cost: an estimate from the request sizes, not a quote. Each split sends 168 requests (28 recipes with six cases each), about 555 KB of JSON: 551,003 bytes in development and 564,132 in holdout. At 4 bytes a token that is about 138,000 and 141,000 input tokens. A reply is a reason and a verdict, about 150 tokens, so about 25,000 output tokens a split. At the list price of anthropic/claude-haiku-5.5 on OpenRouter, USD 0.10 for a million input tokens and USD 0.50 for a million output tokens (price page), one split costs about 2.6 cents and both splits about 5 cents, in one language. The two languages together cost twice that. A model that thinks bills its thinking tokens as output: at 1,500 output tokens a call, one split costs about 14 cents. A tokenizer, a changed fixture or another price gives other numbers.

  • Freeze. Development writes the freeze file before the first model call. Holdout refuses a changed model, settings, template, fixture bytes, or hash of the requests that the judges send (rubric, labels, selected evidence and temperature). Do not tune on holdout: revise a recipe with a new development freeze and report path. The holdout cases of faithfulness, context-sufficiency and conversation-tool-use guided the last change of their templates, so treat those three holdouts as development data: they no longer measure a template that never saw them.
  • Report. The command reserves a new report file after it loads the module and checks the settings, and before the first call. It refuses an existing entry, also a dangling symlink, and never unlinks it. A report holds, for each recipe, judge (the template, model, prompt hash, settings hash and prompt source that the scores record in data.judge: compare it with the scores of an eval to check that they come from the calibrated judge), agreement and its 95% interval agreementInterval (6 of 6 is 61% to 100%), kappa (null when undefined), confusion, one result for each case, disagreements, errors and usage, with the models, settings and hashes of the freeze. A failed run writes error and completed through the reserved handle and exits non-zero. The fixture contract defines the report.
  • Limits. Each recipe has six cases in a split, so a run makes 168 calls, and it needs full n and zero errors. Before any call, a local fake model sends every case of both splits through the real judge. No request may have more than 65,536 UTF-8 bytes, and nothing is trimmed.

Definitions and data contracts: judge templates, fixture contract, TypeScript API, Python API.