Skip to content

Record and replay model replies

Use replay when required CI must exercise the real request builder, SDK and parser without a model call.

import { type Cassette, recordingFetch, replayFetch } from '@trialkit/replay';
import { loadCassette, saveCassette } from '@trialkit/replay/node';
function recorder(): typeof fetch {
const cassette: Cassette = []; // A fresh recording needs no existing file.
return recordingFetch(fetch, cassette, {
save: () => saveCassette('cassettes/triage.json', cassette),
});
}
function replayer(): typeof fetch {
return replayFetch(loadCassette('cassettes/triage.json'), {
hint: 'Record this request again.',
});
}
  • Give either wrapper to the app client’s fetch option. A replay matches the recorded request exactly; headers count only when matchHeaders names them, and credentials must be redacted.
  • A changed prompt, model, rubric or case gets a replay 404 and therefore a trial error. A person records that change. Use mode: 'auto' to replay hits and record misses.
  • Cassettes hold valid 2xx JSON replies and streams in call order. Replay tests wiring and scoring, not a new model sample; fix a bad recording and record again instead of editing it or hiding it with xfail.

Use the same wrapper for task and judge calls. The replay reference covers cassette matching, redaction, streams and files, and examples/replay runs a whole eval offline.

Python has no replay package: record the HTTP replies with VCR.py and pytest-recording. See Record and replay for the recipe that makes a missing recording a trial error.