Skip to content

Example: draft cases for review

Copy this to draft eval cases from a few seed cases, save them to a YAML file, and review them before they join a dataset. A fake model stands in for yours: no model, API key or network call.

Run these in this directory after the repository build. Node.js 22.23.4 or later.

Terminal window
pnpm test
pnpm generate

pnpm generate drafts four candidates from seeds.jsonl, drops the one that repeats a seed, and prints the draft that it wrote to a new temporary folder:

3 new cases, 1 repeated an existing case
draft: <temporary folder>/draft.yaml
  • generateCases calls the model once and returns rows. Each row has the id generated-<sha256 of its input>, the tag generated and metadata that names the model, the template (cases@1), the seed and two hashes. Keep them after the review: they record the origin, not an approval.
  • writeCases creates the file and never overwrites one. Edit the draft, then copy the cases that you approve into your dataset. The references that a model writes are not ground truth.
  • For a real model, pass openaiModel({ client, model }) or aiSdkModel({ model }) as model. The credentials come from your app’s settings: trialkit reads no environment variable. See the judge README and the guide.
File Role
seeds.jsonl Three seed cases.
src/draft.ts The fake model, and draftCases: read the seeds, generate, write.
scripts/generate.ts pnpm generate.
generate.test.ts Checks the dropped repeat, the provenance and the refusal to overwrite.