Example: draft cases for review
Copy this to draft eval cases from a few seed cases, save them to a YAML file, and review them before they join a dataset. A fake model stands in for yours: no model, API key or network call.
Run these in this directory after the repository build. Node.js 22.23.4 or later.
pnpm testpnpm generatepnpm generate drafts four candidates from seeds.jsonl, drops the one that repeats a seed, and prints the draft that it wrote to a new temporary folder:
3 new cases, 1 repeated an existing casedraft: <temporary folder>/draft.yamlgenerateCasescalls the model once and returns rows. Each row has the idgenerated-<sha256 of its input>, the taggeneratedand metadata that names the model, the template (cases@1), the seed and two hashes. Keep them after the review: they record the origin, not an approval.writeCasescreates the file and never overwrites one. Edit the draft, then copy the cases that you approve into your dataset. The references that a model writes are not ground truth.- For a real model, pass
openaiModel({ client, model })oraiSdkModel({ model })asmodel. The credentials come from your app’s settings: trialkit reads no environment variable. See the judge README and the guide.
| File | Role |
|---|---|
seeds.jsonl |
Three seed cases. |
src/draft.ts |
The fake model, and draftCases: read the seeds, generate, write. |
scripts/generate.ts |
pnpm generate. |
generate.test.ts |
Checks the dropped repeat, the provenance and the refusal to overwrite. |