@trialkit/cases
Read trialkit eval cases from YAML or TOML files, and write draft cases to YAML or JSON Lines. The readers are synchronous, so a runner test can declare cases without top-level await.
Install
Section titled “Install”Install the reader and the parser that its subpath uses:
npm install --save-dev @trialkit/cases @trialkit/core yamlnpm install --save-dev @trialkit/cases @trialkit/core smol-tomlyaml ^2.8.3 and smol-toml ^1.9.0 are optional peers. No other dependencies: your installed parser is used. CI runs the shared corpus with the floor and current version of each parser. readYamlCases names npm i yaml when YAML is absent; readTomlCases names npm i smol-toml when TOML is absent. Node.js 22.23.4+ is required.
Case files allow at most 64 nested arrays and mappings. The top-level mapping is level 1.
import { readYamlCases } from '@trialkit/cases/yaml';
const cases = readYamlCases('evals/cases.yaml');A YAML file has exactly one cases key. Its value is an array of the same id, input, expected, tags, metadata, skip, and xfail case fields that @trialkit/core accepts. YAML uses the 1.2 core schema. Aliases, bare timestamps, tags, binary values, sets, non-string keys, non-finite numbers, duplicate keys, and multiple documents fail.
Only %YAML 1.2 is accepted. A plain scalar with a YYYY-M-D date prefix followed by its end, T, t or a space is a bare timestamp; quote it to keep text. LF and CRLF read as LF; lone CR, raw tabs, raw control characters (including DEL/C1), U+2028/U+2029 and U+FFFE/U+FFFF fail. Use quoted escapes for those characters. Paired \u surrogate escapes read as one Unicode character.
Use cases.schema.json with YAML Language Server:
# yaml-language-server: $schema=https://raw.githubusercontent.com/lzehrung/trialkit/main/spec/schemas/cases.schema.jsoncases: - id: refund input: |- The customer was charged twice. Find the duplicate invoice and refund it. expected: billingAnother YAML file
Section titled “Another YAML file”import { readYamlMapping } from '@trialkit/cases/yaml';
const settings = readYamlMapping('evals/settings.yaml');readYamlMapping(path) reads any YAML file whose root is a mapping, by the rules above, and returns the mapping as plain data. It has no cases key and no case rules. A root that is not a mapping, and an empty file, fail with <path>:1: the document must be a mapping. Its messages name readYamlMapping, and it needs the yaml peer as readYamlCases does. readTemplate and readPrompt of @trialkit/judge/yaml read judge templates and prompts with it.
import { readTomlCases } from '@trialkit/cases/toml';
const cases = readTomlCases('evals/cases.toml');TOML 1.0 uses [[cases]] tables and supports multi-line strings; CRLF reads as LF. TOML 1.1 escapes, newlines or trailing commas in inline tables, and times without seconds are refused. It has the same case fields and JSON-value rules as YAML. Date and time values, non-finite numbers, and integers that are not exactly representable as an ECMAScript double fail. Taplo can use the same cases.schema.json.
Write a draft
Section titled “Write a draft”import { writeCases } from '@trialkit/cases/write';
await writeCases('draft.yaml', result.rows); // rows from generateCases in @trialkit/judge/generatewriteCases(path, rows) makes a file for a person to review, for example the rows of generateCases. The extension chooses the format: .yaml and .yml write one cases array (the optional yaml peer is required), and .jsonl writes the canonical JSON of each row on its own line, ended by LF.
- The rows pass
casesFromRows, so a problem nameswriteCases rows: row <n>. A YAML draft must also fit in 64 levels. A row has only the fields that have a value:nullfields, emptytagsand emptymetadataare left out. - The whole text is made before the file is, and the file is created exclusively: an existing file is never overwritten or extended, and the call throws. A failed write deletes the file that it created, and no other. A draft that is gone already is no failure, and the write error is thrown as it is. When the delete fails for another reason, the call throws an
Errorthat names the draft that stays, with the write error as itscause. The directory must exist. - YAML has no alias and no folded line. Text with several lines is a block, unless it starts with a space or a line break: a block of it would need an indentation indicator, which readers do not agree on, so it is double-quoted. A scalar that a reader could take for another type (a date, a number,
yes) is double-quoted, and so is text with control characters. - Numbers follow ECMAScript doubles. A whole number that is no safe integer (2^53 or more) is written in exponent notation, for example
1.152921504606847e+18for 2^60, because the readers refuse an integer that is not exactly a double. The Python writer keeps an integer that a double holds exactly, so the text of a number can differ between the two drafts. Both readers recover the same numeric value. readYamlCasesandreadCasesread the file back. Review and edit it, then copy the approved rows into your dataset.
Reference
Section titled “Reference”| Export | Behavior |
|---|---|
readYamlCases(path) from @trialkit/cases/yaml |
Synchronously reads one UTF-8 YAML file and returns ordered EvalCase<unknown, unknown>[]. It reports a parser or case problem as TypeError with path, source line when available, and reason. |
readYamlMapping(path) from @trialkit/cases/yaml |
Synchronously reads one UTF-8 YAML file whose root is a mapping and returns it as plain data, Record<string, unknown>, by the YAML rules of the case files. It reports a problem as TypeError with path, source line, and reason. |
readTomlCases(path) from @trialkit/cases/toml |
Synchronously reads one UTF-8 TOML file and returns ordered EvalCase<unknown, unknown>[]. It reports a parser problem as TypeError with path, source line, and reason. A TOML parser gives no line for a case, so it names the row: <path>: row <n>: <reason>. |
writeCases(path, rows) from @trialkit/cases/write |
Asynchronously writes rows to a new .yaml, .yml or .jsonl file. It refuses an existing file, and an invalid row or a missing parser leaves no file. See Write a draft. |
The shared read-case-files corpus pins both readers, and the shared write-cases fixture pins both writers.