Skip to content

trialkit-cases

Read eval cases from YAML or TOML and use them with trialkit-core, and write draft cases to YAML or JSON Lines. TOML needs no extra dependency; YAML uses the optional ruamel.yaml parser.

Terminal window
pip install trialkit-cases
pip install 'trialkit-cases[yaml]' # also install the YAML parser

The yaml extra accepts ruamel.yaml>=0.17.11,<0.20. No other dependencies: your installed parser is used. CI runs the case-file tests with ruamel.yaml 0.17.11 and the locked version.

from trialkit.cases import read_toml_cases, read_yaml_cases
from trialkit.core import Dataset
yaml_cases = read_yaml_cases("cases.yaml")
toml_cases = read_toml_cases("cases.toml")
dataset = Dataset(id="support-v1", cases=yaml_cases)

Both functions read UTF-8 files synchronously and return ordered EvalCase values. read_yaml_cases(path) reads one YAML 1.2 document; read_toml_cases(path) reads a TOML 1.0 document. The path can be a string or a path-like object. The YAML parser is loaded only when read_yaml_cases is called.

Each file has a top-level cases array and no other top-level keys. For example, cases.yaml can contain:

cases:
- id: refund
input: |
Please refund my order.
It arrived damaged.
expected: refund
tags: [support]

The same case in cases.toml:

[[cases]]
id = "refund"
input = """
Please refund my order.
It arrived damaged.
"""
expected = "refund"
tags = ["support"]

Case fields and their validation match read_cases. Inputs and expected values must be JSON values. Quote dates when they are text: YAML timestamps, TOML dates and times, binary values, sets, non-string keys, non-finite numbers and integers that a double cannot represent exactly are rejected. YAML aliases, explicit tags, duplicate keys and multiple YAML documents are rejected. A document permits at most 64 nested arrays and mappings, counting the top-level mapping as level 1.

Only %YAML 1.2 is accepted. A plain scalar with a YYYY-M-D date prefix followed by its end, T, t or a space is a bare timestamp; quote it to keep text. LF and CRLF read as LF; lone CR, raw tabs, raw control characters (including DEL/C1), U+2028/U+2029 and U+FFFE/U+FFFF fail. Use quoted escapes for those characters. Paired \u surrogate escapes read as one Unicode character. TOML 1.1 escapes, newlines or trailing commas in inline tables, and times without seconds are refused.

Invalid documents or cases raise TypeError with the path and reason. Parser errors include a location when available. A problem in a case of a TOML file names the row: <path>: row <n>: <reason>. A missing YAML parser raises ImportError with the install command. The case document schema supports YAML language servers and Taplo.

from trialkit.cases import read_yaml_mapping
settings = read_yaml_mapping("evals/settings.yaml")

read_yaml_mapping(path) reads any YAML file whose root is a mapping, by the YAML rules above, and returns the mapping as plain data (dict[str, Any]). It has no cases key and no case rules. A root that is not a mapping, and an empty file, raise TypeError with <path>:1: the document must be a mapping. Its messages name read_yaml_mapping, it needs ruamel.yaml as read_yaml_cases does, and the path can be a string or a path-like object. read_template and read_prompt of trialkit.judge.yaml read judge templates and prompts with it.

from trialkit.cases import write_cases
write_cases("draft.yaml", result.rows) # rows from generate_cases in trialkit.judge.generate

write_cases(path, rows) makes a new file for a person to review. rows are EvalCase values or case rows. The extension chooses the format: .yaml and .yml write one cases array (it needs ruamel.yaml; a missing parser raises ImportError with the install command), and .jsonl writes the canonical JSON of each row on its own line, ended by LF. The path can be a string or a path-like object.

  • The rows pass cases_from_rows, so a problem raises TypeError that names write_cases rows: row <n>. A YAML draft must also fit in 64 levels. A row has only the fields that have a value: None fields, empty tags and empty metadata are left out.
  • The whole text is made before the file is, and the file is created exclusively: an existing file raises FileExistsError and is never overwritten or extended. A failed write deletes the file that it created, and no other. A draft that is gone already is no failure, and the write error is raised as it is. When the delete fails for another reason, the call raises an OSError that names the draft that stays, with the write error as its cause (a KeyboardInterrupt or a SystemExit goes on as it is, with that report as a note). The directory must exist.
  • YAML has no alias and no folded line. Text with several lines is a block, unless it starts with a space or a line break: a block of it would need an indentation indicator, which readers do not agree on, so it is double-quoted. A scalar that a reader could take for another type (a date, a number, yes) is double-quoted, and so is text with control characters.
  • A whole number that a double holds exactly (2**60) stays an integer. A whole number that no double holds (2**63 - 1) is written as the float that it rounds to, because the readers refuse an integer that is not exactly a double. A float stays a float. The TypeScript writer uses exponent notation from 2^53 on, so the text of a number can differ between the two drafts. Both readers recover the same numeric value.
  • read_yaml_cases and read_cases read the file back. Review and edit it, then copy the approved rows into your dataset. TypeScript has the same function, writeCases.