13. Testing the library
Tests cover core logic, integrations, shared language contracts, package consumers and example apps. No library or CI test calls a model or another paid API, and live checks are a release task. CONTRIBUTING.md lists the gate, and its test layers map each layer to its files. The TypeScript blocks of the READMEs and docs compile against the built packages (scripts/check-doc-code.mjs), and the Python blocks pass mypy --strict against the local packages (scripts/check-doc-code-python.mjs). Why: a README snippet that nobody compiled kept a type error (unknown is not string for the prompt option) while its behavior was right.
Exhaustive and generated tests
Section titled “Exhaustive and generated tests”Property tests pin properties of trial verdicts and comparisons, not selected examples: early stop gives the full-run verdict, an error never becomes a pass or a fail, and a history compared with itself shows no difference. fast-check and Hypothesis generate policies with every kind of scripted step, and exhaustive tests run every pass and fail sequence up to five trials.
Shared language contracts
Section titled “Shared language contracts”The two cores implement the same contracts, and a behavior found in one core but not the other is a bug. Pin it in a scenario, or in a test in each language when the harness cannot script it (a throwing filter). Generated parity asks both cores the same questions, including cases that no fixture lists, and compares each event in full. It replaces only a run id, a timestamp, a duration, the offset of a log line, an error stack and the package name of the core. When concurrency is above 1, it groups events of cases that ran together by case, and it keeps the order of events within a case. A serial run keeps the event order. Its first run found null marks, oversized integers, folder links and filename order differences, and each became a shared fixture. Fake models exercise every label, disagreement and unreadable reply, so fake agreement proves plumbing, not judge accuracy (recipe reviews).
Release checks
Section titled “Release checks”Live checks cover integrations that fakes and recordings cannot prove. Before a release, a person checks the rubric judge through OpenAI and Databricks Unity Gateway; both TypeScript and Python MLflow reporters and uploaders against open-source MLflow and Databricks; the Python judge and model client against a real model; and support-triage on real models and a real environment. The OpenRouter recording agrees with 13 of 14 human labels (kappa 0.857) and replays in CI.