11. Events, results and backends
Events are the versioned record of a run. JSON Lines is the canonical history; reporters and backends derive other views from it, and reports and comparisons need no backend. Gates compare matched evidence and do not treat infrastructure errors as quality failures.
Event stream
Section titled “Event stream”Every event has type, schemaVersion, runId and an ISO 8601 time. The event schema lists every field and the order of events in a run.
| Event | Marks |
|---|---|
RunStarted |
The eval, dataset, policy, labels and optional config |
CaseStarted |
A case begins |
TrialStarted |
A trial begins |
TrialCompleted |
A trial’s outcome, scores, usage, error and spans |
Attachment |
A value that a trial attached: the run call, logs |
CaseCompleted |
A case’s outcome, counts, reason and input hash |
CheckCompleted |
A set check’s outcome |
RunCompleted |
The summary, with the first stoppedBy cause |
Schema version 1 covers additive fields: bump it for a breaking change, not for an added optional field, and readers ignore unknown fields. Event times come from a monotonic clock anchored at one wall-time origin, so changing the calendar clock cannot reorder events.
Reporters and JSON Lines
Section titled “Reporters and JSON Lines”A reporter gets events in order without blocking the run. A reporter that throws gets no more events and adds at most one error to RunResult.reporterErrors; it never changes an outcome, so check reporterErrors as well as summary.ok. Each run writes its own file, created exclusively, so concurrent runs and pytest-xdist workers never share one. JSON Lines is the complete record; the other reporters carry less.
Reading histories
Section titled “Reading histories”The TypeScript and Python readers apply the same file and event rules, so the TypeScript CLI reads a file that Python wrote. Each consumer decides what a problem means. Reports and comparisons warn and continue, and the MLflow uploader skips a whole file that has an invalid event line, because a run could miss that line. The rules are in the TypeScript and Python references and the shared corpus.
Reports and history
Section titled “Reports and history”A dataset hash or trial-policy change marks a run as not comparable to the previous completed run in its series, so use the same trials and earlyStop: false for comparable rates. A report never invents evidence: missing evidence says not recorded, and a trial without completion keeps its logs and gets no verdict (CLI reference).
Comparing runs
Section titled “Comparing runs”Comparison pairs runs and cases by identity before it computes changes (reference).
- A series is an eval plus the selected label,
confighash orconfig.<path>values, by defaultmodel, andcomparepairs the latest completed run of each series. Why: a matrix eval (one run per model) compared by eval name alone paired different models. - A case is left out only when both runs have an input hash for it and the hashes differ, so the first comparison after an upgrade, and a case that did not start, still count for the gate.
- Rates count only cases that passed or failed on both sides. Why: rates over different case sets make a new hard case look like a regression.
- Changes in dataset, policy, scorer version, judge or model warn. Config and label changes inform.
Regression gate
Section titled “Regression gate”The regression rule defines the gate. A flip of a case without a gating tag does not fail it (--gate-flips and --no-gate-rate change the rule). Why: a gate that fails on any single flip fails an unchanged prompt often, because a few cases flip by chance. The gate fails on tagged cases, set checks, a lost verdict and the rate interval, and it lists other flips as noise. Set check thresholds from the low bound of a baseline.
JUnit and OpenTelemetry
Section titled “JUnit and OpenTelemetry”These reporters map the same events to CI test reports and traces (JUnit, OpenTelemetry). The OpenTelemetry reporter uses the app’s tracer and never registers a provider. Datadog can consume these traces and CI can consume JUnit, so separate Datadog and Allure reporters are not built, and neither are Braintrust and LangSmith uploaders.
MLflow
Section titled “MLflow”The live reporter and the uploader use one mapping for open-source MLflow and Databricks (TypeScript, Python). Both languages call the REST API without an MLflow SDK or a global tracer provider. The app provides the server, authentication and experiment, and the reporter does not retry (fetch and transport are hooks for retries or a proxy). The reporter sanitizes user-derived keys to MLflow’s limits. Why: MLflow rejects a bad key, and one bad key fails a whole batch. Uploading a file twice creates two runs (§14). Not built: config contents, trial metrics, a run artifact, expected-value assessments, Unity Catalog trace assessments and prompts.