Skip to content

6. Core API

Core runs a task against cases, repeats trials, separates quality results from outages, and emits a runner-neutral record. Members and options are in the TypeScript and Python references.

One engine runs both forms, so policy, early stop, verdicts, filters, budgets, reporters and events cannot differ, and a reader treats a test-form run like a definition-form run. describeEval is the primary way to write an eval in a runner. Its shape follows vitest-evals 0.17, with a plain function as the task and the output from run. The trial fields sit at the top level of both, with no policy key (a policy key throws, like any unknown key).

Core owns trials, timeouts and outcome rules. The runner owns selection, ordering and display. A thrown value follows the shared classifier, which does not depend on an assertion-error class, so it works across realms and package copies, and a body that catches an infrastructure error cannot turn it into a pass. The first run input fixes the case input, so a comparison pairs the case by it. The contracts are in the Vitest and pytest references and in spec/.

Core validates before work starts or an event is written, so a typo cannot silently change an eval and a reporter never receives an invalid value (value rules). Config is data, not state. A run takes config, a JSON object that RunStarted records with a hash and that the CLI groups and compares by. trialkit records, groups, diffs and gates configs but does not store them, so a team keeps its config where it already does.

trialTimeoutMs has a finite default (reference) because adapters turn off the runner’s own timeouts, and a body that returns after its deadline times out, even if it blocked the timer. A reporter gets at least that default to finish, because a short trial limit says nothing about how long a file or network reporter needs. A guard, the token budget or a signal stops new work, and stoppedBy records the first cause, only when it changed what ran. Running trials finish, so a budget can overshoot by about one trial per active case.

Core records finished agent operations as data. It imports no OpenTelemetry and reads no global tracer, so test collection stays apart from production tracing and core keeps no dependency. Agent traces use the span list in core, not a new case type or package. The toolCalls scorer caps its lists because its search under order any grows with their cube. A missing span list is misuse, not zero calls, and recorded order does not prove that a model used a tool result (spans, logs).

The test form stays comparable through the input that run gets (CaseCompleted.inputHash), not through a hash of a test’s source code. Labels record what no hash covers: expected values, assertions, task code, prompts and models. Cases come from inline data, readCases or casesFromRows, so a case source need not be a file (files, rows). YAML and TOML files are in @trialkit/cases with the parsers as optional peers, so core keeps no dependency.

One bounded classifier keeps unavailable infrastructure out of quality rates. An infrastructure value is an error: it leaves pass rates and may use errorRetries. Any other throw from a task, scorer, set check or test body is a quality fail, so a parser error on a model reply or an app TypeError lowers the pass rate. A 401 or a 404 is no answer of the model, so it is an error. Misuse is an error too, but not an infrastructure error. The TypeScript classifier reads structural names, not instanceof or SDK imports, so it works across realms and package copies; the shared rule lists the signals.

Both languages give a function the same name, in camelCase or snake_case. A few names exist in one language only, because of the language: Python has call_in_thread (JavaScript has no threads) and whole_number (JavaScript has one kind of number), and TypeScript has fromStream and the AI SDK converter spansOfAiSdk. Both provide an OpenTelemetry span converter.