Skip to content

trialkit-core

Run LLM eval cases as repeated trials, apply verdict rules and set checks, and record events. Python runs use the same JSON Lines format as @trialkit/core, so the trialkit CLI can report and compare them.

Terminal window
pip install trialkit-core
import asyncio
from trialkit.core import (
Dataset,
EvalCase,
EvalDefinition,
ScoreValue,
TrialContext,
console_reporter,
jsonl_file,
rate,
run_eval,
scorer,
)
CAPITALS = {"France": "Paris", "Japan": "Tokyo"}
@scorer(name="exact", version="1")
def exact(case: EvalCase[str, str], output: str, ctx: TrialContext) -> ScoreValue:
passed = output == case.expected
return ScoreValue(1.0 if passed else 0.0, passed, f"expected {case.expected}, got {output}")
def task(country: str) -> str: # a plain function runs in a worker thread
return CAPITALS.get(country, "")
definition = EvalDefinition(
name="capital-cities",
dataset=Dataset(
id="capitals-v1",
cases=[
EvalCase(id="france", input="France", expected="Paris"),
EvalCase(id="japan", input="Japan", expected="Tokyo"),
],
),
task=task,
scorers=[exact],
checks=[rate("pass rate").at_least(0.9)],
)
result = asyncio.run(
run_eval(definition, reporters=[console_reporter(), jsonl_file(dir="eval-results")])
)
for error in result.reporter_errors: # a reporter can fail without changing an outcome
print(f"reporter error: {error.name}: {error.message}")
raise SystemExit(0 if result.summary.ok and not result.reporter_errors else 1)
  • For ordinary pytest tests, use pytest-trialkit instead of building a definition.
  • Infrastructure problems and trialkit misuse are error, never quality failures. Other task, scorer and test-body exceptions are fail; see the error classification spec.
  • Requires Python 3.11 or later; import trialkit.core. The package has no dependencies and is not yet published.

The exported definitions are frozen dataclasses, except protocols and TrialContext. Optional policy fields accept None as their default. Integral floats such as 3.0 are accepted for integer options.

Name Options and defaults Meaning
EvalCase(id, input, expected=None, tags=(), metadata={}, skip=None, xfail=None) id and input required A case has a unique non-empty id. input, expected and metadata must be JSON; tags is a list or tuple of strings. skip and xfail, when set, are non-empty reasons. xfail changes fail to xfail and pass to xpass. None for tags or metadata means empty.
Dataset(*, cases, id=None) cases required; id defaults to the eval name A non-empty ordered sequence of cases. Its id is not part of the dataset hash.
EvalDefinition(name, dataset, task, scorers, *, trials=1, verdict="all", early_stop=True, error_retries=0, checks=()) First four required A task accepts (input) or (input, ctx); scorers have unique names. Policy fields are keyword-only. The old policy argument is rejected.
Task (input) or (input, ctx) A plain or async callable. Core passes ctx when the signature accepts both.
Scorer, scorer(*, name, version) name and version required A scorer has name, version and score(case, output, ctx). The decorator creates one from a plain or async function. Change its version when its logic changes.
ScoreValue(value, passed, reason=None, data=None) value and passed required A scorer value is 0–1; a set-check value is any finite number. passed is boolean and data must be JSON. An invalid result is error.
SetCheck, CasePredicate A check has name, evaluate(results) and optional where Check functions run synchronously in a worker thread. A predicate selects cases by truthiness.
rate(name, *, where=None, over="cases").at_least(threshold) threshold required, 0–1 over is "cases" or "trials". Use early_stop=False for a fixed trial count. rate(...) alone is not a check. tag(name) selects cases with that tag.
with_usage(task, usage) Both required Wrap a task and add the TokenUsage returned by usage(output), without changing the output.

Case metadata maps strings to strings, finite numbers or booleans.

Policy option Default Meaning
trials 1 Maximum counted pass/fail trials per case; integer ≥ 1.
verdict / Verdict "all" "all", "majority", or AtLeast(n) where 1 ≤ n ≤ trials.
early_stop True Stop when remaining trials cannot change the verdict.
error_retries 0 Extra trials for error only; errors do not count toward trials.
checks () Set checks evaluated after cases.

await run_eval(definition, ...) runs the dataset and returns RunResult. Adapter authors can import start_run and Run from trialkit.core.adapter to run cases individually. Both entry points validate the definition and options before starting; invalid types raise TypeError and out-of-range values raise ValueError.

Each definition-form trial receives fresh copies of exact built-in dict, list, and tuple containers in its case input. Other values, including other mappings and container subclasses, are shared by reference and are not traversed. A task or scorer that mutates a shared value changes the dataset and what later trials see. Input must still satisfy Python’s JSON rules; Python does not call toJSON.

Keyword option Default Meaning
reporters () Reporters for the events. Core alone writes no file.
concurrency 1 run_eval only: concurrent cases, an integer ≥ 1 or math.inf. A case’s trials stay sequential.
trial_timeout_ms 120_000 Trial deadline in milliseconds, 1–2,147,483,647. Timeout is error.
run_timeout_ms None Whole-run case-execution deadline in milliseconds, 1–2,147,483,647. It stops in-flight trials and new work, not set checks or reporter closing.
max_errors None Positive limit on error trials, including retries; stops in-flight and new work.
labels None String-to-string run labels.
config None JSON object copied to RunStarted with its canonical configHash. Keep secrets out.
filter None Case predicate; non-matching cases skip with filtered out.
partial_checks / PartialChecks "skip" "skip" or "fail" for checks with cases that did not run.
token_budget None Positive token limit. Once reached, no new trial starts; concurrent work can overshoot it. Report usage with ctx.add_usage.

Every optional keyword accepts None as its default. A guard preserves a verdict already decided and sets summary.stopped_by only when it stopped work. The causes include "tokenBudget", "runTimeout", "maxErrors", "signal" and "error"; the run is then not ok. A run with no case verdict is also not ok. A check partial because a case skipped while running makes the run not ok.

Run.id, await Run.run_case(case) and await Run.finish() are the adapter interface. A case can start only once; finish() is idempotent, evaluates checks and closes reporters. A case still running at finish starts no new trial and its result is not recorded. run_eval finishes even when filter raises, then reraises; cancel the awaiting asyncio task to stop a run. Cancellation propagates and the record is marked signal.

The pytest adapter runs each test body as a case’s trials. run(input) calls the task; score(scorer, output, expected=None) records a score and raises AssertionError when it fails. Each pass or fail also records assert@1; assert is reserved. Await both calls in an async body. An unawaited or unfinished call, a non-JSON input, an invalid scorer result or duplicate score name ends the trial in error, even if the body catches the mistake. A sync body uses the adapter’s blocking thread methods. See the pytest reference.

Python run(input) returns a coroutine and starts the task when it is awaited; TypeScript starts the task at the call. Python score(...) starts when called. Await each result before the body returns.

Each task call records a run attachment with a copy of its input and, when it returned, its output. A non-JSON output is recorded as descriptive text. If the task changed its input into something that is not JSON, the call is misuse when the task returned; when the task raised, the input is recorded as not JSON: <reason> and its error stays the result. The first input gives the case’s input_hash and must stay the same across trials; later calls in a trial can be other steps.

Member Meaning
case_id, trial Case id and trial number, starting at 1.
trace_id String trace id, or None to clear it. Other values are misuse.
attach(name, value) Snapshot a JSON value and record it. run is reserved; convert dataclasses and other non-JSON values before calling.
log(message, *, data=None) Snapshot optional JSON data and emit a bounded trial log.
add_usage(input_tokens=None, output_tokens=None, *, cached_input_tokens=None, reasoning_tokens=None, cost=None) Add finite non-negative counts and USD cost. Counts round half up. None is not given; an empty call records nothing. Missing input/output counts become zero when another field is given.
metric(name, value) Record a finite number under a string name; the last value of a name wins. A non-string name is misuse.
span(span), spans Record finished agent operations; spans is an immutable tuple in record order.
skip(reason) End the case as skipped unless its verdict is already decided. The reason must be a string; other values are misuse. Usage before the skip still counts.

Task and scorer usage are included in totals; usage_by separates them when a scorer reports usage. Context methods are thread-safe and do nothing after the trial ends. skip raises a BaseException, also after the trial ended. Python has no context signal: cancellation reaches the asyncio task. ctx.log("Retrieved documents", data={"count": 3}) records trialkit.log.v1 attachments with a one-based sequence and monotonic trial-relative offset. Each compact payload is at most 8 KiB; a trial accepts at most 1,000 entries and 256 KiB total. Log data is serialized once; non-JSON values and non-finite numbers are misuse rather than silently omitted or changed. Invalid or excess input is misuse. Log data can contain secrets; redact it before recording.

Span and SpanAttributes use event keys name, spanId, attributes, status and optional parentSpanId, start, end. Names are non-empty, ids are 1–64 Unicode code points and unique per trial, and a parent must already be recorded. Status is "ok" or "error"; times are finite, non-negative milliseconds since trial start, with start <= end. A trial accepts at most 1,000 spans and 1,048,576 compact UTF-8 bytes. Attributes are JSON; values have at most 496 levels. An invalid span is misuse.

Span snapshots and results that contain them support copy.copy, copy.deepcopy and pickle. Deep copies and unpickled snapshots use ordinary dicts and lists; changing a copy does not change the original snapshot.

tool_calls(spans) projects execute_tool spans to ToolCall entries with name, arguments, hasArguments, result, hasResult, spanId, index and optional error=True. An absent attribute and a JSON null both read None; hasArguments and hasResult say whether the span has the attribute. model_calls(spans) projects chat and generate_content spans to ModelCall entries with name, model, spanId and index. Both preserve order. Spans do not add token usage.

trialkit.core.otel.spans_of(readable_spans, *, origin_ns=None) converts explicitly collected, finished spans from one trace. ReadableSpan, SpanContext, SpanStatus and SpanStatusCode are structural protocols; no OpenTelemetry SDK or global provider is required. With origin_ns, times are relative milliseconds; without it, times are omitted. Parents are ordered first and missing external parents are detached. Resources, events and links are ignored.

The importer decodes JSON strings only for gen_ai.input.messages, gen_ai.output.messages, gen_ai.system_instructions, gen_ai.tool.definitions, gen_ai.tool.call.arguments and gen_ai.tool.call.result. Other strings and malformed JSON remain text. Captured messages and tool data may contain secrets.

Plain tasks and scorers, and set checks, run in daemon worker threads; async tasks and scorers run on the event loop. A timed-out thread can continue, but its result is discarded and it cannot keep the process alive. Set SDK timeouts. Do not call a blocking SDK directly inside async def: it blocks the trial timer and cannot be interrupted if it never returns. A task, scorer or test body that settles after the deadline ends in error with trial timed out after <n> ms, even if it blocked the timer. asyncio.to_thread uses the loop’s default executor, which can delay loop shutdown; use it only for bounded work.

Trial outcomes are pass, fail, error; case outcomes add skip, xfail, xpass; check outcomes add skip. See TypeScript outcomes for the shared verdict rules. An error is not in a pass rate. Infrastructure errors include timeouts, connection errors and HTTP 400–599 responses; InfrastructureError brands your own infrastructure exception. is_infrastructure_error(value) applies the shared bounded cause-chain rule.

The infrastructure walk also follows BaseExceptionGroup.exceptions, including ExceptionGroup from asyncio.TaskGroup, under the same depth and value bounds as linked errors.

RunResult contains run_id, eval_name, dataset_id, dataset_hash, ordered cases, checks, summary and reporter_errors. Result types are CaseResult, TrialResult, CheckResult, RunSummary, Score, Attachment, TokenUsage, UsageBy, ErrorInfo, TrialOutcome, CaseOutcome and CheckOutcome. CaseResult.input_hash identifies the input. TrialResult.exception is None for run_eval and start_run; only adapter runs retain exceptions for tracebacks. It is not an event. RunSummary contains case and check counts, ok, duration_ms, usage, usage_by and stopped_by. A skipped case retains counts from earlier trials.

Result type Constructor fields and defaults
TokenUsage input_tokens=0, output_tokens=0, cached_input_tokens=None, reasoning_tokens=None, cost=None
UsageBy task=None, scorers=None
ErrorInfo Required name, message; stack=None, cause=None
Attachment Required name, value
Score Required name, version, value, passed; reason=None, data=None
TrialResult Required trial, outcome, scores, duration_ms; usage=None, metrics=None, error=None, trace_id=None, attachments=(), spans=(), exception=None, usage_by=None
CaseResult Required case_id, tags, outcome, trials, passes, failures, errors; reason=None, input_hash=None
CheckResult Required name, outcome; value=None, reason=None, data=None
RunSummary Required cases, checks, ok, duration_ms, usage; stopped_by=None, usage_by=None
RunResult All fields listed above are required.

Inputs, expected values, config, attachment values and score data must be JSON: None, booleans, finite numbers, strings, lists or tuples, and dicts with string keys. Case metadata is a dict of string, finite-number or boolean values. A dataset or config cannot contain lone surrogates; attachments and score data can, and the writer escapes them. Numbers use ECMAScript double semantics, including rounding large integers. Invalid values fail where supplied, rather than cutting off the record.

The shared JSON line limit is 500 nested arrays/objects. A case field permits 498 levels; config, attachment and set-check data permit 499; scorer data permits 497. canonical_json itself permits 500.

Events are dicts with the camelCase fields of the event schema. Exports: Event, RunStarted, CaseStarted, TrialStarted, TrialCompleted, AttachmentAdded, CaseCompleted, CheckCompleted, RunCompleted, ScoreEvent, ErrorEvent and SCHEMA_VERSION. The library version appears in RunStarted.library.

Export Options and defaults Meaning
Reporter on_event(event), optional close() Methods can be async. Events are ordered; a slow reporter does not block the run. A reporter error is recorded once in RunResult.reporter_errors, stops its events, and never changes a verdict.
console_reporter(*, write=None, verbose=False) write=None prints to the console Print a run summary; verbose=True includes passing cases and bounded trial logs.
jsonl_file(*, dir=None, path=None) Give exactly one of dir or path Write one JSON event per line. dir makes a file per run; path must not already exist.

Event times come from a monotonic clock anchored at one wall-time origin (perf_counter_ns plus a captured wall-time offset), so moving or freezing the calendar clock does not reorder events. With dir, jsonl_file writes <slug>.<run id>.jsonl, where the slug is the eval name with each run of characters other than Unicode letters, digits, _, . and - replaced by -, cut to 100 characters, as in TypeScript. The run id keeps each name unique, so pytest-xdist workers never share a file.

wilson(passes, n) returns an Interval(low, high); rate_difference(a, b) takes Rate(passes, n) values and returns RateDifference(difference, low, high); pass_rates(results) returns PassRates(cases, trials). They match TypeScript and validate counts. dataset_hash(dataset) returns sha256:<hex> of the canonical case JSON, omitting absent optional fields and empty tags or metadata; equivalent Python and TypeScript datasets hash alike.

read_cases(path) reads UTF-8 .jsonl or .csv into ordered EvalCase values. JSON Lines requires id and input and accepts expected, tags, metadata, skip and xfail. CSV requires id,input; optional expected and tags are strings, tags use ;, and other columns become string metadata. Empty expected cells are absent. Invalid or empty files and duplicate or empty ids raise TypeError with the path and physical line. See the case file contract.

cases_from_rows(rows, *, source, lines=None) validates case rows that your app already has, for example from a database or an API, and returns EvalCase values in row order. It reads no file and does not change rows: a case is a new EvalCase, and the values inside it are shared with its row. It applies the case, id, metadata and JSON rules of read_cases; an optional field that is None counts as not given.

Argument Default Meaning
rows Required A list of dicts with string keys.
source Required Non-empty str that says where the rows came from: a file path, tickets-api or a table name.
lines None A sequence with one positive int per row: the line of the row in the source. Without it, messages give a row number.
from trialkit.core import Dataset, cases_from_rows
rows = [{"id": "refund", "input": "Refund my order.", "expected": "billing", "tags": None}]
dataset = Dataset(id="tickets", cases=cases_from_rows(rows, source="tickets table"))

A problem raises TypeError: <source>:<line>: <reason> with lines, or <source>: row <n>: <reason> without them (n counts from 1). rows that is not a list raises <source>: cases must be an array, and an empty list raises <source>: no cases. See the shared contract.

read_runs(paths, *, attachments=None, run_ids=None) returns an iterator of FileRuns(file, runs, problems). Paths are a list or tuple of files or directories; directories are searched recursively for .jsonl. Files sort by UTF-16 path order and real paths are read once. An explicitly named missing path raises FileNotFoundError. Directory symlinks found during traversal are not followed.

Each ReadRun has run_id, events, attachments and complete; ReadEvent is its event type. Attachments are omitted unless their names are requested. run_ids selects a list of non-empty run ids; None selects all, and an empty list selects none. The reader silently omits unselected runs and their attachments before it checks event fields, but still reports invalid JSON and lines without a usable run id. ReadProblem(reason, line=None) explains omitted content; unreadable files become problems. The reader validates required fields, not field types or values. Its line, schema, duplicate-run and error rules match readRuns.

Import these from trialkit.core.util, not from the package root. Their shared behavior matches the TypeScript utilities, except that error_message(TimeoutError()) returns "TimeoutError" while TypeScript errorMessage(new Error("")) returns "".

Export Meaning
canonical_json(value) Canonical JSON with UTF-16 key ordering and ECMAScript numbers; rejects non-JSON, lone surrogates and depth over 500.
sha256_hex(text) SHA-256 of UTF-8 text as 64 lowercase hex digits; rejects lone surrogates.
trim_text(text) Remove ECMAScript whitespace, including U+FEFF but not U+0085 or U+001C–U+001F.
json_levels(value) The levels of lists and dicts in a JSON value: none for a scalar, one for [], two for [[]]; it does not recurse.
MAX_CASE_FILE_DEPTH, case_row_too_deep(row) Case-file depth limit (64) and whether a row exceeds it, including the document and its cases array.
trim_text(text) Remove ECMAScript whitespace, including U+FEFF but not U+0085 or U+001C–U+001F.
describe_value(value) Total value description for messages, clipped to 200 characters.
error_message(error) Total error message; uses the class name for an empty exception message.
is_plain_object(value) True for a mapping with string keys.
case_row(case) The row of an EvalCase, as cases_from_rows reads it: only the fields that have a value. No TypeScript twin.
js_number(value) ECMAScript number text.
whole_number(value) Integral int or float as an int, otherwise None; excludes booleans.
check_paths(who, paths) Validate a list or tuple of non-empty string or path values.
validate_spans(value, label="ctx.span") Validate a span list and return immutable snapshots, or raise a misuse error that starts with label.
await call_in_thread(function) Run a plain function in a daemon thread without blocking the event loop.

trialkit.core.adapter is runner-independent. pytest-trialkit uses it; the shared behavior matches the TypeScript runner adapters.

start_run(definition, ...) starts a run for an adapter. Its Run exposes id, run_case and finish.

Export Options and defaults Meaning
create_suite(definition, *, name=None, reporters=(), trial_timeout_ms=120_000, run_timeout_ms=None, max_errors=None, concurrency=1, labels=None, config=None, filter=None, partial_checks="skip", token_budget=None) name defaults to eval name; other options match run_eval Validate the definition and prepare a Suite; start the run when a test first runs. Case ids must be unique and cannot be set checks when checks exist.
Suite, SuiteTest, TestReport TestReport(status, message="", warning=None) suite.tests lists case tests and optional set checks; suite.select(names) supplies the runner’s selection. await test.run() reports pass/skip or raises for failure. Repeated calls return the first result. await suite.close() returns failure lines.
eval_settings(name, *, task, trials=1, verdict="all", early_stop=True, error_retries=0, checks=(), reporters=(), trial_timeout_ms=120_000, run_timeout_ms=None, max_errors=None, labels=None, config=None, filter=None, partial_checks="skip", token_budget=None, assertion_errors=(AssertionError,)) name and task required Validate test-form settings. Policy and run options match the tables above.
create_eval_suite(settings, cases, *, reporters=()) settings and (id, CaseMarks) cases required Make an EvalSuite with cases, optional checks and close(). await EvalTest.run(body) runs a body as trials.
CaseMarks(tags=(), metadata={}, skip=None, xfail=None) All optional Marks for a test-form case.
EvalTestContext, Body run(input), score(scorer, output, expected=None), skip(reason), log(message, *, data=None), span(span), spans Async body interface; sync bodies use run_from_thread and score_from_thread.
trial_counts(passes, failures, errors=0, needs=None) Last two optional Format counts for a runner.
named_cases(names), misuse(error) Required arguments Format a bounded list of case names; brand a trialkit misuse exception.
LOG_ATTACHMENT, is_log_entry(value), format_logs(attachments) Required arguments Recognize v1 log attachments and format bounded diagnostics for runner adapters.
EvalSettings, CHECKS_TEST, DEFAULT_TRIAL_TIMEOUT_MS, ASSERT_SCORE, ASSERT_VERSION CHECKS_TEST="set checks", DEFAULT_TRIAL_TIMEOUT_MS=120_000, ASSERT_SCORE="assert", ASSERT_VERSION="1" Settings type and adapter constants.

Run all tests and close() of one suite on the same event loop. Keep set checks last and disable runner timeouts and retries: a case may run several trials, and reporter closing may upload. close() reports reporter errors, stopped runs, no verdict, unexpected partial checks and a token budget without reported usage. It does not fail a deliberate selection of only set checks.