Skip to content

@trialkit/core

A dependency-free engine for LLM evals: it repeats each case, decides a verdict, keeps outages separate from quality failures, and emits JSON Lines events.

Terminal window
npm install --save-dev @trialkit/core
Requirement Value
Node.js 22.23.4 or later. The root entry also runs in Bun and Deno.
TypeScript 5.4 or later.
Dependencies None. File APIs are only in @trialkit/core/node.
import { defineEval, runEval } from '@trialkit/core';
const capitals = defineEval({
name: 'capital-cities',
dataset: { cases: [{ id: 'france', input: 'France', expected: 'Paris' }] },
task: (country) => (country === 'France' ? 'Paris' : ''),
scorers: [
{
name: 'exact',
version: '1',
score: ({ output, evalCase }) => {
const pass = output === evalCase.expected;
return { value: Number(pass), pass };
},
},
],
});
const result = await runEval(capitals);
if (!result.summary.ok || result.reporterErrors.length > 0) process.exitCode = 1;
  • TrialPolicy defaults to one trial and an all verdict; change it for variable model output.
  • An outage, timeout, or infrastructure error is an error; a wrong answer is a fail.
  • Add a Reporter to save the event stream; jsonlFile writes one JSON Lines file per run.

defineEval(definition) returns its argument unchanged so TypeScript infers I, O, and E. It does not validate; runEval and the adapter factories validate before a run starts. Adapter authors can import startRun and Run from @trialkit/core/adapter.

Export Contract
EvalDefinition<I, O, E> name, dataset, task, and at least one uniquely named Scorer; optional checks (none by default) and the TrialPolicy fields below.
Dataset<I, E> cases is a non-empty readonly list. id defaults to the eval name and does not change the dataset hash.
EvalCase<I, E> One unique, non-empty id, an input, and the optional fields below.
Task<I, O> (input, ctx) => O | Promise<O>; normally call the same application path that production uses.
Metadata Record<string, string | number | boolean>.
Scorer<I, O, E> A non-empty name and version, plus score(args). Change version when scoring logic changes.
ScoreArgs<I, O, E> evalCase, output, ctx, and optional spans for direct scorer calls. Core always supplies spans.
ScoreValue value, pass, optional reason, and optional JSON data. A scorer value is 0–1; a set-check value is any finite number.
Score A recorded ScoreValue with the scorer name and version.
SetCheck A unique name, optional where (all non-skipped cases by default), and evaluate(results). See Set checks.
CasePredicate (evalCase) => boolean; a falsy result excludes the case.
EvalCase field Default Meaning
expected None Value supplied to scorers.
tags None String tags; tag() selects them.
metadata None String, finite-number, or boolean labels.
skip None Non-empty reason; the case does not run.
xfail None Non-empty reason; a failed verdict becomes xfail, and a passing verdict becomes xpass.

Each definition-form trial receives fresh copies of plain objects and arrays whose prototype is exactly Array.prototype. Their own property descriptors, including non-enumerable and symbol properties, are preserved; data values are copied recursively, while getters and setters are retained without invoking them. Getter return values are not traversed, so an accessor can expose shared state. Other values, including array subclasses, class instances, Date, URL, and Buffer, are shared by reference and are not traversed. A task or scorer that mutates a shared value changes the dataset and what later trials see.

Member Contract
caseId, trial Case id and one-based trial number.
signal Aborts on run cancellation, trial timeout, or trial end. Pass it to work that can wait.
traceId Optional string recorded with the trial; assign undefined to clear it.
attach(name, value) Snapshots JSON data and emits an Attachment event. undefined becomes null.
log(message, data?) Snapshot optional JSON data and emit a bounded trial log. null and undefined omit data.
span(span), spans Add a validated finished span and read immutable recorded spans. See Agent spans.
addUsage(usage) Adds finite, non-negative token usage and USD cost. Counts are rounded.
metric(name, value) Records a finite number under a string name; the last value for a name wins.
skip(reason) Ends the open case as skip with a string reason; usage already added still counts.

Calls or assignments after a trial ends do nothing, except that skip still throws. TokenUsage has optional inputTokens, outputTokens, cachedInputTokens, reasoningTokens, and cost; UsageInput also accepts AI SDK inputTokenDetails.cacheReadTokens and outputTokenDetails.reasoningTokens; UsageBy separates optional task and scorers totals. Usage fields set to null are absent. addUsage rejects a non-object and a nonempty object with no recognized usage fields as misuse; an empty object records nothing. A nested detail must be an object (a class instance counts): core reads only cacheReadTokens and reasoningTokens and ignores its other keys, and a non-object detail is misuse. ctx.log('Retrieved documents', { count: 3 }) records trialkit.log.v1 attachments with a one-based sequence and monotonic trial-relative offset. Each compact payload is at most 8 KiB; a trial accepts at most 1,000 entries and 256 KiB total. Log data is serialized once; nested undefined, functions, symbols, sparse arrays and non-finite numbers are misuse rather than silently omitted or changed. Invalid or excess input is misuse. Log data can contain secrets; redact it before recording.

Verdict is 'all' | 'majority' | { atLeast: number }. A missing, undefined, or null policy field takes its default.

Field Default Meaning
trials 1 Maximum counted trials; integer at least 1.
verdict 'all' Required passes: all, more than half, or atLeast. atLeast is 1 through trials.
earlyStop true Stop when remaining trials cannot change the verdict.
errorRetries 0 Extra trials after error outcomes. Errors do not count toward trials.
  • runEval(definition, options?) runs every case, finishes checks and reporters, and returns Promise<RunResult>. Cases can run concurrently; trials of one case stay sequential.
  • StartRunOptions contains the options below except concurrency.
  • RunOptions adds concurrency; only runEval uses it.

undefined and null option fields mean “not supplied.” All options are checked before work starts.

StartRunOptions / RunOptions field Default Meaning
trialTimeoutMs 120000 Per-trial limit in milliseconds; 1 through 2147483647. Timeout is an error.
runTimeoutMs Off Whole-run case-execution limit in milliseconds; 1 through 2147483647.
maxErrors Off Stop after this positive number of error trials, including retries.
reporters None readonly Reporter[] that receive events.
filter All cases CasePredicate; nonmatches are skip. A throw stops the run and is rethrown by runEval.
signal None AbortSignal for the run. Running trials error; work prevented from starting makes the run incomplete.
labels {} String-valued run labels recorded by RunStarted.
config None Plain JSON object recorded with a hash; see The config of a run.
partialChecks 'skip' Outcome when a covered case did not run: 'skip' or 'fail'.
tokenBudget No budget Positive limit for input plus output tokens reported through addUsage. Running trials finish; later work does not start.
concurrency 1 RunOptions only: integer at least 1 or Infinity.

runCase accepts an optional case-only AbortSignal; it has no default signal. Guards preserve a verdict already decided. A stopped run sets RunSummary.stoppedBy to tokenBudget, runTimeout, maxErrors, signal, or error only when that cause changed what ran. A filter that throws inside runCase stops new work with stoppedBy: 'error', records no event for that case, and rethrows. A call that is rejected up front (a duplicate id, an id outside the dataset, a call after finish()) does not mark the run.

config records the settings tested; it does not apply or retain settings outside the event stream. It must be a plain JSON object; members such as undefined that JSON would discard are refused. RunStarted contains a snapshot as config and configHash (sha256: plus canonical JSON); keep secrets out of it. The CLI can group and compare this field.

TrialOutcome is pass | fail | error; CaseOutcome adds skip | xfail | xpass; CheckOutcome is pass | fail | error | skip.

Subject Outcome rule
Trial pass when every score passes; fail when a score fails or code throws a non-infrastructure value; error for infrastructure, timeouts, aborts, or trialkit misuse.
Case pass or fail when counted trials decide the policy; error when no verdict can be reached; skip for a mark, filter, budget, signal, or ctx.skip; xfail and xpass transform a decided verdict.
Set check error for no covered cases, a covered case error, invalid result, or infrastructure throw; skip or fail for a partial set as partialChecks says; otherwise its pass decides.

A non-infrastructure throw from a task, scorer, set check, or test body is a quality fail. An infrastructure signal or a misuse error is an error, never a pass or fail. A case may retain completed trial counts when it later skips. RunSummary.ok is false for a failure, error, early stop, no case verdict, or a partial check caused by a case that skipped while running.

A decided verdict stands, whatever ends the case later (error trials, a guard, an abort, finish() or ctx.skip). With verdict: 'all', trials: 3 and earlyStop: false, a fail followed by an error is a fail. One exception: a runner feature that core refuses ends the case in error after a decided verdict too. An open verdict (the counted trials have not decided it) takes its outcome from the cause:

Cause Outcome Reason
The error trials outnumber errorRetries, even when a guard also fired error The error of the last trial, <name>: <message>. An abort that ended the last retry shows as AbortError: ....
A run guard (runTimeoutMs, maxErrors) or the token budget keeps another trial from starting skip <cause> before the verdict: the cause is run timed out after <n> ms, error limit of <n> reached or token budget of <n> reached. A case that did not start has the cause alone.
The run signal aborts error run aborted before the verdict (stoppedBy is 'signal').
The signal of runCase aborts error case aborted before the verdict (no stoppedBy).
finish() ends the run while the case runs error run finished before the verdict. Nobody records that result.
A trial calls ctx.skip(reason) (pytest.skip(reason) in pytest) skip The reason of the call.
  • ; last error: <name>: <message> follows an abort or finish() reason only when the last recorded trial ended in error. A trial that failed adds none. An error that a later trial superseded is not in the reason: an error trial, a pass trial, then an abort gives the cause alone.
  • A skip after some trials keeps their counts (passes, failures and errors can be more than 0). The trial that called ctx.skip is not counted, has no TrialCompleted event and starts no later trial. Its tokens count toward summary.usage and tokenBudget. It never sets stoppedBy and is never an error trial.
  • The shared scenarios run-timeout.json, max-errors.json and skip-after-verdict.json pin these rules in both languages.

rate({ name, where?, over? }).atLeast(threshold) returns a SetCheck; tag(name) returns a CasePredicate for tagged cases.

RateOptions field Default Meaning
name Required Unique check name.
where All non-skipped cases Cases covered by the check.
over 'cases' 'cases' counts passing decided cases; 'trials' counts passing counted trials.
atLeast(threshold) Required Number from 0 through 1. The check data includes counts and the 95% Wilson interval.

SetCheck.evaluate receives only covered pass and fail cases. Covered error cases error the check; all xfail/xpass cases skip it; a partial set never passes over its subset.

A check runs after all cases of its eval. Core decides these outcomes before it calls evaluate, and the first row that applies wins:

Situation Outcome and reason
No case is covered: none matches where, or every match has a skip mark (left out by design) error, no cases match: a misconfigured check, not a quality result
A covered case has no result or ended in skip (a name filter such as -t <case id>, a shard, a runner skip, filter, ctx.skip, the token budget or, in Jest and node:test, a random order that runs the check first) skip, or fail with partialChecks: 'fail', for a CI job that requires every check; partial: <k> of <n> cases ran
A covered case ended in error error, <k> of <n> cases errored: its quality is unknown
Every covered case ended in xfail or xpass skip, every covered case is xfail or xpass

Otherwise evaluate gets the covered pass and fail cases (xfail and xpass do not count in a rate: their status is known and reported per case). A where that throws, or an evaluate that throws an infrastructure value, ends the check in error (the check threw: <message>). Any other value that evaluate throws is a fail, with its message as the reason. A returned result that is not a valid score ends in error. A check does not depend on test order: it reads the case results that the adapter recorded in the same process, so a check that runs too early is skip (partial).

Export Contract
infrastructureError(message, { cause? }) Creates a branded InfrastructureError for an outage that your code identifies, such as a client that reports a plain error for an HTTP failure.
isInfrastructureError(value) Safely detects the brand and known infrastructure shapes across realms and package copies.

The classifier recognizes timeouts, aborts, connection errors, exhausted SDK retries, network codes, TypeError: fetch failed, and integer HTTP status/statusCode 400–599 on an error or its response. It follows cause, lastError, and errors breadth-first for at most three links and 64 values. The exact shared list is in the error-classification contract.

Export Contract
RunResult runId, evalName, datasetId, datasetHash, ordered cases and checks, summary, and non-fatal reporterErrors.
RunSummary Outcome counts, ok, durationMs, aggregate usage, optional usageBy, and optional stoppedBy.
CaseResult caseId, tags, outcome, optional reason and input hash, trials, and pass/failure/error counts.
TrialResult Trial number, outcome, scores, duration, attachments, optional usage, spans, metrics, trace id, and ErrorInfo.
CheckResult Name, outcome, optional value, reason, and JSON data.
Attachment Name and JSON value.
ErrorInfo Error name, message, optional stack, and an optional cause chain.

Every TrialkitEvent has schemaVersion, runId, and ISO-8601 time. Events for one run begin with RunStarted, preserve each reporter’s order, and end with RunCompleted; concurrent cases can interleave.

Export Event or contract
RunStarted Eval, dataset id/hash/size, library version, resolved policy, labels, optional config and config hash.
CaseStarted Case id and tags.
TrialStarted Case id and trial number.
TrialCompleted TrialResult without attachments, plus case id.
AttachmentAdded Event type 'Attachment', with case id, trial, name, and value.
CaseCompleted Case outcome, optional reason/input hash, and counts.
CheckCompleted A CheckResult.
RunCompleted The RunSummary.
Reporter onEvent(event) and optional close(). close() runs once per run after its last event unless the event queue times out. A reporter error is recorded once and never changes an outcome.
SCHEMA_VERSION 1; bump only for a breaking event-shape change. Readers must ignore unknown fields.

Event times come from a monotonic clock anchored at one wall-time origin (performance.timeOrigin plus performance.now() in TypeScript), so moving or freezing the calendar clock does not reorder events. The event schema does not require summary.usage, so a reader must accept a RunCompleted without it, although RunSummary always has usage in TypeScript.

Each reporter has at most trialTimeoutMs, and never less than its 120 s default, to finish its queued events and close() after the run finishes. runTimeoutMs bounds case execution, not reporter closing. A timeout appears in reporterErrors; it does not change case outcomes or summary.ok. Inspect both summary.ok and reporterErrors when deciding whether a run succeeded.

Export or option Default Contract
ConsoleReporterOptions / consoleReporter({ write?, verbose? }) write: console.log; verbose: false Prints a run summary and non-passing cases. verbose also prints passing cases and bounded trial logs.
jsonl(sink) — Writes one JSON event plus newline through a LineSink. It keeps no per-run state.
LineSink close: absent write(line) and optional close(). A shared sink must tolerate one close per run.
Export Contract
ModelClient id is <provider>:<model>; generate(request, signal?) resolves to ModelReply.
ModelMessage role is system, user, or assistant; content is text.
ModelRequest Required messages; optional fields below.
ModelReply text, provider-reported model, optional parsed json, and optional TokenUsage.
fromFunction(id, generate) Makes a ModelClient from an application or test function.
ModelRequest field Default Meaning
jsonSchema None { name, schema } for a provider that supports structured output.
temperature None Provider request setting.
maxOutputTokens None Provider request setting.

fromStream(stream, collect) makes a Task from (input, ctx) => AsyncIterable<chunk>. It collects chunks in order, records firstChunkMs after the first chunk, stops reading on ctx.signal, asks the source to return without waiting for the answer (a stuck source may never give one), and calls collect(chunks) once at the end. A synchronous iterable is invalid.

withUsage(task, usage) wraps a Task, calls usage(output), records its UsageInput, and returns the unchanged output.

Span records a finished agent operation without an OpenTelemetry dependency or global tracer.

Export Contract
Span Required non-empty name, unique 1–64-character spanId, JSON attributes, and status: 'ok' | 'error'; optional parentSpanId, start, and end. Parent spans must be recorded first; times are non-negative trial-relative milliseconds.
SpanAttributes Readonly JSON key/value mapping, including GenAI semantic-convention keys.
toolCalls(spans) / ToolCall Projects execute_tool spans in record order with name, arguments, result, span id, index, and optional error flag. An absent attribute and a JSON null both read null; hasArguments and hasResult say whether the span has the attribute.
modelCalls(spans) / ModelCall Projects chat and generate_content spans in record order with name, model, span id, and index.

ctx.span snapshots valid input. A trial accepts at most 1000 spans and 1,048,576 UTF-8 bytes; a parent must precede its child. Captured tool arguments, results, and messages can be sensitive.

spansOfAiSdk(result) from @trialkit/core/ai-sdk converts completed AI SDK v6/v7 generateText or consumed streamText steps to spans. Await it, then record each result with ctx.span. It has no AI SDK runtime dependency; content capture is opt-in. Its name distinguishes this async result converter from the synchronous OpenTelemetry spansOf.

spansOf(readableSpans, options?) from @trialkit/core/otel converts explicitly collected, finished SDK spans. It reads no global provider and requires no OpenTelemetry package.

Export or option Default Contract
SpanContext — traceId and spanId.
ReadableSpan — Finished span fields: name, context, optional parent context, start/end tuples, status, attributes, and ended.
SpansOfOptions / origin Omitted origin is finite epoch milliseconds. With it, output start/end are trial-relative; without it, both are omitted.
spansOf — Requires one trace, validates ids/times/status, orders parents before children, detaches uncollected parents, and ignores resources, events, and links.
Export Contract
Interval { low, high }.
Rate { passes, n }.
RateDifference An Interval plus difference (a - b).
wilson(passes, n) 95% Wilson interval; n = 0 yields { low: 0, high: 1 }.
rateDifference(a, b) 95% Newcombe hybrid interval for independent rates; each n must be positive.
passRates(results) { cases, trials } for decided pass/fail cases only. rate() uses the same counts.

wilson and rateDifference throw a RangeError (Python: ValueError) for counts that are not whole numbers with 0 <= passes <= n. passRates counts the passing trials out of the counted trials of the decided cases; a case that ended in error, skip, xfail or xpass is in neither count, and neither are its trials. rateDifference takes square roots with Math.sqrt and not Math.hypot, so that Python gets the same bits (spec/fixtures/rate-difference.json).

Dataset fields, config, attachments, and score/check data must be JSON. Plain objects, arrays, strings, finite numbers, booleans, null, and toJSON values such as Date work. Classes, Map, bigint, non-finite numbers, cycles, and values over 500 levels do not. Dataset and config hashes also reject lone surrogates; attachments and score data serialize them. canonicalJson describes the shared canonical form.

casesFromRows(rows, options) validates case rows that your app already has, for example from a database or an API, and returns EvalCase<unknown, unknown>[] in row order. It reads no file and does not change rows: a case is a new object, and the values inside it are shared with its row. It applies the case, id, metadata, and JSON rules of readCases; an optional field that is null counts as not given.

options field Default Meaning
source Required Non-empty string that says where the rows came from: a file path, tickets-api, or a table name.
lines None One positive integer per row: the line of the row in the source. Without it, messages give a row number.
import { casesFromRows } from '@trialkit/core';
const cases = casesFromRows(await db.query('select * from eval_cases'), {
source: 'eval_cases table',
});

A problem throws a TypeError: <source>:<line>: <reason> with lines, or <source>: row <n>: <reason> without them (n counts from 1). rows that is not an array throws <source>: cases must be an array, and an empty array throws <source>: no cases. See the shared contract.

The @trialkit/core/node subpath uses file APIs and works in Node.js, Bun, and Deno.

Export Contract
RunFileOptions Exactly one non-empty { dir } or { path }. A directory produces one exclusive file per run; a path must not exist.
jsonlFile(options) Reporter that writes one JSON Lines file per run.
runFileOpener(options, reporter, extension) Validates file options once and returns an exclusive FileHandle opener for a custom file reporter.
withDefaultReporters(options) Adds jsonlFile({ dir: 'eval-results' }) only when reporters is absent or undefined; reporters: [] stays empty.

A directory writes <slug>.<run id>.<extension>. The slug is the eval name with each run of characters other than Unicode letters, digits, _, . and - replaced by -, cut to 100 code points: the run id already makes the name unique, and long names hit path limits.

readCases(path) reads one UTF-8 .jsonl or .csv file into ordered EvalCase<unknown, unknown>[]; it has no options. JSON Lines accepts id, input, expected, tags, metadata, skip, and xfail. CSV requires id and input; expected and semicolon-separated tags are optional, and other columns become string metadata. Invalid, empty, or duplicate rows throw a TypeError with path, physical line, and reason.

readRuns(paths, options?) returns an AsyncIterable<FileRuns>. It accepts non-empty file paths and recursively scans .jsonl files below named directories, reads each physical file once, and reports bad lines rather than stopping the history scan.

Export Contract
FileRuns file, fresh runs, and problems for one file.
ReadRun runId, non-attachment events beginning with RunStarted, selected attachments, and complete.
ReadEvent Every TrialkitEvent except AttachmentAdded.
ReadProblem reason and optional one-based line.
ReadRunsOptions Options below.
ReadRunsOptions field Default Meaning
check None (event) => reason | undefined; rejected events become line problems.
attachments None Attachment names to retain; unrequested attachments are omitted.
runIds All Non-empty run ids to include; an empty list selects none.

The reader accepts a BOM, \n, \r\n, or \r line endings, and blank space/tab lines. It keeps a run only when RunStarted is first, records no event after RunCompleted, and marks a run incomplete when its final event is missing.

Unselected run ids are omitted silently before event-field validation, including their attachments. Invalid JSON and lines without a usable run id still produce problems.

The runtime-neutral @trialkit/core/util subpath exports these helpers; the root entry does not.

Export Contract
canonicalJson(value) Canonical JSON with UTF-16-sorted keys and ECMAScript number formatting; rejects non-canonical JSON values.
sha256Hex(text) SHA-256 of UTF-8 text as 64 lowercase hexadecimal characters.
Html HTML already marked safe; .text and toString() return its markup.
html Template tag that escapes &, <, >, ", and ' in interpolated text; joins arrays and leaves Html fragments intact.
raw(text) Marks trusted text as Html without escaping it. Never pass untrusted data.
trimText(text) Remove ECMAScript whitespace, including U+FEFF but not U+0085 or U+001C–U+001F.
jsonLevels(value) The levels of arrays and objects in a JSON value: none for a scalar, one for [], two for [[]]; it does not recurse.
MAX_CASE_FILE_DEPTH, caseRowTooDeep(row) Case-file depth limit (64) and whether a row exceeds it, including the document and its cases array.
describeValue(value) Total, JSON-like error text clipped to 200 characters. A value of any depth gives the start of its JSON text.
errorMessage(value) A thrown value’s string message, or describeValue(value); safe across realms.
misuseError(message, { cause? }) TypeError for invalid dataset/options discovered by a scorer; core records it as an error.
isPlainObject(value) True for object literals, null-prototype objects, and matching cross-realm objects; false for arrays, classes, and built-ins.
validateSpans(value, label?) Validate a span list and return immutable snapshots, or throw a misuse error that starts with label (default ctx.span).
LOG_ATTACHMENT, LogEntry, isLogEntry(value) The v1 log attachment name, payload type and payload validator.
formatLogs(attachments) Format valid log attachments as bounded diagnostic lines.
MAX_LOGS, MAX_LOG_BYTES Maximum per-trial log entries and their combined JSON payload bytes: 1,000 and 262,144 UTF-8 bytes.

For adapter authors. @trialkit/core/adapter is runner-neutral. Declare one runner test for every SuiteTest in order, call select() before a test starts when the runner knows selection, await each run(), map its TestReport, and call close() from an unbounded after-tests hook. Disable runner retries, runner timeouts, and test reordering; trialkit owns repeats, per-trial timeouts, and the final set-check test.

startRun(definition, options?) starts a run for an adapter. Its Run<I, E> exposes id, runCase and idempotent finish.

Export Contract
createSuite(definition, options = {}, caller = createSuite) Creates a definition-form Suite, validates at declaration, and starts the run at the first test.
createTestSuite(name, options, declare, caller = createTestSuite) Creates a test-form Suite; declare receives EvalIt.
Suite name, ordered tests, select(names), and close(). close() rejects for reporter errors, incomplete runs, unchecked failures, or partial checks caused by running skips.
SuiteTest Case id or set checks, optional mode: 'only' | 'skip', and one-shot run(signal?, hooks?).
TestReport { status: 'pass'; warning? }, { status: 'skip'; message }, or { status: 'fail'; error }, optionally with case outcome, trial-count text and bounded logs for a failed case.
createTrialHooks(options) Builds shared assertion-count and snapshot checks; accepts runner state, assertion reset, optional trial binding and runner-specific refusal.

For createTrialHooks, getState() returns the runner’s expectedAssertionsNumber, isExpectingAssertions and snapshotState; resetAssertions() clears assertion-count expectations. Optional withinTrial binds the matcher to a trial, and optional refused() checks runner-specific features first. Both optional hooks default to absent.

Case or check outcome Runner action
pass Pass.
fail or error Fail; an error uses an infrastructure error.
skip or xfail Skip with its reason.
xpass Pass with a warning.
Set checks Fail for a failed or errored check, skip when all skip, otherwise pass (with a warning for partial skips).

SuiteOptions is StartRunOptions without signal, plus name and concurrency. createSuite itself has no default reporter; call withDefaultReporters when the adapter wants the standard file output.

SuiteOptions field Default Meaning
name Eval name Runner suite name.
reporters None Event reporters; withDefaultReporters changes an omitted value to eval-results/ JSON Lines output.
trialTimeoutMs 120000 Per-trial milliseconds.
runTimeoutMs Off Whole-run case-execution deadline.
maxErrors Off Positive error-trial limit.
concurrency 1 Definition-form case concurrency; integer at least 1 or Infinity.
labels {} String run labels.
config None Recorded JSON configuration.
partialChecks 'skip' Partial check outcome.
tokenBudget No budget Input-plus-output token cap.
filter All cases Case predicate.

EvalOptions<I, O> powers createTestSuite: it is SuiteOptions without concurrency, plus required task, optional checks (default none), and the TrialPolicy fields with their defaults above. CaseMarks has optional tags, metadata, skip, and xfail, all defaulting to absent. EvalTestBody<I, O> receives EvalTestContext<I, O>, which is a TrialContext plus run(input) and score(scorer, output, expected?). score takes the output type of its scorer, so a body can score evidence that it builds from the result of run. Both promises must be awaited.

In TypeScript, run(input) starts the task at the call. In Python, run(input) returns a coroutine and starts the task when awaited. Both require awaiting the result.

Export Contract
EvalIt<I, O> Declares it(name, body), marks, for(rows), only, and skip; names become case ids.
EvalTestContext<I, O> run calls the task and records its input/output; score records a score and rejects on a failed score.
EvalTestBody<I, O> One body invocation per trial. A returning body passes; throws use the shared outcome rule.
MatcherContext Adapter matcher bridge: score follows the same pipeline; reject creates a misuse error.
TrialHooks Optional withinTrial(matcher, body) and afterTrial(); no hooks means core invokes the body directly.
RefusedFeature 'expect.soft' | 'expect.assertions' | 'snapshot'; return one from afterTrial to end the case in error.

createEvalMatchers() from @trialkit/core/node creates an opt-in toPass matcher and its withinTrial hook. Call it once per adapter copy so concurrent and timed-out trials keep separate matcher contexts.

The Python twin is trialkit.core.adapter; its runner-specific integration is documented in pytest-trialkit.