@trialkit/core
A dependency-free engine for LLM evals: it repeats each case, decides a verdict, keeps outages separate from quality failures, and emits JSON Lines events.
Install
Section titled “Install”npm install --save-dev @trialkit/core| Requirement | Value |
|---|---|
| Node.js | 22.23.4 or later. The root entry also runs in Bun and Deno. |
| TypeScript | 5.4 or later. |
| Dependencies | None. File APIs are only in @trialkit/core/node. |
Example
Section titled “Example”import { defineEval, runEval } from '@trialkit/core';
const capitals = defineEval({ name: 'capital-cities', dataset: { cases: [{ id: 'france', input: 'France', expected: 'Paris' }] }, task: (country) => (country === 'France' ? 'Paris' : ''), scorers: [ { name: 'exact', version: '1', score: ({ output, evalCase }) => { const pass = output === evalCase.expected; return { value: Number(pass), pass }; }, }, ],});
const result = await runEval(capitals);if (!result.summary.ok || result.reporterErrors.length > 0) process.exitCode = 1;TrialPolicydefaults to one trial and anallverdict; change it for variable model output.- An outage, timeout, or infrastructure error is an
error; a wrong answer is afail. - Add a
Reporterto save the event stream;jsonlFilewrites one JSON Lines file per run.
API reference
Section titled “API reference”Definitions
Section titled “Definitions”defineEval(definition) returns its argument unchanged so TypeScript infers I, O, and E. It does not validate; runEval and the adapter factories validate before a run starts. Adapter authors can import startRun and Run from @trialkit/core/adapter.
| Export | Contract |
|---|---|
EvalDefinition<I, O, E> |
name, dataset, task, and at least one uniquely named Scorer; optional checks (none by default) and the TrialPolicy fields below. |
Dataset<I, E> |
cases is a non-empty readonly list. id defaults to the eval name and does not change the dataset hash. |
EvalCase<I, E> |
One unique, non-empty id, an input, and the optional fields below. |
Task<I, O> |
(input, ctx) => O | Promise<O>; normally call the same application path that production uses. |
Metadata |
Record<string, string | number | boolean>. |
Scorer<I, O, E> |
A non-empty name and version, plus score(args). Change version when scoring logic changes. |
ScoreArgs<I, O, E> |
evalCase, output, ctx, and optional spans for direct scorer calls. Core always supplies spans. |
ScoreValue |
value, pass, optional reason, and optional JSON data. A scorer value is 0–1; a set-check value is any finite number. |
Score |
A recorded ScoreValue with the scorer name and version. |
SetCheck |
A unique name, optional where (all non-skipped cases by default), and evaluate(results). See Set checks. |
CasePredicate |
(evalCase) => boolean; a falsy result excludes the case. |
EvalCase field |
Default | Meaning |
|---|---|---|
expected |
None | Value supplied to scorers. |
tags |
None | String tags; tag() selects them. |
metadata |
None | String, finite-number, or boolean labels. |
skip |
None | Non-empty reason; the case does not run. |
xfail |
None | Non-empty reason; a failed verdict becomes xfail, and a passing verdict becomes xpass. |
Each definition-form trial receives fresh copies of plain objects and arrays whose prototype is exactly Array.prototype. Their own property descriptors, including non-enumerable and symbol properties, are preserved; data values are copied recursively, while getters and setters are retained without invoking them. Getter return values are not traversed, so an accessor can expose shared state. Other values, including array subclasses, class instances, Date, URL, and Buffer, are shared by reference and are not traversed. A task or scorer that mutates a shared value changes the dataset and what later trials see.
TrialContext
Section titled “TrialContext”| Member | Contract |
|---|---|
caseId, trial |
Case id and one-based trial number. |
signal |
Aborts on run cancellation, trial timeout, or trial end. Pass it to work that can wait. |
traceId |
Optional string recorded with the trial; assign undefined to clear it. |
attach(name, value) |
Snapshots JSON data and emits an Attachment event. undefined becomes null. |
log(message, data?) |
Snapshot optional JSON data and emit a bounded trial log. null and undefined omit data. |
span(span), spans |
Add a validated finished span and read immutable recorded spans. See Agent spans. |
addUsage(usage) |
Adds finite, non-negative token usage and USD cost. Counts are rounded. |
metric(name, value) |
Records a finite number under a string name; the last value for a name wins. |
skip(reason) |
Ends the open case as skip with a string reason; usage already added still counts. |
Calls or assignments after a trial ends do nothing, except that skip still throws. TokenUsage has optional inputTokens, outputTokens, cachedInputTokens, reasoningTokens, and cost; UsageInput also accepts AI SDK inputTokenDetails.cacheReadTokens and outputTokenDetails.reasoningTokens; UsageBy separates optional task and scorers totals.
Usage fields set to null are absent. addUsage rejects a non-object and a nonempty object with no recognized usage fields as misuse; an empty object records nothing. A nested detail must be an object (a class instance counts): core reads only cacheReadTokens and reasoningTokens and ignores its other keys, and a non-object detail is misuse.
ctx.log('Retrieved documents', { count: 3 }) records trialkit.log.v1 attachments with a one-based sequence and monotonic trial-relative offset. Each compact payload is at most 8 KiB; a trial accepts at most 1,000 entries and 256 KiB total. Log data is serialized once; nested undefined, functions, symbols, sparse arrays and non-finite numbers are misuse rather than silently omitted or changed. Invalid or excess input is misuse. Log data can contain secrets; redact it before recording.
TrialPolicy
Section titled “TrialPolicy”Verdict is 'all' | 'majority' | { atLeast: number }. A missing, undefined, or null policy field takes its default.
| Field | Default | Meaning |
|---|---|---|
trials |
1 |
Maximum counted trials; integer at least 1. |
verdict |
'all' |
Required passes: all, more than half, or atLeast. atLeast is 1 through trials. |
earlyStop |
true |
Stop when remaining trials cannot change the verdict. |
errorRetries |
0 |
Extra trials after error outcomes. Errors do not count toward trials. |
Running and options
Section titled “Running and options”runEval(definition, options?)runs every case, finishes checks and reporters, and returnsPromise<RunResult>. Cases can run concurrently; trials of one case stay sequential.StartRunOptionscontains the options below exceptconcurrency.RunOptionsaddsconcurrency; onlyrunEvaluses it.
undefined and null option fields mean “not supplied.” All options are checked before work starts.
StartRunOptions / RunOptions field |
Default | Meaning |
|---|---|---|
trialTimeoutMs |
120000 |
Per-trial limit in milliseconds; 1 through 2147483647. Timeout is an error. |
runTimeoutMs |
Off | Whole-run case-execution limit in milliseconds; 1 through 2147483647. |
maxErrors |
Off | Stop after this positive number of error trials, including retries. |
reporters |
None | readonly Reporter[] that receive events. |
filter |
All cases | CasePredicate; nonmatches are skip. A throw stops the run and is rethrown by runEval. |
signal |
None | AbortSignal for the run. Running trials error; work prevented from starting makes the run incomplete. |
labels |
{} |
String-valued run labels recorded by RunStarted. |
config |
None | Plain JSON object recorded with a hash; see The config of a run. |
partialChecks |
'skip' |
Outcome when a covered case did not run: 'skip' or 'fail'. |
tokenBudget |
No budget | Positive limit for input plus output tokens reported through addUsage. Running trials finish; later work does not start. |
concurrency |
1 |
RunOptions only: integer at least 1 or Infinity. |
runCase accepts an optional case-only AbortSignal; it has no default signal. Guards preserve a verdict already decided. A stopped run sets RunSummary.stoppedBy to tokenBudget, runTimeout, maxErrors, signal, or error only when that cause changed what ran. A filter that throws inside runCase stops new work with stoppedBy: 'error', records no event for that case, and rethrows. A call that is rejected up front (a duplicate id, an id outside the dataset, a call after finish()) does not mark the run.
The config of a run
Section titled “The config of a run”config records the settings tested; it does not apply or retain settings outside the event stream. It must be a plain JSON object; members such as undefined that JSON would discard are refused. RunStarted contains a snapshot as config and configHash (sha256: plus canonical JSON); keep secrets out of it. The CLI can group and compare this field.
Outcomes
Section titled “Outcomes”TrialOutcome is pass | fail | error; CaseOutcome adds skip | xfail | xpass; CheckOutcome is pass | fail | error | skip.
| Subject | Outcome rule |
|---|---|
| Trial | pass when every score passes; fail when a score fails or code throws a non-infrastructure value; error for infrastructure, timeouts, aborts, or trialkit misuse. |
| Case | pass or fail when counted trials decide the policy; error when no verdict can be reached; skip for a mark, filter, budget, signal, or ctx.skip; xfail and xpass transform a decided verdict. |
| Set check | error for no covered cases, a covered case error, invalid result, or infrastructure throw; skip or fail for a partial set as partialChecks says; otherwise its pass decides. |
A non-infrastructure throw from a task, scorer, set check, or test body is a quality fail. An infrastructure signal or a misuse error is an error, never a pass or fail. A case may retain completed trial counts when it later skips. RunSummary.ok is false for a failure, error, early stop, no case verdict, or a partial check caused by a case that skipped while running.
When a case stops
Section titled “When a case stops”A decided verdict stands, whatever ends the case later (error trials, a guard, an abort, finish() or ctx.skip). With verdict: 'all', trials: 3 and earlyStop: false, a fail followed by an error is a fail. One exception: a runner feature that core refuses ends the case in error after a decided verdict too. An open verdict (the counted trials have not decided it) takes its outcome from the cause:
| Cause | Outcome | Reason |
|---|---|---|
The error trials outnumber errorRetries, even when a guard also fired |
error |
The error of the last trial, <name>: <message>. An abort that ended the last retry shows as AbortError: .... |
A run guard (runTimeoutMs, maxErrors) or the token budget keeps another trial from starting |
skip |
<cause> before the verdict: the cause is run timed out after <n> ms, error limit of <n> reached or token budget of <n> reached. A case that did not start has the cause alone. |
| The run signal aborts | error |
run aborted before the verdict (stoppedBy is 'signal'). |
The signal of runCase aborts |
error |
case aborted before the verdict (no stoppedBy). |
finish() ends the run while the case runs |
error |
run finished before the verdict. Nobody records that result. |
A trial calls ctx.skip(reason) (pytest.skip(reason) in pytest) |
skip |
The reason of the call. |
; last error: <name>: <message>follows an abort orfinish()reason only when the last recorded trial ended inerror. A trial that failed adds none. An error that a later trial superseded is not in the reason: anerrortrial, apasstrial, then an abort gives the cause alone.- A
skipafter some trials keeps their counts (passes,failuresanderrorscan be more than 0). The trial that calledctx.skipis not counted, has noTrialCompletedevent and starts no later trial. Its tokens count towardsummary.usageandtokenBudget. It never setsstoppedByand is never anerrortrial. - The shared scenarios
run-timeout.json,max-errors.jsonandskip-after-verdict.jsonpin these rules in both languages.
Set checks
Section titled “Set checks”rate({ name, where?, over? }).atLeast(threshold) returns a SetCheck; tag(name) returns a CasePredicate for tagged cases.
RateOptions field |
Default | Meaning |
|---|---|---|
name |
Required | Unique check name. |
where |
All non-skipped cases | Cases covered by the check. |
over |
'cases' |
'cases' counts passing decided cases; 'trials' counts passing counted trials. |
atLeast(threshold) |
Required | Number from 0 through 1. The check data includes counts and the 95% Wilson interval. |
SetCheck.evaluate receives only covered pass and fail cases. Covered error cases error the check; all xfail/xpass cases skip it; a partial set never passes over its subset.
A check runs after all cases of its eval. Core decides these outcomes before it calls evaluate, and the first row that applies wins:
| Situation | Outcome and reason |
|---|---|
No case is covered: none matches where, or every match has a skip mark (left out by design) |
error, no cases match: a misconfigured check, not a quality result |
A covered case has no result or ended in skip (a name filter such as -t <case id>, a shard, a runner skip, filter, ctx.skip, the token budget or, in Jest and node:test, a random order that runs the check first) |
skip, or fail with partialChecks: 'fail', for a CI job that requires every check; partial: <k> of <n> cases ran |
A covered case ended in error |
error, <k> of <n> cases errored: its quality is unknown |
Every covered case ended in xfail or xpass |
skip, every covered case is xfail or xpass |
Otherwise evaluate gets the covered pass and fail cases (xfail and xpass do not count in a rate: their status is known and reported per case). A where that throws, or an evaluate that throws an infrastructure value, ends the check in error (the check threw: <message>). Any other value that evaluate throws is a fail, with its message as the reason. A returned result that is not a valid score ends in error. A check does not depend on test order: it reads the case results that the adapter recorded in the same process, so a check that runs too early is skip (partial).
Errors
Section titled “Errors”| Export | Contract |
|---|---|
infrastructureError(message, { cause? }) |
Creates a branded InfrastructureError for an outage that your code identifies, such as a client that reports a plain error for an HTTP failure. |
isInfrastructureError(value) |
Safely detects the brand and known infrastructure shapes across realms and package copies. |
The classifier recognizes timeouts, aborts, connection errors, exhausted SDK retries, network codes, TypeError: fetch failed, and integer HTTP status/statusCode 400–599 on an error or its response. It follows cause, lastError, and errors breadth-first for at most three links and 64 values. The exact shared list is in the error-classification contract.
Results
Section titled “Results”| Export | Contract |
|---|---|
RunResult |
runId, evalName, datasetId, datasetHash, ordered cases and checks, summary, and non-fatal reporterErrors. |
RunSummary |
Outcome counts, ok, durationMs, aggregate usage, optional usageBy, and optional stoppedBy. |
CaseResult |
caseId, tags, outcome, optional reason and input hash, trials, and pass/failure/error counts. |
TrialResult |
Trial number, outcome, scores, duration, attachments, optional usage, spans, metrics, trace id, and ErrorInfo. |
CheckResult |
Name, outcome, optional value, reason, and JSON data. |
Attachment |
Name and JSON value. |
ErrorInfo |
Error name, message, optional stack, and an optional cause chain. |
Events and reporters
Section titled “Events and reporters”Every TrialkitEvent has schemaVersion, runId, and ISO-8601 time. Events for one run begin with RunStarted, preserve each reporter’s order, and end with RunCompleted; concurrent cases can interleave.
| Export | Event or contract |
|---|---|
RunStarted |
Eval, dataset id/hash/size, library version, resolved policy, labels, optional config and config hash. |
CaseStarted |
Case id and tags. |
TrialStarted |
Case id and trial number. |
TrialCompleted |
TrialResult without attachments, plus case id. |
AttachmentAdded |
Event type 'Attachment', with case id, trial, name, and value. |
CaseCompleted |
Case outcome, optional reason/input hash, and counts. |
CheckCompleted |
A CheckResult. |
RunCompleted |
The RunSummary. |
Reporter |
onEvent(event) and optional close(). close() runs once per run after its last event unless the event queue times out. A reporter error is recorded once and never changes an outcome. |
SCHEMA_VERSION |
1; bump only for a breaking event-shape change. Readers must ignore unknown fields. |
Event times come from a monotonic clock anchored at one wall-time origin (performance.timeOrigin plus performance.now() in TypeScript), so moving or freezing the calendar clock does not reorder events. The event schema does not require summary.usage, so a reader must accept a RunCompleted without it, although RunSummary always has usage in TypeScript.
Each reporter has at most trialTimeoutMs, and never less than its 120 s default, to finish its queued events and close() after the run finishes. runTimeoutMs bounds case execution, not reporter closing. A timeout appears in reporterErrors; it does not change case outcomes or summary.ok. Inspect both summary.ok and reporterErrors when deciding whether a run succeeded.
| Export or option | Default | Contract |
|---|---|---|
ConsoleReporterOptions / consoleReporter({ write?, verbose? }) |
write: console.log; verbose: false |
Prints a run summary and non-passing cases. verbose also prints passing cases and bounded trial logs. |
jsonl(sink) |
— | Writes one JSON event plus newline through a LineSink. It keeps no per-run state. |
LineSink |
close: absent |
write(line) and optional close(). A shared sink must tolerate one close per run. |
Model port
Section titled “Model port”| Export | Contract |
|---|---|
ModelClient |
id is <provider>:<model>; generate(request, signal?) resolves to ModelReply. |
ModelMessage |
role is system, user, or assistant; content is text. |
ModelRequest |
Required messages; optional fields below. |
ModelReply |
text, provider-reported model, optional parsed json, and optional TokenUsage. |
fromFunction(id, generate) |
Makes a ModelClient from an application or test function. |
ModelRequest field |
Default | Meaning |
|---|---|---|
jsonSchema |
None | { name, schema } for a provider that supports structured output. |
temperature |
None | Provider request setting. |
maxOutputTokens |
None | Provider request setting. |
Streaming
Section titled “Streaming”fromStream(stream, collect) makes a Task from (input, ctx) => AsyncIterable<chunk>. It collects chunks in order, records firstChunkMs after the first chunk, stops reading on ctx.signal, asks the source to return without waiting for the answer (a stuck source may never give one), and calls collect(chunks) once at the end. A synchronous iterable is invalid.
withUsage(task, usage) wraps a Task, calls usage(output), records its UsageInput, and returns the unchanged output.
Agent spans
Section titled “Agent spans”Span records a finished agent operation without an OpenTelemetry dependency or global tracer.
| Export | Contract |
|---|---|
Span |
Required non-empty name, unique 1–64-character spanId, JSON attributes, and status: 'ok' | 'error'; optional parentSpanId, start, and end. Parent spans must be recorded first; times are non-negative trial-relative milliseconds. |
SpanAttributes |
Readonly JSON key/value mapping, including GenAI semantic-convention keys. |
toolCalls(spans) / ToolCall |
Projects execute_tool spans in record order with name, arguments, result, span id, index, and optional error flag. An absent attribute and a JSON null both read null; hasArguments and hasResult say whether the span has the attribute. |
modelCalls(spans) / ModelCall |
Projects chat and generate_content spans in record order with name, model, span id, and index. |
ctx.span snapshots valid input. A trial accepts at most 1000 spans and 1,048,576 UTF-8 bytes; a parent must precede its child. Captured tool arguments, results, and messages can be sensitive.
spansOfAiSdk(result) from @trialkit/core/ai-sdk converts completed AI SDK v6/v7 generateText or consumed streamText steps to spans. Await it, then record each result with ctx.span. It has no AI SDK runtime dependency; content capture is opt-in. Its name distinguishes this async result converter from the synchronous OpenTelemetry spansOf.
OpenTelemetry import
Section titled “OpenTelemetry import”spansOf(readableSpans, options?) from @trialkit/core/otel converts explicitly collected, finished SDK spans. It reads no global provider and requires no OpenTelemetry package.
| Export or option | Default | Contract |
|---|---|---|
SpanContext |
— | traceId and spanId. |
ReadableSpan |
— | Finished span fields: name, context, optional parent context, start/end tuples, status, attributes, and ended. |
SpansOfOptions / origin |
Omitted | origin is finite epoch milliseconds. With it, output start/end are trial-relative; without it, both are omitted. |
spansOf |
— | Requires one trace, validates ids/times/status, orders parents before children, detaches uncollected parents, and ignores resources, events, and links. |
Statistics
Section titled “Statistics”| Export | Contract |
|---|---|
Interval |
{ low, high }. |
Rate |
{ passes, n }. |
RateDifference |
An Interval plus difference (a - b). |
wilson(passes, n) |
95% Wilson interval; n = 0 yields { low: 0, high: 1 }. |
rateDifference(a, b) |
95% Newcombe hybrid interval for independent rates; each n must be positive. |
passRates(results) |
{ cases, trials } for decided pass/fail cases only. rate() uses the same counts. |
wilson and rateDifference throw a RangeError (Python: ValueError) for counts that are not whole numbers with 0 <= passes <= n. passRates counts the passing trials out of the counted trials of the decided cases; a case that ended in error, skip, xfail or xpass is in neither count, and neither are its trials. rateDifference takes square roots with Math.sqrt and not Math.hypot, so that Python gets the same bits (spec/fixtures/rate-difference.json).
JSON values
Section titled “JSON values”Dataset fields, config, attachments, and score/check data must be JSON. Plain objects, arrays, strings, finite numbers, booleans, null, and toJSON values such as Date work. Classes, Map, bigint, non-finite numbers, cycles, and values over 500 levels do not. Dataset and config hashes also reject lone surrogates; attachments and score data serialize them. canonicalJson describes the shared canonical form.
Cases from rows
Section titled “Cases from rows”casesFromRows(rows, options) validates case rows that your app already has, for example from a database or an API, and returns EvalCase<unknown, unknown>[] in row order. It reads no file and does not change rows: a case is a new object, and the values inside it are shared with its row. It applies the case, id, metadata, and JSON rules of readCases; an optional field that is null counts as not given.
options field |
Default | Meaning |
|---|---|---|
source |
Required | Non-empty string that says where the rows came from: a file path, tickets-api, or a table name. |
lines |
None | One positive integer per row: the line of the row in the source. Without it, messages give a row number. |
import { casesFromRows } from '@trialkit/core';
const cases = casesFromRows(await db.query('select * from eval_cases'), { source: 'eval_cases table',});A problem throws a TypeError: <source>:<line>: <reason> with lines, or <source>: row <n>: <reason> without them (n counts from 1). rows that is not an array throws <source>: cases must be an array, and an empty array throws <source>: no cases. See the shared contract.
Node files
Section titled “Node files”The @trialkit/core/node subpath uses file APIs and works in Node.js, Bun, and Deno.
| Export | Contract |
|---|---|
RunFileOptions |
Exactly one non-empty { dir } or { path }. A directory produces one exclusive file per run; a path must not exist. |
jsonlFile(options) |
Reporter that writes one JSON Lines file per run. |
runFileOpener(options, reporter, extension) |
Validates file options once and returns an exclusive FileHandle opener for a custom file reporter. |
withDefaultReporters(options) |
Adds jsonlFile({ dir: 'eval-results' }) only when reporters is absent or undefined; reporters: [] stays empty. |
A directory writes <slug>.<run id>.<extension>. The slug is the eval name with each run of characters other than Unicode letters, digits, _, . and - replaced by -, cut to 100 code points: the run id already makes the name unique, and long names hit path limits.
Cases from a file
Section titled “Cases from a file”readCases(path) reads one UTF-8 .jsonl or .csv file into ordered EvalCase<unknown, unknown>[]; it has no options. JSON Lines accepts id, input, expected, tags, metadata, skip, and xfail. CSV requires id and input; expected and semicolon-separated tags are optional, and other columns become string metadata. Invalid, empty, or duplicate rows throw a TypeError with path, physical line, and reason.
Reading JSON Lines files
Section titled “Reading JSON Lines files”readRuns(paths, options?) returns an AsyncIterable<FileRuns>. It accepts non-empty file paths and recursively scans .jsonl files below named directories, reads each physical file once, and reports bad lines rather than stopping the history scan.
| Export | Contract |
|---|---|
FileRuns |
file, fresh runs, and problems for one file. |
ReadRun |
runId, non-attachment events beginning with RunStarted, selected attachments, and complete. |
ReadEvent |
Every TrialkitEvent except AttachmentAdded. |
ReadProblem |
reason and optional one-based line. |
ReadRunsOptions |
Options below. |
ReadRunsOptions field |
Default | Meaning |
|---|---|---|
check |
None | (event) => reason | undefined; rejected events become line problems. |
attachments |
None | Attachment names to retain; unrequested attachments are omitted. |
runIds |
All | Non-empty run ids to include; an empty list selects none. |
The reader accepts a BOM, \n, \r\n, or \r line endings, and blank space/tab lines. It keeps a run only when RunStarted is first, records no event after RunCompleted, and marks a run incomplete when its final event is missing.
Unselected run ids are omitted silently before event-field validation, including their attachments. Invalid JSON and lines without a usable run id still produce problems.
Utilities
Section titled “Utilities”The runtime-neutral @trialkit/core/util subpath exports these helpers; the root entry does not.
| Export | Contract |
|---|---|
canonicalJson(value) |
Canonical JSON with UTF-16-sorted keys and ECMAScript number formatting; rejects non-canonical JSON values. |
sha256Hex(text) |
SHA-256 of UTF-8 text as 64 lowercase hexadecimal characters. |
Html |
HTML already marked safe; .text and toString() return its markup. |
html |
Template tag that escapes &, <, >, ", and ' in interpolated text; joins arrays and leaves Html fragments intact. |
raw(text) |
Marks trusted text as Html without escaping it. Never pass untrusted data. |
trimText(text) |
Remove ECMAScript whitespace, including U+FEFF but not U+0085 or U+001C–U+001F. |
jsonLevels(value) |
The levels of arrays and objects in a JSON value: none for a scalar, one for [], two for [[]]; it does not recurse. |
MAX_CASE_FILE_DEPTH, caseRowTooDeep(row) |
Case-file depth limit (64) and whether a row exceeds it, including the document and its cases array. |
describeValue(value) |
Total, JSON-like error text clipped to 200 characters. A value of any depth gives the start of its JSON text. |
errorMessage(value) |
A thrown value’s string message, or describeValue(value); safe across realms. |
misuseError(message, { cause? }) |
TypeError for invalid dataset/options discovered by a scorer; core records it as an error. |
isPlainObject(value) |
True for object literals, null-prototype objects, and matching cross-realm objects; false for arrays, classes, and built-ins. |
validateSpans(value, label?) |
Validate a span list and return immutable snapshots, or throw a misuse error that starts with label (default ctx.span). |
LOG_ATTACHMENT, LogEntry, isLogEntry(value) |
The v1 log attachment name, payload type and payload validator. |
formatLogs(attachments) |
Format valid log attachments as bounded diagnostic lines. |
MAX_LOGS, MAX_LOG_BYTES |
Maximum per-trial log entries and their combined JSON payload bytes: 1,000 and 262,144 UTF-8 bytes. |
Runner adapters
Section titled “Runner adapters”For adapter authors. @trialkit/core/adapter is runner-neutral. Declare one runner test for every SuiteTest in order, call select() before a test starts when the runner knows selection, await each run(), map its TestReport, and call close() from an unbounded after-tests hook. Disable runner retries, runner timeouts, and test reordering; trialkit owns repeats, per-trial timeouts, and the final set-check test.
startRun(definition, options?) starts a run for an adapter. Its Run<I, E> exposes id, runCase and idempotent finish.
createSuite(definition, options?)
Section titled “createSuite(definition, options?)”| Export | Contract |
|---|---|
createSuite(definition, options = {}, caller = createSuite) |
Creates a definition-form Suite, validates at declaration, and starts the run at the first test. |
createTestSuite(name, options, declare, caller = createTestSuite) |
Creates a test-form Suite; declare receives EvalIt. |
Suite |
name, ordered tests, select(names), and close(). close() rejects for reporter errors, incomplete runs, unchecked failures, or partial checks caused by running skips. |
SuiteTest |
Case id or set checks, optional mode: 'only' | 'skip', and one-shot run(signal?, hooks?). |
TestReport |
{ status: 'pass'; warning? }, { status: 'skip'; message }, or { status: 'fail'; error }, optionally with case outcome, trial-count text and bounded logs for a failed case. |
createTrialHooks(options) |
Builds shared assertion-count and snapshot checks; accepts runner state, assertion reset, optional trial binding and runner-specific refusal. |
For createTrialHooks, getState() returns the runner’s expectedAssertionsNumber, isExpectingAssertions and snapshotState; resetAssertions() clears assertion-count expectations. Optional withinTrial binds the matcher to a trial, and optional refused() checks runner-specific features first. Both optional hooks default to absent.
| Case or check outcome | Runner action |
|---|---|
pass |
Pass. |
fail or error |
Fail; an error uses an infrastructure error. |
skip or xfail |
Skip with its reason. |
xpass |
Pass with a warning. |
| Set checks | Fail for a failed or errored check, skip when all skip, otherwise pass (with a warning for partial skips). |
SuiteOptions is StartRunOptions without signal, plus name and concurrency. createSuite itself has no default reporter; call withDefaultReporters when the adapter wants the standard file output.
SuiteOptions field |
Default | Meaning |
|---|---|---|
name |
Eval name | Runner suite name. |
reporters |
None | Event reporters; withDefaultReporters changes an omitted value to eval-results/ JSON Lines output. |
trialTimeoutMs |
120000 |
Per-trial milliseconds. |
runTimeoutMs |
Off | Whole-run case-execution deadline. |
maxErrors |
Off | Positive error-trial limit. |
concurrency |
1 |
Definition-form case concurrency; integer at least 1 or Infinity. |
labels |
{} |
String run labels. |
config |
None | Recorded JSON configuration. |
partialChecks |
'skip' |
Partial check outcome. |
tokenBudget |
No budget | Input-plus-output token cap. |
filter |
All cases | Case predicate. |
EvalOptions<I, O> powers createTestSuite: it is SuiteOptions without concurrency, plus required task, optional checks (default none), and the TrialPolicy fields with their defaults above. CaseMarks has optional tags, metadata, skip, and xfail, all defaulting to absent. EvalTestBody<I, O> receives EvalTestContext<I, O>, which is a TrialContext plus run(input) and score(scorer, output, expected?). score takes the output type of its scorer, so a body can score evidence that it builds from the result of run. Both promises must be awaited.
In TypeScript, run(input) starts the task at the call. In Python, run(input) returns a coroutine and starts the task when awaited. Both require awaiting the result.
| Export | Contract |
|---|---|
EvalIt<I, O> |
Declares it(name, body), marks, for(rows), only, and skip; names become case ids. |
EvalTestContext<I, O> |
run calls the task and records its input/output; score records a score and rejects on a failed score. |
EvalTestBody<I, O> |
One body invocation per trial. A returning body passes; throws use the shared outcome rule. |
MatcherContext |
Adapter matcher bridge: score follows the same pipeline; reject creates a misuse error. |
TrialHooks |
Optional withinTrial(matcher, body) and afterTrial(); no hooks means core invokes the body directly. |
RefusedFeature |
'expect.soft' | 'expect.assertions' | 'snapshot'; return one from afterTrial to end the case in error. |
createEvalMatchers() from @trialkit/core/node creates an opt-in toPass matcher and its withinTrial hook. Call it once per adapter copy so concurrent and timed-out trials keep separate matcher contexts.
The Python twin is trialkit.core.adapter; its runner-specific integration is documented in pytest-trialkit.