Level 3: a black-box eval
Use a black-box eval when a change crosses steps, or when you need what users see: measure the product through the same interface as a user.
import { infrastructureError, type TrialContext } from '@trialkit/core';
async function callTriage(url: string, ticket: string, ctx: TrialContext) { ctx.attach('request', { method: 'POST', url, body: { ticket } }); const response = await fetch(url, { method: 'POST', body: JSON.stringify({ ticket }), signal: ctx.signal, }); const text = await response.text(); ctx.attach('response', { status: response.status, body: text }); if (!response.ok) throw infrastructureError(`POST ${url}: HTTP ${response.status}`); try { return JSON.parse(text); } catch (cause) { throw infrastructureError(`POST ${url}: response is not JSON`, { cause }); }}- Select the environment by configuration. Check it once with a time limit; if setup skips cases, use
partialChecks: 'fail'and expose the setup reason as a test failure. - Keep shared state unchanged or restore it in cancellation-safe teardown. Score response facts deterministically. Attach requests and responses, and put a request or trace id in
ctx.traceId. - Treat a bad status, unreadable body or invalid response shape as infrastructure; a valid response with a wrong field is a failed score. Run a shared-environment eval on demand or on a schedule, not as a PR gate.
examples/blackbox and the support-triage black-box eval show health checks, shape validation and replay.