Skip to content

Level 3: a black-box eval

Use a black-box eval when a change crosses steps, or when you need what users see: measure the product through the same interface as a user.

import { infrastructureError, type TrialContext } from '@trialkit/core';
async function callTriage(url: string, ticket: string, ctx: TrialContext) {
ctx.attach('request', { method: 'POST', url, body: { ticket } });
const response = await fetch(url, {
method: 'POST',
body: JSON.stringify({ ticket }),
signal: ctx.signal,
});
const text = await response.text();
ctx.attach('response', { status: response.status, body: text });
if (!response.ok) throw infrastructureError(`POST ${url}: HTTP ${response.status}`);
try {
return JSON.parse(text);
} catch (cause) {
throw infrastructureError(`POST ${url}: response is not JSON`, { cause });
}
}
  • Select the environment by configuration. Check it once with a time limit; if setup skips cases, use partialChecks: 'fail' and expose the setup reason as a test failure.
  • Keep shared state unchanged or restore it in cancellation-safe teardown. Score response facts deterministically. Attach requests and responses, and put a request or trace id in ctx.traceId.
  • Treat a bad status, unreadable body or invalid response shape as infrastructure; a valid response with a wrong field is a failed score. Run a shared-environment eval on demand or on a schedule, not as a PR gate.

examples/blackbox and the support-triage black-box eval show health checks, shape validation and replay.