Skip to content

Example: black-box

Copy this to score an HTTP service without importing it. The service routes a support note. The same note always gets the same JSON from a fake model, and the call stays on this machine. For a real model, copy support-triage. The guide’s section is Level 3.

Run these in this directory. From the repository root, pnpm --filter example-blackbox test is the same pnpm test. In PowerShell: $env:BLACKBOX_URL = 'http://127.0.0.1:8787'.

Command What it does
pnpm test Project local: starts the service on a free port, runs the eval, and runs the regression tests.
pnpm serve Listens on 127.0.0.1:8787. PORT overrides the port.
pnpm blackbox Project remote: the eval against BLACKBOX_URL. A missing URL fails the run.

No command spends money. pnpm test clears BLACKBOX_URL and refuses fetch to any host but the loopback interface. The tests listen on port 0.

Case results from pnpm test:

double-charge: 1/1 trials passed
upload-down: 1/1 trials passed
locked-out: 1/1 trials passed
thanks: 1/1 trials passed
long-export: 1/1 trials passed
set checks: pass
  • Setup calls GET /health once, with a limit of 10 seconds. When that fails, every case skips and partialChecks: 'fail' fails the run. The test setup: <target> is ready throws the same reason, because Vitest does not print a skip reason.
  • A status other than 2xx, a body that is not JSON, or a body that is not a route ends the trial in error. fetch does not throw for an HTTP status, so the task throws infrastructureError. A wrong queue, urgency, or title fails expect. The trial limit is 20 seconds, so a service that does not answer skips instead of timing the trial out.
  • Each trial stores the request and the response. traceId is the x-request-id header. The set check regressions requires every case tagged regression to pass. The service is deterministic (trials: 1). The suite writes eval-results/note-router.<run id>.jsonl (Git ignores that directory). pnpm report reads it after a build of the repository.
File Role
src/route.ts Routing rules. The eval does not import this file.
src/model.ts Deterministic fake model.
src/server.ts, src/main.ts GET /health and POST /route ({"note"} returns {"queue","urgency","title"}). pnpm serve. No route exists only for the eval.
evals/notes.json Dataset.
evals/blackbox.ts HTTP task. Imports the dataset, not the service.
evals/suite.ts describeEval. expect checks the route.
evals/router.local.test.ts Starts the service, runs the eval, and closes the service.
evals/router.remote.test.ts The same eval against BLACKBOX_URL.
test/ The rules, the HTTP interface, and the pass, fail, error, and setup outcomes.