Example: black-box
Copy this to score an HTTP service without importing it. The service routes a support note. The same note always gets the same JSON from a fake model, and the call stays on this machine. For a real model, copy support-triage. The guide’s section is Level 3.
Run these in this directory. From the repository root, pnpm --filter example-blackbox test is the same pnpm test. In PowerShell: $env:BLACKBOX_URL = 'http://127.0.0.1:8787'.
| Command | What it does |
|---|---|
pnpm test |
Project local: starts the service on a free port, runs the eval, and runs the regression tests. |
pnpm serve |
Listens on 127.0.0.1:8787. PORT overrides the port. |
pnpm blackbox |
Project remote: the eval against BLACKBOX_URL. A missing URL fails the run. |
No command spends money. pnpm test clears BLACKBOX_URL and refuses fetch to any host but the loopback interface. The tests listen on port 0.
Case results from pnpm test:
double-charge: 1/1 trials passedupload-down: 1/1 trials passedlocked-out: 1/1 trials passedthanks: 1/1 trials passedlong-export: 1/1 trials passedset checks: pass- Setup calls
GET /healthonce, with a limit of 10 seconds. When that fails, every case skips andpartialChecks: 'fail'fails the run. The testsetup: <target> is readythrows the same reason, because Vitest does not print a skip reason. - A status other than 2xx, a body that is not JSON, or a body that is not a route ends the trial in
error.fetchdoes not throw for an HTTP status, so the task throwsinfrastructureError. A wrong queue, urgency, or title failsexpect. The trial limit is 20 seconds, so a service that does not answer skips instead of timing the trial out. - Each trial stores the request and the response.
traceIdis thex-request-idheader. The set checkregressionsrequires every case taggedregressionto pass. The service is deterministic (trials: 1). The suite writeseval-results/note-router.<run id>.jsonl(Git ignores that directory).pnpm reportreads it after a build of the repository.
| File | Role |
|---|---|
src/route.ts |
Routing rules. The eval does not import this file. |
src/model.ts |
Deterministic fake model. |
src/server.ts, src/main.ts |
GET /health and POST /route ({"note"} returns {"queue","urgency","title"}). pnpm serve. No route exists only for the eval. |
evals/notes.json |
Dataset. |
evals/blackbox.ts |
HTTP task. Imports the dataset, not the service. |
evals/suite.ts |
describeEval. expect checks the route. |
evals/router.local.test.ts |
Starts the service, runs the eval, and closes the service. |
evals/router.remote.test.ts |
The same eval against BLACKBOX_URL. |
test/ |
The rules, the HTTP interface, and the pass, fail, error, and setup outcomes. |