CI layout
Use separate jobs when cost or environment changes.
| Job | When | Model call | Gate |
|---|---|---|---|
| Unit tests | Every PR | No | Yes |
| Replayed component evals | Every PR or changed import graph | No | Yes |
| Live component evals | On demand | Yes | Start optional |
| Shared-environment black-box evals | Schedule and on demand | Yes | No |
import { evalSuite } from '@trialkit/vitest';import { evaluation } from './evaluation.ts';
evalSuite(evaluation, { tokenBudget: 400_000, trialTimeoutMs: 60_000, runTimeoutMs: 600_000, maxErrors: 5,});- A required job replays recordings with no secret. A person starts paid jobs; grant the key only to their eval step and pin every action by commit SHA.
tokenBudget,runTimeoutMsandmaxErrorsstop new work; incomplete cases skip, partial checks followpartialChecks, and the closing hook and compare gate fail the incomplete run.- Report task and judge usage with
ctx.addUsage. Label replaysmodel: 'replay:<model>'. Import code and datasets so thatvitest run --changedselects the eval;examples/support-triagetests its selection.
The core run options and CONTRIBUTING.md cover limits, usage, CI and secrets.