Skip to content

CI layout

Use separate jobs when cost or environment changes.

Job When Model call Gate
Unit tests Every PR No Yes
Replayed component evals Every PR or changed import graph No Yes
Live component evals On demand Yes Start optional
Shared-environment black-box evals Schedule and on demand Yes No
import { evalSuite } from '@trialkit/vitest';
import { evaluation } from './evaluation.ts';
evalSuite(evaluation, {
tokenBudget: 400_000,
trialTimeoutMs: 60_000,
runTimeoutMs: 600_000,
maxErrors: 5,
});
  • A required job replays recordings with no secret. A person starts paid jobs; grant the key only to their eval step and pin every action by commit SHA.
  • tokenBudget, runTimeoutMs and maxErrors stop new work; incomplete cases skip, partial checks follow partialChecks, and the closing hook and compare gate fail the incomplete run.
  • Report task and judge usage with ctx.addUsage. Label replays model: 'replay:<model>'. Import code and datasets so that vitest run --changed selects the eval; examples/support-triage tests its selection.

The core run options and CONTRIBUTING.md cover limits, usage, CI and secrets.