Errors and failures
Use this rule whenever a task, scorer, set check or test body throws. A fail lowers the pass rate. An error leaves it out, may use errorRetries, and makes a former verdict an error regression in trialkit compare.
The rule
Section titled “The rule”Infrastructure is error; every other throw is fail with its message as the reason.
| Throw | Outcome |
|---|---|
Reply parser SyntaxError, assertion or app TypeError |
fail |
| SDK HTTP 400–599, timeout or network error | error |
TypeError: fetch failed or missing replay (404) |
error |
infrastructureError('POST /triage: HTTP 502') |
error |
- Core also looks inside wrapped errors. The spec has the complete rule, in both languages.
- A plain
Error('HTTP 502')is afail. Mark your own infrastructure failure withinfrastructureError(message, { cause }), as the black-box eval does. - In a component eval, a parser rejects the model’s bad answer, so it fails. In a black-box eval, the service’s malformed response is infrastructure, so mark it as an error. Trialkit misuse is always an
error.