Skip to content

Errors and failures

Use this rule whenever a task, scorer, set check or test body throws. A fail lowers the pass rate. An error leaves it out, may use errorRetries, and makes a former verdict an error regression in trialkit compare.

Infrastructure is error; every other throw is fail with its message as the reason.

Throw Outcome
Reply parser SyntaxError, assertion or app TypeError fail
SDK HTTP 400–599, timeout or network error error
TypeError: fetch failed or missing replay (404) error
infrastructureError('POST /triage: HTTP 502') error
  • Core also looks inside wrapped errors. The spec has the complete rule, in both languages.
  • A plain Error('HTTP 502') is a fail. Mark your own infrastructure failure with infrastructureError(message, { cause }), as the black-box eval does.
  • In a component eval, a parser rejects the model’s bad answer, so it fails. In a black-box eval, the service’s malformed response is infrastructure, so mark it as an error. Trialkit misuse is always an error.