Skip to content

9. Model port

ModelClient is the small port between judges and a model provider: an id (<provider>:<model>, recorded in every judge score), generate(request, signal?), and fromFunction for any other client or a test fake (reference). The port carries text only: a message content is a string, so a judge cannot inspect pixels or audio, and media support would need a media-aware port first. Only judges and model-based scorers use the port. The task calls the app’s own client, so the eval tests the app’s real call path.

  • openaiModel and aiSdkModel are subpaths of @trialkit/judge, not packages. Each takes the app’s SDK as an optional peer and adds no retries and no streaming, because the SDK owns transport retries. A judge wraps an SDK error, so the trial ends in error.
  • Databricks Unity Gateway needs no package. It is OpenAI-compatible, so openaiModel reaches it with an openai client whose baseURL is the gateway (setup). The app gives the token to the client, and the adapter reads no DATABRICKS_TOKEN. CI runs this path against a fake fetch; a person checks a live workspace before a release.
  • Python’s OpenAI-compatible client lives in trialkit-judge and uses the standard library, so it covers the gateway without a separate distribution.
  • Not built: the OpenAI Responses API in @trialkit/judge/openai, and @trialkit/judge/anthropic.