Skip to content

3. Goals and non-goals

Goals

  • A core with no runtime dependencies.
  • Any model provider (one small port), test framework (thin adapters; the core also runs without one), results backend (reporters on one event stream) and app framework (the task is a plain async function).
  • Evals as ordinary tests: one runner test is one case, and a component eval imports the production code and runs in the app’s own runner.
  • Explicit nondeterminism and comparable results: trials, intervals, early stop, and outcomes that keep errors apart from quality failures. Each result records the dataset hash, the input hash of each case, the judge model and prompt version, and the library version.

Non-goals

  • A hosted store, accounts or a service: reporters write to tools that already have one.
  • A web UI beyond the opt-in local studio, an annotation tool, and storage for prompts and datasets.
  • Red-team input generators (a possible plugin later).
  • Choosing which evals to run for a change: the runner does that, and evals import the production code so that its change selection finds them.