Skip to content

1. Problem

Teams write their own judge and repeat loop, adopt an eval runner, or run evals inside the app’s tests. Tools that bundle a runner, a dataset store and a UI bring these problems:

  • A second runner next to the test framework: its loader often cannot import the app’s code (decorators, path aliases), and the test runner’s change selection cannot find its evals.
  • Dependency trees to audit, with a second copy of an SDK that the app already has.
  • History on a platform: a run without an upload keeps no local history to trend or compare.
  • Python-first evaluation: a TypeScript app is reached only through its public interface.

Other tools already offer repeated trials, local results and in-runner integrations; the comparison records which, with a source and a date for each fact. trialkit adds:

  • A verdict rule for each case, with early stop.
  • The error outcome: a timeout or a 5xx is never a wrong answer, and never a pass.
  • Set checks safe on a partial run: a rate over a filtered, sharded or stopped run is skip, never a pass or a fail over a subset.
  • An offline record: one JSON Lines format for TypeScript and Python, which a CLI turns into trends with intervals and into pull request gates.