1. Problem
Teams write their own judge and repeat loop, adopt an eval runner, or run evals inside the app’s tests. Tools that bundle a runner, a dataset store and a UI bring these problems:
- A second runner next to the test framework: its loader often cannot import the app’s code (decorators, path aliases), and the test runner’s change selection cannot find its evals.
- Dependency trees to audit, with a second copy of an SDK that the app already has.
- History on a platform: a run without an upload keeps no local history to trend or compare.
- Python-first evaluation: a TypeScript app is reached only through its public interface.
Other tools already offer repeated trials, local results and in-runner integrations; the comparison records which, with a source and a date for each fact. trialkit adds:
- A verdict rule for each case, with early stop.
- The
erroroutcome: a timeout or a 5xx is never a wrong answer, and never a pass. - Set checks safe on a partial run: a rate over a filtered, sharded or stopped run is
skip, never a pass or a fail over a subset. - An offline record: one JSON Lines format for TypeScript and Python, which a CLI turns into trends with intervals and into pull request gates.