Skip to content

Example: quickstart for pytest

Copy this for a pytest eval of a support-message router: each case repeats, the verdict is a majority, and hand-written replies keep the run offline. No API key and no network call.

From the repository root, after uv sync:

Terminal window
uv run pytest examples/quickstart-pytest -v

From this directory, pnpm test uses the root Python environment:

Terminal window
uv run --project ../.. pytest -c pyproject.toml -v

pnpm typecheck checks the app, the fake and the tests with mypy --strict. The tests annotate the fixture run with RunFixture from trialkit.pytest and the output with the Reply type of the app.

test_router.py::test_password_reset PASSED (3/3 trials passed)
test_router.py::test_duplicate_charge PASSED (3/4 trials passed)
test_router.py::test_upload_crash PASSED (3/4 trials passed)
test_router.py::test_email_change PASSED (3/4 trials passed)
test_router.py::set checks PASSED
  • trials=5 and verdict="majority". Early stop ends a case at three passes or three failures. A wrong answer or invalid model JSON fails a trial. Infrastructure errors stay separate.
  • The set check requires every case tagged regression to pass. -k duplicate leaves out other cases, so a set check that needs them does not pass.
  • The replies are written by hand, not recorded. They show trial counts. For a comparison of trial rates, set early_stop=False and use the same trial policy on both sides. The adapter writes JSON Lines to eval-results/ in the working directory.
File Role
app.py One model call through a small protocol, then a JSON parse. The eval runs this function.
fake_model.py Canned replies in order, including wrong routes and JSON that a token limit cut off.
test_router.py describe_eval("support-router", ...) and one test per message.
test_app.py Ordinary pytest for the parser.
conftest.py Puts this folder on the import path.

Next: minimal Vitest, replay, history and PR gates, black-box HTTP.