Example: quickstart for pytest
Copy this for a pytest eval of a support-message router: each case repeats, the verdict is a majority, and hand-written replies keep the run offline. No API key and no network call.
From the repository root, after uv sync:
uv run pytest examples/quickstart-pytest -vFrom this directory, pnpm test uses the root Python environment:
uv run --project ../.. pytest -c pyproject.toml -vpnpm typecheck checks the app, the fake and the tests with mypy --strict. The tests annotate the fixture run with RunFixture from trialkit.pytest and the output with the Reply type of the app.
test_router.py::test_password_reset PASSED (3/3 trials passed)test_router.py::test_duplicate_charge PASSED (3/4 trials passed)test_router.py::test_upload_crash PASSED (3/4 trials passed)test_router.py::test_email_change PASSED (3/4 trials passed)test_router.py::set checks PASSEDtrials=5andverdict="majority". Early stop ends a case at three passes or three failures. A wrong answer or invalid model JSON fails a trial. Infrastructure errors stay separate.- The set check requires every case tagged
regressionto pass.-k duplicateleaves out other cases, so a set check that needs them does not pass. - The replies are written by hand, not recorded. They show trial counts. For a comparison of trial rates, set
early_stop=Falseand use the same trial policy on both sides. The adapter writes JSON Lines toeval-results/in the working directory.
| File | Role |
|---|---|
app.py |
One model call through a small protocol, then a JSON parse. The eval runs this function. |
fake_model.py |
Canned replies in order, including wrong routes and JSON that a token limit cut off. |
test_router.py |
describe_eval("support-router", ...) and one test per message. |
test_app.py |
Ordinary pytest for the parser. |
conftest.py |
Puts this folder on the import path. |
Next: minimal Vitest, replay, history and PR gates, black-box HTTP.