pytest-trialkit
Write LLM evals as ordinary pytest tests. Each case runs repeated trials and passes by its verdict rule; infrastructure problems are error, not quality failures.
Install
Section titled “Install”pip install pytest-trialkitDepends on trialkit-core and pytest>=9.0.3,<10, nothing else: your installed pytest 9 is used. Python 3.11 or later.
Example
Section titled “Example”from trialkit.core import ratefrom trialkit.pytest import AsyncRunFixture, RunFixture, describe_eval, eval_case
CAPITALS = {"France": "Paris", "Japan": "Tokyo"}
def capital(country: str) -> str: # your app's own function return CAPITALS.get(country, "")
pytestmark = describe_eval( "capital-cities", task=capital, trials=3, checks=[rate("pass rate").at_least(0.9)],)
def test_france(run: RunFixture) -> None: assert run("France") == "Paris"
@eval_case(tags=["regression"])async def test_japan(run: AsyncRunFixture) -> None: assert await run("Japan") == "Tokyo"Run it with pytest -v:
test_capital_cities.py::test_france PASSED (3/3 trials passed)test_capital_cities.py::test_japan PASSED (3/3 trials passed)test_capital_cities.py::set checks PASSEDrunandscoreare fixtures. Adeftest calls them normally; anasync deftest awaits them. No asyncio plugin is required. Annotate them withRunFixtureandAsyncRunFixturefromtrialkit.pytest.- Without
reporters, each run writes a JSON Lines file toeval-results/. Usereporters=[]to record nothing. - Requires Python 3.11 or later and pytest 9.0.3 or later in the 9.x series. The package is not yet published. pytest loads its
pytest_trialkitplugin automatically; ifPYTEST_DISABLE_PLUGIN_AUTOLOADis set, use-p pytest_trialkit.
The test form: describe_eval
Section titled “The test form: describe_eval”describe_eval(name, *, task, ...) marks a module via pytestmark or a class via a decorator. Each marked module or class is one eval and one run. Options are checked when the mark is made. A nested class with its own mark is a separate eval; the closest class or module mark wins. A function mark, two marks on one scope, or a marked scope with no tests fails collection.
| Option | Default | Meaning |
|---|---|---|
name |
Required | Eval name and dataset id. |
task |
Required | Plain or async function accepting (input) or (input, ctx). |
trials |
1 |
Maximum counted pass/fail trials per case. |
verdict |
"all" |
"all", "majority" or AtLeast(n). |
early_stop |
True |
Stop once remaining trials cannot change the verdict. |
error_retries |
0 |
Extra trials only for errors; error trials do not count toward trials. |
checks |
() |
Set checks. Their names must be unique; they add a set checks test. |
reporters |
None |
JSON Lines file per run in eval-results/. A supplied sequence replaces this default; [] disables recording. |
trial_timeout_ms |
120_000 |
Trial deadline, in milliseconds, 1–2,147,483,647; timeout is error. |
run_timeout_ms |
None |
Optional whole-run case-execution deadline, 1–2,147,483,647 ms. |
max_errors |
None |
Optional positive limit on error trials, including retries. |
labels |
None |
String-to-string run labels. |
config |
None |
JSON settings recorded with a hash; do not include secrets. |
filter |
None |
Case predicate; non-matching cases skip. In the test form a case has its id and marks, not an input. |
partial_checks |
"skip" |
"skip" or "fail" for a check with cases that did not run. |
token_budget |
None |
Optional positive token limit; report tokens with ctx.add_usage. |
All options except name and task are keyword-only and accept None as their default. There is no per-case policy or concurrency in this form. For different policies, use distinct marked classes or modules. Shared policy and run semantics match trialkit-core.
Each collected test function is a case. Its id is its node id relative to the marked module or class. The dataset hash covers ids and marks, not test bodies or parameter values. A parameter that is an EvalCase supplies its id, tags, metadata, skip and xfail; explicit pytest ids= or pytest.param(..., id=...) wins. Use the parameter’s input and expected in the body. pytest escapes ids by its own rules.
eval_case(*, tags=None, metadata=None, skip=None, xfail=None) marks a test, class or module. tags is a list or tuple of strings, and metadata maps strings to strings, finite numbers or booleans. skip and xfail are non-empty reason strings. Tags and reasons are checked when the mark is made; metadata is checked when pytest collects each test, so its error names that test. The closest mark replaces, rather than merges with, a more distant mark. A test cannot combine an EvalCase parameter with eval_case, or take two EvalCase parameters. Duplicate eval name and value-equal labels in a test file are rejected across both forms; a different display name or config does not distinguish them.
Use eval_case(xfail="reason"), not pytest.mark.xfail, which is rejected. An expected failure skips; an unexpected pass passes with pytest.PytestWarning (-W error makes it fail). pytest skip and skipif marks work normally. Function-scoped fixtures and mocks are set up once per case, not per trial; create per-trial state in the body.
The fixtures run and score
Section titled “The fixtures run and score”| Fixture | Meaning |
|---|---|
run(input) |
Calls the task and returns its output. Input must be JSON. Each call records a run attachment; the first input supplies the case’s inputHash and must match across its trials. Later calls can be other steps. |
score(scorer, output, expected=None) |
Records a named, versioned score and returns it. A failing score raises AssertionError; a caught assertion can leave the body passing with a failing score recorded. |
Annotate the fixtures with the types of trialkit.pytest: RunFixture and ScoreFixture in a def test, and AsyncRunFixture and AsyncScoreFixture in an async def test. The async forms give coroutines, so mypy --strict reports a call that the test did not await. run gives Any, because pytest passes a fixture by its name and no type checker knows the task of the eval: write the type where you use the output (output: str = run("France")). score gives a trialkit.core.Score.
A body that returns passes; a non-infrastructure exception, including an assertion, fails. An uncaught infrastructure exception is error. A body can catch a task exception to test rejection, but a scorer infrastructure error or trialkit misuse remains error even if caught. Await every run and score in an async test: an unawaited or unfinished call is error. A sync test that returns an awaitable is also error. Each passing or failing trial records the implicit assert@1 score; assert is reserved. The fixtures are available only in a marked eval, and calls after a timeout cannot start work.
A sync body runs in a worker thread; an async body runs on the eval’s loop. A thread you start must propagate context, for example with contextvars.copy_context().run(...) or asyncio.to_thread. The same fixture values are used for each trial.
The definition form: eval_suite
Section titled “The definition form: eval_suite”Use eval_suite when a script shares an EvalDefinition. Assign its result to a top-level test name:
from trialkit.core import ( Dataset, EvalCase, EvalDefinition, ScoreValue, TrialContext, rate, scorer,)from trialkit.pytest import eval_suite
@scorer(name="exact", version="1")def exact(case: EvalCase[str, str], output: str, ctx: TrialContext) -> ScoreValue: passed = output == case.expected return ScoreValue(1.0 if passed else 0.0, passed, f"expected {case.expected}, got {output}")
definition = EvalDefinition( name="capital-cities", dataset=Dataset( id="capitals-v1", cases=[ EvalCase(id="france", input="France", expected="Paris"), EvalCase(id="japan", input="Japan", expected="Tokyo"), ], ), task=lambda country: {"France": "Paris", "Japan": "Tokyo"}.get(country, ""), scorers=[exact], checks=[rate("pass rate").at_least(0.9)],)
test_capital_cities = eval_suite(definition)eval_suite(definition, *, name=None, reporters=None, trial_timeout_ms=120_000, run_timeout_ms=None, max_errors=None, concurrency=1, labels=None, config=None, filter=None, partial_checks="skip", token_budget=None) returns a pytest test function. All keywords are optional and accept None as their default. Their meanings match the table above, except:
| Option | Default | Meaning |
|---|---|---|
name |
None |
Suite display name in adapter messages; defaults to the eval name. |
concurrency |
1 |
Concurrent cases, an integer ≥ 1 or math.inf; only the definition form supports it. |
pytest collects a test per case id and, when checks exist, a final set checks test. Do not call the returned function or add fixtures or parametrization to it; pytest marks work. The adapter validates the definition, options and case ids at declaration. Each pytest session creates a fresh run when its first selected test runs. Selected cases can start ahead up to concurrency; they cannot use setup from a future test. Results remain in dataset order, and concurrent token use can overshoot the budget.
Record and replay
Section titled “Record and replay”Python has no @trialkit/replay: VCR.py and pytest-recording (pip install vcrpy pytest-recording) record the HTTP replies of your model client once and replay them in CI with no key. A request that is not in the cassette raises vcr.errors.CannotOverwriteExistingCassetteException, which is a fail. Raise InfrastructureError for it, so the trial ends in error: record again.
import os
import openaiimport pytestfrom trialkit.core import InfrastructureErrorfrom trialkit.pytest import describe_evalfrom vcr.errors import CannotOverwriteExistingCassetteException
@pytest.fixture(scope="module")def vcr_config(): return { "filter_headers": ["authorization"], "match_on": ["method", "scheme", "host", "port", "path", "query", "body"], }
def triage(ticket: str) -> str: client = openai.OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "replay")) try: reply = client.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "user", "content": ticket}] ) except CannotOverwriteExistingCassetteException as exc: raise InfrastructureError("no recording for this request: record again") from exc return reply.choices[0].message.content
pytestmark = [describe_eval("triage", task=triage, trials=2), pytest.mark.vcr]- A person records with
pytest --record-mode=once; commitcassettes/next to the test file. CI uses the default mode,none, and needs no key. To record a changed case again, delete its cassette file first. - VCR.py ignores the body by default, so a changed prompt would replay the old reply;
match_onadds it. Withoutfilter_headers, the cassette heldAuthorization: Bearer <key>. - A cassette plays each reply once per recording, so a replay with more
trialsthan were recorded also raises the exception.
Verified with vcrpy 9.4.2, pytest-recording 0.14.1, openai 3.46.0 and pytest 9.2.0 against a local OpenAI-compatible server. The VCR.py facts above come from that run. Other versions or HTTP clients can wrap the exception; catch what your client raises.
Results and selection
Section titled “Results and selection”| Outcome | pytest result |
|---|---|
pass |
Pass. |
fail |
Fail with AssertionError, trial counts and the failing reason. |
error |
Fail with InfrastructureError, or the original misuse error class. |
skip |
Skip with the case reason. |
xfail |
Skip with xfail: <reason>. |
xpass |
Pass with a warning. |
Verbose output and the JUnit trials property show counted passes/trials and any errors. The last failing test-form assertion retains its traceback. A set-check test fails if a check fails or errors, skips if all checks skip, and otherwise passes; a mix of passing and skipped checks warns.
Failed or errored cases include bounded trial logs in a trialkit logs report section. Passing-case logs stay in reporters and the history report.
Use -k, node ids, --deselect or --lf to select tests. Deselected cases can make checks partial; this intentional selection does not by itself fail the session. partial_checks="fail" makes a partial check fail. A case that skips while running or is dropped by filter is not a pytest selection: a partial check then fails the session at closing. A dataset or eval_case skip mark intentionally excludes the case from checks. To select only the set checks, use -k checks or its node id; -k "set checks" is a pytest usage error.
The plugin moves set checks after the last case of its eval, including with random-order plugins and --ff or --lf. A case still run after it fails. A runner retry reports the first case result without rerunning trials. unittest.TestCase tests in a marked module remain ordinary tests.
The closing step
Section titled “The closing step”A session hook finishes every started run, even after -x, Ctrl+C or deselection of the set checks test. It runs remaining checks, emits RunCompleted, closes reporters and reports unreported check failures. It fails an otherwise successful session for a reporter or close error, a run stopped by a guard, a run where cases ran but none reached a verdict, unexpected partial checks, or a token budget with no reported usage. It does not fail an intentional selection of only set checks or a suite with no selected tests. Existing nonzero exit statuses remain unchanged.
A reporter you provide replaces the default JSON Lines file. To keep both console and file output, pass reporters=[console_reporter(), jsonl_file(dir="eval-results")]. pytest captures console output from set checks; use -s or -rP to see it.
The event loop
Section titled “The event loop”Each eval has its own asyncio loop, closed at session end. Async tests need no asyncio plugin. With pytest-asyncio, an async fixture may belong to a different loop from the eval body; create loop-bound objects in the test or task. Ctrl+C cancels the running case, and closing records the run as stopped by signal.
pytest-xdist
Section titled “pytest-xdist”Each worker has its own run and file. Use --dist loadfile so a module’s eval and its set checks stay together and definition-form look-ahead can use concurrency. Other xdist distributions run each worker’s cases sequentially: a worker cannot know which cases xdist assigned elsewhere. --dist loadscope also keeps a class eval together. A split eval with partial set checks fails the session. The controller reports worker closing errors.
pytest-timeout
Section titled “pytest-timeout”The plugin adds timeout(0) to eval tests so pytest-timeout cannot end a case mid-trial. Core still applies trial_timeout_ms. An explicit timeout mark remains in force and can end the process before the run closes, especially with the plugin’s thread method on Windows.