Skip to content

pytest-trialkit

Write LLM evals as ordinary pytest tests. Each case runs repeated trials and passes by its verdict rule; infrastructure problems are error, not quality failures.

Terminal window
pip install pytest-trialkit

Depends on trialkit-core and pytest>=9.0.3,<10, nothing else: your installed pytest 9 is used. Python 3.11 or later.

test_capital_cities.py
from trialkit.core import rate
from trialkit.pytest import AsyncRunFixture, RunFixture, describe_eval, eval_case
CAPITALS = {"France": "Paris", "Japan": "Tokyo"}
def capital(country: str) -> str: # your app's own function
return CAPITALS.get(country, "")
pytestmark = describe_eval(
"capital-cities",
task=capital,
trials=3,
checks=[rate("pass rate").at_least(0.9)],
)
def test_france(run: RunFixture) -> None:
assert run("France") == "Paris"
@eval_case(tags=["regression"])
async def test_japan(run: AsyncRunFixture) -> None:
assert await run("Japan") == "Tokyo"

Run it with pytest -v:

test_capital_cities.py::test_france PASSED (3/3 trials passed)
test_capital_cities.py::test_japan PASSED (3/3 trials passed)
test_capital_cities.py::set checks PASSED
  • run and score are fixtures. A def test calls them normally; an async def test awaits them. No asyncio plugin is required. Annotate them with RunFixture and AsyncRunFixture from trialkit.pytest.
  • Without reporters, each run writes a JSON Lines file to eval-results/. Use reporters=[] to record nothing.
  • Requires Python 3.11 or later and pytest 9.0.3 or later in the 9.x series. The package is not yet published. pytest loads its pytest_trialkit plugin automatically; if PYTEST_DISABLE_PLUGIN_AUTOLOAD is set, use -p pytest_trialkit.

describe_eval(name, *, task, ...) marks a module via pytestmark or a class via a decorator. Each marked module or class is one eval and one run. Options are checked when the mark is made. A nested class with its own mark is a separate eval; the closest class or module mark wins. A function mark, two marks on one scope, or a marked scope with no tests fails collection.

Option Default Meaning
name Required Eval name and dataset id.
task Required Plain or async function accepting (input) or (input, ctx).
trials 1 Maximum counted pass/fail trials per case.
verdict "all" "all", "majority" or AtLeast(n).
early_stop True Stop once remaining trials cannot change the verdict.
error_retries 0 Extra trials only for errors; error trials do not count toward trials.
checks () Set checks. Their names must be unique; they add a set checks test.
reporters None JSON Lines file per run in eval-results/. A supplied sequence replaces this default; [] disables recording.
trial_timeout_ms 120_000 Trial deadline, in milliseconds, 1–2,147,483,647; timeout is error.
run_timeout_ms None Optional whole-run case-execution deadline, 1–2,147,483,647 ms.
max_errors None Optional positive limit on error trials, including retries.
labels None String-to-string run labels.
config None JSON settings recorded with a hash; do not include secrets.
filter None Case predicate; non-matching cases skip. In the test form a case has its id and marks, not an input.
partial_checks "skip" "skip" or "fail" for a check with cases that did not run.
token_budget None Optional positive token limit; report tokens with ctx.add_usage.

All options except name and task are keyword-only and accept None as their default. There is no per-case policy or concurrency in this form. For different policies, use distinct marked classes or modules. Shared policy and run semantics match trialkit-core.

Each collected test function is a case. Its id is its node id relative to the marked module or class. The dataset hash covers ids and marks, not test bodies or parameter values. A parameter that is an EvalCase supplies its id, tags, metadata, skip and xfail; explicit pytest ids= or pytest.param(..., id=...) wins. Use the parameter’s input and expected in the body. pytest escapes ids by its own rules.

eval_case(*, tags=None, metadata=None, skip=None, xfail=None) marks a test, class or module. tags is a list or tuple of strings, and metadata maps strings to strings, finite numbers or booleans. skip and xfail are non-empty reason strings. Tags and reasons are checked when the mark is made; metadata is checked when pytest collects each test, so its error names that test. The closest mark replaces, rather than merges with, a more distant mark. A test cannot combine an EvalCase parameter with eval_case, or take two EvalCase parameters. Duplicate eval name and value-equal labels in a test file are rejected across both forms; a different display name or config does not distinguish them.

Use eval_case(xfail="reason"), not pytest.mark.xfail, which is rejected. An expected failure skips; an unexpected pass passes with pytest.PytestWarning (-W error makes it fail). pytest skip and skipif marks work normally. Function-scoped fixtures and mocks are set up once per case, not per trial; create per-trial state in the body.

Fixture Meaning
run(input) Calls the task and returns its output. Input must be JSON. Each call records a run attachment; the first input supplies the case’s inputHash and must match across its trials. Later calls can be other steps.
score(scorer, output, expected=None) Records a named, versioned score and returns it. A failing score raises AssertionError; a caught assertion can leave the body passing with a failing score recorded.

Annotate the fixtures with the types of trialkit.pytest: RunFixture and ScoreFixture in a def test, and AsyncRunFixture and AsyncScoreFixture in an async def test. The async forms give coroutines, so mypy --strict reports a call that the test did not await. run gives Any, because pytest passes a fixture by its name and no type checker knows the task of the eval: write the type where you use the output (output: str = run("France")). score gives a trialkit.core.Score.

A body that returns passes; a non-infrastructure exception, including an assertion, fails. An uncaught infrastructure exception is error. A body can catch a task exception to test rejection, but a scorer infrastructure error or trialkit misuse remains error even if caught. Await every run and score in an async test: an unawaited or unfinished call is error. A sync test that returns an awaitable is also error. Each passing or failing trial records the implicit assert@1 score; assert is reserved. The fixtures are available only in a marked eval, and calls after a timeout cannot start work.

A sync body runs in a worker thread; an async body runs on the eval’s loop. A thread you start must propagate context, for example with contextvars.copy_context().run(...) or asyncio.to_thread. The same fixture values are used for each trial.

Use eval_suite when a script shares an EvalDefinition. Assign its result to a top-level test name:

test_capital_cities.py
from trialkit.core import (
Dataset,
EvalCase,
EvalDefinition,
ScoreValue,
TrialContext,
rate,
scorer,
)
from trialkit.pytest import eval_suite
@scorer(name="exact", version="1")
def exact(case: EvalCase[str, str], output: str, ctx: TrialContext) -> ScoreValue:
passed = output == case.expected
return ScoreValue(1.0 if passed else 0.0, passed, f"expected {case.expected}, got {output}")
definition = EvalDefinition(
name="capital-cities",
dataset=Dataset(
id="capitals-v1",
cases=[
EvalCase(id="france", input="France", expected="Paris"),
EvalCase(id="japan", input="Japan", expected="Tokyo"),
],
),
task=lambda country: {"France": "Paris", "Japan": "Tokyo"}.get(country, ""),
scorers=[exact],
checks=[rate("pass rate").at_least(0.9)],
)
test_capital_cities = eval_suite(definition)

eval_suite(definition, *, name=None, reporters=None, trial_timeout_ms=120_000, run_timeout_ms=None, max_errors=None, concurrency=1, labels=None, config=None, filter=None, partial_checks="skip", token_budget=None) returns a pytest test function. All keywords are optional and accept None as their default. Their meanings match the table above, except:

Option Default Meaning
name None Suite display name in adapter messages; defaults to the eval name.
concurrency 1 Concurrent cases, an integer ≥ 1 or math.inf; only the definition form supports it.

pytest collects a test per case id and, when checks exist, a final set checks test. Do not call the returned function or add fixtures or parametrization to it; pytest marks work. The adapter validates the definition, options and case ids at declaration. Each pytest session creates a fresh run when its first selected test runs. Selected cases can start ahead up to concurrency; they cannot use setup from a future test. Results remain in dataset order, and concurrent token use can overshoot the budget.

Python has no @trialkit/replay: VCR.py and pytest-recording (pip install vcrpy pytest-recording) record the HTTP replies of your model client once and replay them in CI with no key. A request that is not in the cassette raises vcr.errors.CannotOverwriteExistingCassetteException, which is a fail. Raise InfrastructureError for it, so the trial ends in error: record again.

test_triage.py
import os
import openai
import pytest
from trialkit.core import InfrastructureError
from trialkit.pytest import describe_eval
from vcr.errors import CannotOverwriteExistingCassetteException
@pytest.fixture(scope="module")
def vcr_config():
return {
"filter_headers": ["authorization"],
"match_on": ["method", "scheme", "host", "port", "path", "query", "body"],
}
def triage(ticket: str) -> str:
client = openai.OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "replay"))
try:
reply = client.chat.completions.create(
model="gpt-4o-mini", messages=[{"role": "user", "content": ticket}]
)
except CannotOverwriteExistingCassetteException as exc:
raise InfrastructureError("no recording for this request: record again") from exc
return reply.choices[0].message.content
pytestmark = [describe_eval("triage", task=triage, trials=2), pytest.mark.vcr]
  • A person records with pytest --record-mode=once; commit cassettes/ next to the test file. CI uses the default mode, none, and needs no key. To record a changed case again, delete its cassette file first.
  • VCR.py ignores the body by default, so a changed prompt would replay the old reply; match_on adds it. Without filter_headers, the cassette held Authorization: Bearer <key>.
  • A cassette plays each reply once per recording, so a replay with more trials than were recorded also raises the exception.

Verified with vcrpy 9.4.2, pytest-recording 0.14.1, openai 3.46.0 and pytest 9.2.0 against a local OpenAI-compatible server. The VCR.py facts above come from that run. Other versions or HTTP clients can wrap the exception; catch what your client raises.

Outcome pytest result
pass Pass.
fail Fail with AssertionError, trial counts and the failing reason.
error Fail with InfrastructureError, or the original misuse error class.
skip Skip with the case reason.
xfail Skip with xfail: <reason>.
xpass Pass with a warning.

Verbose output and the JUnit trials property show counted passes/trials and any errors. The last failing test-form assertion retains its traceback. A set-check test fails if a check fails or errors, skips if all checks skip, and otherwise passes; a mix of passing and skipped checks warns.

Failed or errored cases include bounded trial logs in a trialkit logs report section. Passing-case logs stay in reporters and the history report.

Use -k, node ids, --deselect or --lf to select tests. Deselected cases can make checks partial; this intentional selection does not by itself fail the session. partial_checks="fail" makes a partial check fail. A case that skips while running or is dropped by filter is not a pytest selection: a partial check then fails the session at closing. A dataset or eval_case skip mark intentionally excludes the case from checks. To select only the set checks, use -k checks or its node id; -k "set checks" is a pytest usage error.

The plugin moves set checks after the last case of its eval, including with random-order plugins and --ff or --lf. A case still run after it fails. A runner retry reports the first case result without rerunning trials. unittest.TestCase tests in a marked module remain ordinary tests.

A session hook finishes every started run, even after -x, Ctrl+C or deselection of the set checks test. It runs remaining checks, emits RunCompleted, closes reporters and reports unreported check failures. It fails an otherwise successful session for a reporter or close error, a run stopped by a guard, a run where cases ran but none reached a verdict, unexpected partial checks, or a token budget with no reported usage. It does not fail an intentional selection of only set checks or a suite with no selected tests. Existing nonzero exit statuses remain unchanged.

A reporter you provide replaces the default JSON Lines file. To keep both console and file output, pass reporters=[console_reporter(), jsonl_file(dir="eval-results")]. pytest captures console output from set checks; use -s or -rP to see it.

Each eval has its own asyncio loop, closed at session end. Async tests need no asyncio plugin. With pytest-asyncio, an async fixture may belong to a different loop from the eval body; create loop-bound objects in the test or task. Ctrl+C cancels the running case, and closing records the run as stopped by signal.

Each worker has its own run and file. Use --dist loadfile so a module’s eval and its set checks stay together and definition-form look-ahead can use concurrency. Other xdist distributions run each worker’s cases sequentially: a worker cannot know which cases xdist assigned elsewhere. --dist loadscope also keeps a class eval together. A split eval with partial set checks fails the session. The controller reports worker closing errors.

The plugin adds timeout(0) to eval tests so pytest-timeout cannot end a case mid-trial. Core still applies trial_timeout_ms. An explicit timeout mark remains in force and can end the process before the run closes, especially with the plugin’s thread method on Windows.