trialkit-report-mlflow
Send trialkit runs to an MLflow tracking server or Databricks, or load a prompt from the MLflow Prompt Registry. Live reporting and JSON Lines upload use the same mapping as @trialkit/report-mlflow.
Install
Section titled “Install”pip install trialkit-report-mlflowExample
Section titled “Example”Start a tracking server with mlflow server, then give the reporter to run_eval:
import asyncio
from trialkit.core import run_evalfrom trialkit.report_mlflow import mlflow_reporter
from triage_eval import definition # your EvalDefinition
reporter = mlflow_reporter(host="http://localhost:5000", experiment_name="triage-evals")result = asyncio.run( run_eval( definition, labels={"model": "openai:gpt-5-mini", "prompt": "triage@3"}, reporters=[reporter], ))- Add
jsonl_file(dir="eval-results")to the reporters to keep the canonical JSON Lines record. Checkresult.reporter_errors; a reporter error does not change the eval verdict. - The reporter makes one MLflow run per eval run. It does not create traces, upload the JSON Lines file as an artifact, log datasets or retry requests.
- Requires Python 3.11 or later and
trialkit-core. HTTP uses the standard library; themlflowpackage is not required. The package is not yet published.
The reporter ignores trial log attachments. Keep JSON Lines to retain them locally.
Options
Section titled “Options”mlflow_reporter(*, ...) and await upload_jsonl(paths, *, ...) use these keyword options. None selects the default where one exists.
| Option | Default | Meaning |
|---|---|---|
host |
Required | Tracking server or Databricks workspace URL; no query or fragment. |
experiment_id |
None |
Id of an existing experiment. Give exactly one of this and experiment_name. |
experiment_name |
None |
Look up or create this experiment. On Databricks, use a workspace path whose folder exists. |
token |
None |
Bearer token string, or plain or async function returning one. The function is called before every request and its result is not cached. |
run_name |
None |
Defaults to <eval name> <RunStarted time>. |
request_timeout_ms |
30_000 |
Per-request deadline, including token lookup, transport and response body, in milliseconds from 1 to 2,147,483,647. Integral floats are accepted. |
transport |
None |
Custom Transport; default uses urllib.request. |
Invalid types and incompatible or missing experiment options raise TypeError at declaration; an out-of-range timeout raises ValueError. One reporter can serve concurrent runs. It finds or creates the experiment, creates an MLflow run at RunStarted, and logs metrics, marks it FINISHED, and sends assessments at RunCompleted. A stopped eval can still be FINISHED; inspect the trialkit.ok and trialkit.stopped_by tags.
Transport
Section titled “Transport”Transport is a plain or async callable from HttpRequest to HttpResponse. A plain callable runs in a daemon thread. It must stop its own request at the supplied timeout; cancellation cannot stop a blocked Python thread.
| Type | Fields | Meaning |
|---|---|---|
HttpRequest |
method, url, headers, body, timeout |
GET or POST; lowercase headers; UTF-8 JSON bytes or None body; timeout in seconds. repr hides headers, which can contain the token. |
HttpResponse |
status, body=b"" |
Return the complete response for any HTTP status. Raise if there is no response. |
The default transport follows urllib proxy settings, creates one connection per request and follows no redirect. Supply a transport for retries, another proxy policy, a session or client certificates. A request deadline cancels async work or stops waiting for a plain function; the daemon thread may continue.
What it sends
Section titled “What it sends”The TypeScript mapping defines the shared MLflow tags, params, metrics, names and assessments. Python emits the same requests for the same events. In particular, errors and skipped or expected-failure cases do not enter pass rates; zero-denominator rates are omitted. The reporter does not log expected values, which events do not contain. User-derived keys are sanitized and shortened to MLflow’s 250-character limit, with stable suffixes to distinguish colliding label and scorer names. Log batches respect the 1,000-metric, 100-param, 100-tag and 1,000-total limits.
The reporter sends values unchanged. MLflow limits param values to 6,000 characters and tag values to 8,000; depending on server settings and version, it truncates a longer value or rejects the whole batch. Not built: config contents and trial metrics. Keep JSON Lines for these facts.
Traces
Section titled “Traces”The application creates and exports its own MLflow trace. Set ctx.trace_id to its string id and return a scorer reason to add a rationale. The reporter sends assessments after the run ends; flush the tracing exporter before the task returns, or upload the JSON Lines file after the trace arrives. A trial without a trace id still counts in metrics but has no assessment. Error trials receive an error assessment, not a failing score. Unity Catalog trace ids beginning trace:/ are not supported.
If an assessment gets an HTTP error, the reporter attempts the others and raises one InfrastructureError with an ExceptionGroup of failures. If there is no response or a timeout, it stops after the first such failure. The MLflow run stays FINISHED with its metrics; some assessments are missing.
upload_jsonl(paths, ...)
Section titled “upload_jsonl(paths, ...)”await upload_jsonl(paths, *, host, token=None, experiment_id=None, experiment_name=None, run_name=None, request_timeout_ms=30_000, transport=None) sends completed runs using the reporter mapping. paths is a list or tuple of file or directory paths; directories are searched recursively for .jsonl. The shared read_runs reader sorts paths and reads each file once.
UploadResult.runs is a tuple of UploadedRun(run_id, mlflow_run_id); UploadResult.skipped is a tuple of Skipped(file, reason). Incomplete runs and unusable files are skipped. A file with a bad line is skipped in full. A missing path, invalid option or failed request raises; earlier uploads remain. The uploader does not check for existing MLflow runs, so uploading twice creates duplicates. A given run_name applies to every uploaded run.
load_prompt(...)
Section titled “load_prompt(...)”await load_prompt(*, host, name, alias=None, version=None, token=None, request_timeout_ms=30_000, transport=None) reads an open-source MLflow Prompt Registry prompt. Put the returned Prompt in the run’s config to record what was tested.
| Option | Default | Meaning |
|---|---|---|
host, name |
Required | Tracking server URL and registered prompt name. |
alias, version |
None |
Give exactly one. version is an integer from 1 to 2^53−1; alias resolves to a version at call time. |
token, request_timeout_ms, transport |
As in Options | Connection options shared with the reporter. |
Prompt is a dict with template (text or a list of chat messages), version (int) and uri (prompts:/<name>/<version>). Placeholders are not filled. Invalid options raise TypeError or ValueError; request, timeout and malformed-response failures raise InfrastructureError; a registered model that is not a prompt or a missing/invalid template raises RuntimeError. No request is retried. Requests and prompt limits match loadPrompt.
Databricks
Section titled “Databricks”Use the workspace host without a UI path or ?o= query, a personal access token or service-principal OAuth token, and an experiment id or a workspace path such as /Shared/trialkit-evals. For a short-lived OAuth token, pass a token function that caches and refreshes it; trialkit calls the function for every request. load_prompt reads the open-source registry, not Unity Catalog prompts.
Errors
Section titled “Errors”Request failures, non-2xx replies and unreadable required responses raise InfrastructureError. A failed live reporter is recorded in RunResult.reporter_errors and receives no more events for that run. A failure before run creation leaves no MLflow run; one after creation but before metric logging marks it FAILED; assessment failures leave it FINISHED. Request messages omit tokens. Experiment lookup is retried on the next run after a failure, but individual requests have no automatic retry.