@trialkit/report-mlflow
Send an eval run to MLflow as one run with tags, params, metrics, and trace assessments. The same mapping uploads completed JSON Lines files. loadPrompt reads a versioned MLflow Prompt Registry prompt for a run’s core config.
Install
Section titled “Install”npm install --save-dev @trialkit/report-mlflow @trialkit/core@trialkit/core is a peer. Node.js 22.23.4+. The root entry uses web APIs (fetch, URL, Date); @trialkit/report-mlflow/node also uses files.
Example
Section titled “Example”import { runEval } from '@trialkit/core';import { mlflowReporter } from '@trialkit/report-mlflow';import { triageEval } from './triage.eval';
await runEval(triageEval, { labels: { model: 'openai:gpt-5-mini', prompt: 'triage@3' }, reporters: [mlflowReporter({ host: 'http://localhost:5000', experimentName: 'triage-evals' })],});- One core run becomes one MLflow run;
experimentNameis found or created. - Keep
jsonlFilealongside this reporter when the JSON Lines file is the canonical record. - A dotted key needs backticks in an MLflow filter:
metrics.\cases.pass_rate` > 0.9`.
API and options
Section titled “API and options”| Entry | Exports |
|---|---|
@trialkit/report-mlflow |
mlflowReporter, loadPrompt, MlflowReporterOptions, LoadPromptOptions, Prompt |
@trialkit/report-mlflow/node |
uploadJsonl, UploadResult |
mlflowReporter(options) and uploadJsonl(paths, options) share MlflowReporterOptions.
| Option | Default | Meaning |
|---|---|---|
host |
Required | HTTP(S) tracking-server or Databricks workspace URL; no query or fragment, optional path. |
experimentId |
Required without experimentName |
Existing experiment id; give exactly one experiment selector. |
experimentName |
Required without experimentId |
Existing experiment name, or a name to create. |
token |
No Authorization header |
Non-empty bearer token, or a function returning one for each request. |
runName |
<eval name> <RunStarted time> |
Name of each MLflow run. |
requestTimeoutMs |
30000 |
Per-request integer milliseconds from 1 through 2147483647, including reply-body reading. |
fetch |
Global fetch |
Fetch implementation for retries, proxying, or custom headers. |
The reporter validates options at construction; host, experiment selector, runName, and literal tokens must be non-empty, and invalid types throw TypeError or RangeError. It keeps state per core run, so one reporter can serve concurrent runs. It looks up a named experiment once; if another process creates it first, it looks it up again.
MLflow mapping
Section titled “MLflow mapping”On RunStarted, the reporter creates the MLflow run and logs initial params. On RunCompleted, it logs final tags, metrics, judge params, marks the run FINISHED, then sends trace assessments. A reporter failure stops reporting that core run, records a reporter error, and never changes its eval outcome.
The reporter ignores trial log attachments. Keep a JSON Lines reporter to retain them locally.
| MLflow run field | Value |
|---|---|
| Experiment | experimentId, or the found/created experimentName. |
| Name | runName, or <eval name> <RunStarted time>. |
| Start / end | RunStarted / RunCompleted time. |
| Status | RUNNING, then FINISHED; FAILED when reporting fails before completion. A stopped core run is still FINISHED. |
| Kind | Keys and values |
|---|---|
| Tags at start | trialkit.eval, trialkit.run_id, trialkit.dataset.hash, optional trialkit.config_hash, and trialkit.label.<key> for every run label. |
| Tags at end | trialkit.ok; trialkit.stopped_by only when core recorded stoppedBy. |
| Params | dataset.id, policy.trials, policy.verdict, policy.earlyStop, policy.errorRetries, library.name, library.version. |
| Judge params | For the first score of each scorer with data.judge: judge.<scorer>.model, .template, and .promptHash. |
All metrics use step 0 and the RunCompleted time.
| Metric | Value |
|---|---|
cases.<outcome> |
Counts for pass, fail, error, skip, xfail, and xpass. |
checks.<outcome> |
Counts for pass, fail, error, and skip. |
tokens.input, tokens.output, tokens.cached, tokens.reasoning, cost |
Recorded run totals when present; cost is USD. |
cases.pass_rate, trials.pass_rate |
Pass rates and .low / .high 95% Wilson bounds. |
tag.<tag>.cases.pass_rate, tag.<tag>.trials.pass_rate |
The same rates and bounds for each case tag. |
Rates omit error, skip, xfail, and xpass; a zero-denominator rate is not logged. User-derived key segments replace every character except Unicode letters or digits, _, -, ., and space with _ (including / and :). Keys are at most 250 characters. Colliding label or scorer keys are disambiguated with a stable hash; colliding tag keys share one metric group.
The reporter sends values unchanged. MLflow limits param values to 6,000 characters and tag values to 8,000; depending on server settings and version, it truncates a longer value or rejects the whole batch. Not built: config contents and trial metrics. Keep JSONL for these facts.
A trial with traceId sends an assessment per score: its scorer name, LLM_JUDGE source for a judge score or CODE otherwise, <name>@<version>, value, optional rationale, time, and metadata (mlflow.assessment.sourceRunId, core run id, case id, trial number, and pass). An error trial with a trace sends trialkit_error, CODE, no value, and the error name, message, and up to 1000 stack characters. Trials without a trace still count in metrics. The package never starts or registers tracing; set ctx.traceId after your application creates a trace.
Assessment HTTP-status failures do not stop later assessments; a no-reply failure stops the remainder and reports one aggregate infrastructure error. The MLflow run remains FINISHED after assessment failures.
Upload JSON Lines
Section titled “Upload JSON Lines”uploadJsonl(paths, options) recursively reads named files and directories with core readRuns, then uses the same mapping. paths must be an array of non-empty strings. It sends files and their runs in order, reads a named file once, and does not deduplicate against prior uploads in MLflow.
UploadResult.runs contains { runId, mlflowRunId }; skipped contains { file, reason }. It skips incomplete runs, duplicate run ids in later files, empty or unreadable files, and a whole file with an invalid event line. Missing paths, invalid options, and failed requests reject; successful uploads before a later failure remain in MLflow.
loadPrompt(options)
Section titled “loadPrompt(options)”loadPrompt returns Prompt ({ template, version, uri }) from an open-source MLflow Prompt Registry. Store that plain JSON result in core config; loading by alias records the resolved immutable version. Text templates stay strings; chat templates are arrays of message objects, and {{variables}} are not filled.
| Option | Default | Meaning |
|---|---|---|
name |
Required | Non-empty prompt name. |
alias |
Required without version |
Non-empty alias, such as production; latest is MLflow’s highest version. |
version |
Required without alias |
Safe integer from 1. |
host, token, requestTimeoutMs, fetch |
As above | Shared connection options. |
Give exactly one of alias and version. The function requests the registry’s model-version endpoint or alias endpoint, verifies that it is a prompt, and returns prompts:/<name>/<version>. Wrong options reject before a request; request failures are infrastructure errors; a registered model that is not a prompt, a missing template, or an invalid chat-template list is a normal Error. It never retries or caches. Databricks Unity Catalog prompts are not supported.
Errors and limits
Section titled “Errors and limits”A non-2xx response, failed fetch, timeout, malformed JSON response, or missing expected server field is an infrastructure error with method and path; it never includes the token. Requests do not retry. The timeout uses a ref’d timer and also applies when a custom fetch ignores its abort signal.
runs/log-batch requests contain at most 1000 metrics, 100 params, 100 tags, and 1000 entries total; the reporter splits batches in order. runs/create receives the first 100 tags. It does not upload the JSON Lines file as an artifact, log datasets or expected values, create traces, or check whether MLflow already has a run.
Databricks
Section titled “Databricks”Use the workspace URL as host, a bearer personal-access or renewable OAuth token as token, and experimentName as an existing workspace path such as /Shared/trialkit-evals, or give experimentId. The reporter creates a missing named experiment; its parent folder must exist.
Sources
Section titled “Sources”| Call | MLflow | Databricks |
|---|---|---|
| Experiments and runs | REST API | Experiments API |
runs/log-batch limits |
Log Batch | Log a batch |
| Trace assessments | Assessment data | Create assessment |
| Prompt versions and aliases | Model Version and alias | Not supported |
The assessment endpoint and body follow RestStore.create_assessment; key rules follow mlflow/utils/validation.py. Prompt tags follow mlflow/prompt/constants.py.