Skip to content

@trialkit/report-mlflow

Send an eval run to MLflow as one run with tags, params, metrics, and trace assessments. The same mapping uploads completed JSON Lines files. loadPrompt reads a versioned MLflow Prompt Registry prompt for a run’s core config.

Terminal window
npm install --save-dev @trialkit/report-mlflow @trialkit/core

@trialkit/core is a peer. Node.js 22.23.4+. The root entry uses web APIs (fetch, URL, Date); @trialkit/report-mlflow/node also uses files.

import { runEval } from '@trialkit/core';
import { mlflowReporter } from '@trialkit/report-mlflow';
import { triageEval } from './triage.eval';
await runEval(triageEval, {
labels: { model: 'openai:gpt-5-mini', prompt: 'triage@3' },
reporters: [mlflowReporter({ host: 'http://localhost:5000', experimentName: 'triage-evals' })],
});
  • One core run becomes one MLflow run; experimentName is found or created.
  • Keep jsonlFile alongside this reporter when the JSON Lines file is the canonical record.
  • A dotted key needs backticks in an MLflow filter: metrics.\cases.pass_rate` > 0.9`.
Entry Exports
@trialkit/report-mlflow mlflowReporter, loadPrompt, MlflowReporterOptions, LoadPromptOptions, Prompt
@trialkit/report-mlflow/node uploadJsonl, UploadResult

mlflowReporter(options) and uploadJsonl(paths, options) share MlflowReporterOptions.

Option Default Meaning
host Required HTTP(S) tracking-server or Databricks workspace URL; no query or fragment, optional path.
experimentId Required without experimentName Existing experiment id; give exactly one experiment selector.
experimentName Required without experimentId Existing experiment name, or a name to create.
token No Authorization header Non-empty bearer token, or a function returning one for each request.
runName <eval name> <RunStarted time> Name of each MLflow run.
requestTimeoutMs 30000 Per-request integer milliseconds from 1 through 2147483647, including reply-body reading.
fetch Global fetch Fetch implementation for retries, proxying, or custom headers.

The reporter validates options at construction; host, experiment selector, runName, and literal tokens must be non-empty, and invalid types throw TypeError or RangeError. It keeps state per core run, so one reporter can serve concurrent runs. It looks up a named experiment once; if another process creates it first, it looks it up again.

On RunStarted, the reporter creates the MLflow run and logs initial params. On RunCompleted, it logs final tags, metrics, judge params, marks the run FINISHED, then sends trace assessments. A reporter failure stops reporting that core run, records a reporter error, and never changes its eval outcome.

The reporter ignores trial log attachments. Keep a JSON Lines reporter to retain them locally.

MLflow run field Value
Experiment experimentId, or the found/created experimentName.
Name runName, or <eval name> <RunStarted time>.
Start / end RunStarted / RunCompleted time.
Status RUNNING, then FINISHED; FAILED when reporting fails before completion. A stopped core run is still FINISHED.
Kind Keys and values
Tags at start trialkit.eval, trialkit.run_id, trialkit.dataset.hash, optional trialkit.config_hash, and trialkit.label.<key> for every run label.
Tags at end trialkit.ok; trialkit.stopped_by only when core recorded stoppedBy.
Params dataset.id, policy.trials, policy.verdict, policy.earlyStop, policy.errorRetries, library.name, library.version.
Judge params For the first score of each scorer with data.judge: judge.<scorer>.model, .template, and .promptHash.

All metrics use step 0 and the RunCompleted time.

Metric Value
cases.<outcome> Counts for pass, fail, error, skip, xfail, and xpass.
checks.<outcome> Counts for pass, fail, error, and skip.
tokens.input, tokens.output, tokens.cached, tokens.reasoning, cost Recorded run totals when present; cost is USD.
cases.pass_rate, trials.pass_rate Pass rates and .low / .high 95% Wilson bounds.
tag.<tag>.cases.pass_rate, tag.<tag>.trials.pass_rate The same rates and bounds for each case tag.

Rates omit error, skip, xfail, and xpass; a zero-denominator rate is not logged. User-derived key segments replace every character except Unicode letters or digits, _, -, ., and space with _ (including / and :). Keys are at most 250 characters. Colliding label or scorer keys are disambiguated with a stable hash; colliding tag keys share one metric group.

The reporter sends values unchanged. MLflow limits param values to 6,000 characters and tag values to 8,000; depending on server settings and version, it truncates a longer value or rejects the whole batch. Not built: config contents and trial metrics. Keep JSONL for these facts.

A trial with traceId sends an assessment per score: its scorer name, LLM_JUDGE source for a judge score or CODE otherwise, <name>@<version>, value, optional rationale, time, and metadata (mlflow.assessment.sourceRunId, core run id, case id, trial number, and pass). An error trial with a trace sends trialkit_error, CODE, no value, and the error name, message, and up to 1000 stack characters. Trials without a trace still count in metrics. The package never starts or registers tracing; set ctx.traceId after your application creates a trace.

Assessment HTTP-status failures do not stop later assessments; a no-reply failure stops the remainder and reports one aggregate infrastructure error. The MLflow run remains FINISHED after assessment failures.

uploadJsonl(paths, options) recursively reads named files and directories with core readRuns, then uses the same mapping. paths must be an array of non-empty strings. It sends files and their runs in order, reads a named file once, and does not deduplicate against prior uploads in MLflow.

UploadResult.runs contains { runId, mlflowRunId }; skipped contains { file, reason }. It skips incomplete runs, duplicate run ids in later files, empty or unreadable files, and a whole file with an invalid event line. Missing paths, invalid options, and failed requests reject; successful uploads before a later failure remain in MLflow.

loadPrompt returns Prompt ({ template, version, uri }) from an open-source MLflow Prompt Registry. Store that plain JSON result in core config; loading by alias records the resolved immutable version. Text templates stay strings; chat templates are arrays of message objects, and {{variables}} are not filled.

Option Default Meaning
name Required Non-empty prompt name.
alias Required without version Non-empty alias, such as production; latest is MLflow’s highest version.
version Required without alias Safe integer from 1.
host, token, requestTimeoutMs, fetch As above Shared connection options.

Give exactly one of alias and version. The function requests the registry’s model-version endpoint or alias endpoint, verifies that it is a prompt, and returns prompts:/<name>/<version>. Wrong options reject before a request; request failures are infrastructure errors; a registered model that is not a prompt, a missing template, or an invalid chat-template list is a normal Error. It never retries or caches. Databricks Unity Catalog prompts are not supported.

A non-2xx response, failed fetch, timeout, malformed JSON response, or missing expected server field is an infrastructure error with method and path; it never includes the token. Requests do not retry. The timeout uses a ref’d timer and also applies when a custom fetch ignores its abort signal.

runs/log-batch requests contain at most 1000 metrics, 100 params, 100 tags, and 1000 entries total; the reporter splits batches in order. runs/create receives the first 100 tags. It does not upload the JSON Lines file as an artifact, log datasets or expected values, create traces, or check whether MLflow already has a run.

Use the workspace URL as host, a bearer personal-access or renewable OAuth token as token, and experimentName as an existing workspace path such as /Shared/trialkit-evals, or give experimentId. The reporter creates a missing named experiment; its parent folder must exist.

Call MLflow Databricks
Experiments and runs REST API Experiments API
runs/log-batch limits Log Batch Log a batch
Trace assessments Assessment data Create assessment
Prompt versions and aliases Model Version and alias Not supported

The assessment endpoint and body follow RestStore.create_assessment; key rules follow mlflow/utils/validation.py. Prompt tags follow mlflow/prompt/constants.py.