Skip to content

@trialkit/report-otel

Export run, case, trial, and recorded agent spans through an OpenTelemetry tracer. The reporter never registers a provider.

Terminal window
npm install --save-dev @trialkit/report-otel @trialkit/core @opentelemetry/api

Peers: @trialkit/core, @opentelemetry/api ^1.0.0; no other dependencies. CI tests the API floor. Node.js 22.23.4+. Register a tracer provider or no spans are exported.

import { runEval } from '@trialkit/core';
import { otelReporter } from '@trialkit/report-otel';
await runEval(definition, { reporters: [otelReporter()] });
  • In a runner suite, reporters replaces the default file; add jsonlFile({ dir: 'eval-results' }) if needed.
  • Shared reporters keep state per run. Tracer errors become core reporter errors and disable that reporter for the run.
  • genAi: true adds Datadog LLM Observability attributes without removing trialkit.*.

Exports: otelReporter(options?), OtelReporterOptions, OtelTracer, OtelSpan, OtelSpanOptions. Custom tracers provide startSpan; spans provide spanContext, setAttribute, setStatus, addEvent, and end. OtelSpanOptions accepts optional startTime and attributes; times are Date or [seconds, nanoseconds].

Option Default Meaning
tracer trace.getTracer('@trialkit/report-otel') Tracer with startSpan.
genAi false Add Datadog-compatible GenAI attributes.

An invalid tracer or non-boolean genAi throws TypeError at creation.

Span Parent Attributes
trialkit.run <eval name> Active context trialkit.eval, trialkit.run_id, trialkit.dataset.hash, trialkit.label.<key>, trialkit.ok, optional trialkit.stopped_by and trialkit.config_hash.
trialkit.case <case id> Run trialkit.case_id, trialkit.outcome.
trialkit.trial <case id> #<n> Case trialkit.case_id, trialkit.trial, trialkit.outcome, trialkit.score.<name>.value / .pass, optional trialkit.tokens.input / .output and trialkit.task_trace_id.

Event times set span start and end. CheckCompleted adds trialkit.check on the run with trialkit.check.name, .outcome, and optional .value and .reason. An errored trial and its case get ERROR status and the error message; a skipped open trial ends at CaseCompleted with outcome skip.

Each valid trialkit.log.v1 attachment adds a trialkit.log event on its trial span at the recorded offset. Its attributes are trialkit.log.sequence, trialkit.log.message and optional JSON text trialkit.log.data. Logs before a skip remain on the trial span. No OpenTelemetry logging provider is needed.

Recorded TrialCompleted.spans become trial descendants using recorded parent ids. Missing parents attach to the trial; parents start first and children end first. Offsets are milliseconds from trial start with nanosecond precision: absent start uses trial start, absent end uses the later of trial completion and start, and end before start is clamped. Primitive attributes and homogeneous primitive arrays remain attributes; objects, mixed arrays, and null become JSON text. Captured messages and tool data can be sensitive.

Not built: config contents, run-level usage and cost, cached and reasoning tokens, and trial metrics. Keep JSONL for these facts.

Set genAi: true. Run and case spans get gen_ai.operation.name=unknown; trial spans get invoke_agent, optional gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, and error error.type. _dd.ml_obs.metadata contains eval, run_id, and labels; case metadata adds case_id and outcome, trial metadata adds case_id, trial, outcome, scores, and optional task_trace_id; run metadata adds ok and optional stopped_by.

Configure the SDK’s OTLP exporter for http/protobuf, the Datadog site endpoint, headers dd-api-key=<API key>,dd-otlp-source=llmobs, and resource service.name (the Datadog ml_app). Flush the provider on shutdown. Scores need Datadog’s external evaluations API to appear as evaluations; trial spans contain no input, output, model, or cost. Live task SDK spans may use another trace because the trial span is not made active; recorded spans are children. Datadog behavior was checked against documentation, not a live account.