@trialkit/report-otel
Export run, case, trial, and recorded agent spans through an OpenTelemetry tracer. The reporter never registers a provider.
Install
Section titled “Install”npm install --save-dev @trialkit/report-otel @trialkit/core @opentelemetry/apiPeers: @trialkit/core, @opentelemetry/api ^1.0.0; no other dependencies. CI tests the API floor. Node.js 22.23.4+. Register a tracer provider or no spans are exported.
Example
Section titled “Example”import { runEval } from '@trialkit/core';import { otelReporter } from '@trialkit/report-otel';
await runEval(definition, { reporters: [otelReporter()] });- In a runner suite,
reportersreplaces the default file; addjsonlFile({ dir: 'eval-results' })if needed. - Shared reporters keep state per run. Tracer errors become core reporter errors and disable that reporter for the run.
genAi: trueadds Datadog LLM Observability attributes without removingtrialkit.*.
Reference
Section titled “Reference”Exports: otelReporter(options?), OtelReporterOptions, OtelTracer, OtelSpan, OtelSpanOptions. Custom tracers provide startSpan; spans provide spanContext, setAttribute, setStatus, addEvent, and end. OtelSpanOptions accepts optional startTime and attributes; times are Date or [seconds, nanoseconds].
| Option | Default | Meaning |
|---|---|---|
tracer |
trace.getTracer('@trialkit/report-otel') |
Tracer with startSpan. |
genAi |
false |
Add Datadog-compatible GenAI attributes. |
An invalid tracer or non-boolean genAi throws TypeError at creation.
| Span | Parent | Attributes |
|---|---|---|
trialkit.run <eval name> |
Active context | trialkit.eval, trialkit.run_id, trialkit.dataset.hash, trialkit.label.<key>, trialkit.ok, optional trialkit.stopped_by and trialkit.config_hash. |
trialkit.case <case id> |
Run | trialkit.case_id, trialkit.outcome. |
trialkit.trial <case id> #<n> |
Case | trialkit.case_id, trialkit.trial, trialkit.outcome, trialkit.score.<name>.value / .pass, optional trialkit.tokens.input / .output and trialkit.task_trace_id. |
Event times set span start and end. CheckCompleted adds trialkit.check on the run with trialkit.check.name, .outcome, and optional .value and .reason. An errored trial and its case get ERROR status and the error message; a skipped open trial ends at CaseCompleted with outcome skip.
Each valid trialkit.log.v1 attachment adds a trialkit.log event on its trial span at the recorded offset. Its attributes are trialkit.log.sequence, trialkit.log.message and optional JSON text trialkit.log.data. Logs before a skip remain on the trial span. No OpenTelemetry logging provider is needed.
Recorded TrialCompleted.spans become trial descendants using recorded parent ids. Missing parents attach to the trial; parents start first and children end first. Offsets are milliseconds from trial start with nanosecond precision: absent start uses trial start, absent end uses the later of trial completion and start, and end before start is clamped. Primitive attributes and homogeneous primitive arrays remain attributes; objects, mixed arrays, and null become JSON text. Captured messages and tool data can be sensitive.
Not built: config contents, run-level usage and cost, cached and reasoning tokens, and trial metrics. Keep JSONL for these facts.
Send to Datadog LLM Observability
Section titled “Send to Datadog LLM Observability”Set genAi: true. Run and case spans get gen_ai.operation.name=unknown; trial spans get invoke_agent, optional gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, and error error.type. _dd.ml_obs.metadata contains eval, run_id, and labels; case metadata adds case_id and outcome, trial metadata adds case_id, trial, outcome, scores, and optional task_trace_id; run metadata adds ok and optional stopped_by.
Configure the SDK’s OTLP exporter for http/protobuf, the Datadog site endpoint, headers dd-api-key=<API key>,dd-otlp-source=llmobs, and resource service.name (the Datadog ml_app). Flush the provider on shutdown. Scores need Datadog’s external evaluations API to appear as evaluations; trial spans contain no input, output, model, or cost. Live task SDK spans may use another trace because the trial span is not made active; recorded spans are children. Datadog behavior was checked against documentation, not a live account.