Skip to content

Agent sessions

Use spans when an agent’s calls, tools or intermediate messages are part of the behavior to assess.

import type { TrialContext } from '@trialkit/core';
async function answer(question: string, ctx: TrialContext) {
ctx.log('Looking up an order', { source: 'orders' });
const result = await executeTool('lookup_order', { question }, { signal: ctx.signal });
ctx.span({
name: 'execute_tool lookup_order',
spanId: 'tool-0',
attributes: { 'gen_ai.operation.name': 'execute_tool', 'gen_ai.tool.name': 'lookup_order' },
status: 'ok',
});
return result;
}
  • Record each parent before its children. ctx.spans starts empty for a trial and is read-only; call ctx.addUsage because span attributes do not count tokens.
  • Capture tool arguments, results and messages only when they are safe to send or store. Redact before a trajectory judge sees them.
  • Logs are opt-in trial attachments (Python: ctx.log("Looking up an order", data={"source": "orders"})). They have a sequence and a monotonic offset, not a span parent. Redact before capture: a collapsed report value is not redaction.
  • View the trace tree and the log list with trialkit report eval-results (HTML, or --format markdown for bounded text). A failed or errored case also shows bounded logs in the runner output.

The core reference defines span fields, toolCalls, modelCalls, AI SDK capture and OpenTelemetry import.

Use toolCalls for required calls and order, toolPolicy for forbidden tools, a limit or repeated calls, and trajectoryJudge for a semantic process rule.

import { toolPolicy } from '@trialkit/scorers';
export const policy = toolPolicy({
forbidden: ['delete_order'],
maxCalls: 5,
allowRepeated: false,
});

Use it as any scorer: await score(policy, await run('Where is order 42?')). A missing span list is missing evidence, not a passing policy; a recorded model-only session has zero tool calls. Compare toolCalls(ctx.spans) and modelCalls(ctx.spans) by their index values only after confirming both calls exist. Order proves recording order, not semantic use. See the scorers and judge references.

Use the converter when your app already collects finished SDK spans locally.

import type { TrialContext } from '@trialkit/core';
import { type ReadableSpan, spansOf } from '@trialkit/core/otel';
export function record(finishedSpans: readonly ReadableSpan[], ctx: TrialContext) {
for (const span of spansOf(finishedSpans)) ctx.span(span);
}

Pass a local collector to the app’s tracing provider; do not register a global provider. The converter orders included parents before children, rejects mixed traces, imports attributes and status only, and omits times unless the supplied origin is this trial’s start. See the core reference.