Skip to content

Ingest trace files

If your agent already exports traces, calibrated-agent-evaluation ingest reads them with no change to the agent. Here it reads the repository's Strands crew captures:

$ calibrated-agent-evaluation ingest fixtures/raw/strands
calibrated-agent-evaluation ingest: 6 ingested, 0 refused, 0 skipped
  [ok] fixtures/raw/strands/CLM-4471.otlp.json → CLM-4471 (strands, 5 span(s), 4 tool call(s), owned)
  [ok] fixtures/raw/strands/CLM-4473.otlp.json → CLM-4473 (strands, 5 span(s), 4 tool call(s), owned)
  [ok] fixtures/raw/strands/CLM-4475.otlp.json → CLM-4475 (strands, 5 span(s), 4 tool call(s), owned)
  [ok] fixtures/raw/strands/CLM-4477.otlp.json → CLM-4477 (strands, 5 span(s), 4 tool call(s), owned)
  [ok] fixtures/raw/strands/CLM-7720-swarm.otlp.json → CLM-7720-swarm (strands, 2 span(s), 3 tool call(s), owned)
  [ok] fixtures/raw/strands/ORD-3310-as-tool.otlp.json → ORD-3310-as-tool (strands, 2 span(s), 2 tool call(s), owned)

Each line is one trace: the file it came from, the id it was stored under, the format it was read as, how many agent steps (spans) and tool calls were recovered, and whether it is owned (Calibrated Agent Evaluation can see its steps) or probed (only the outer input and output survived). A trace id is the file name up to its first dot, so CLM-4471.otlp.json becomes CLM-4471.

What it reads

Your traces are Example file Extra
OpenTelemetry spans exported as OTLP-JSON (GenAI semantic conventions, OpenLLMetry/Traceloop, OpenInference/Phoenix) session.otlp.json none
Binary OTLP protobuf session.otlp otel
A Strands OTel export run.otlp.json none
Inspect .eval logs 2026-07-23_orderbot.eval inspect
A dataset of traces, one per record week-30.jsonl, slice.parquet none for JSONL, datasets for parquet
A HuggingFace dataset repo Exgentic/agent-llm-traces with --format hf hf and datasets

calibrated-agent-evaluation ingest takes files and directories, and walks directories recursively. It works out the format from each file, and for OpenTelemetry files it also detects which attribute convention the spans use. The integration gallery has a recipe for each source with its real output.

calibrated-agent-evaluation ingest exports/                        # every trace file under exports/
calibrated-agent-evaluation ingest captures/week-30.jsonl          # a dataset, one trace per line
calibrated-agent-evaluation ingest --format hf Exgentic/agent-llm-traces
calibrated-agent-evaluation --backend langfuse ingest exports/     # store into Langfuse instead of ./.calibrated-agent-evaluation

Preview first with --dry-run

--dry-run reports what would happen and writes nothing. Run it on a directory you have not ingested before. The repository's OTel fixture directory deliberately mixes good, degraded and bad files:

$ calibrated-agent-evaluation ingest --dry-run fixtures/raw/otel
trace ORD-DEGRADED degraded to ownership='probed': only a terminal input/output envelope was instrumented (no per-step tool spans and no per-step gen_ai message events)
trace degraded_v2 degraded to ownership='probed': no gen_ai.input/output.messages content and no tool calls recovered from the gen_ai/messages session
calibrated-agent-evaluation ingest: 7 would ingest (2 probed), 2 refused, 1 skipped
  [ok] fixtures/raw/otel/ORD-1001.attrs.otlp.json → 411234653c440f74f9c7bd885663e509 (gen_ai/latest, 1 span(s), 3 tool call(s), owned)
  [ok] fixtures/raw/otel/ORD-1001.build2.otlp.json → 180fed21d2542a2cee4e99fc1f050396 (gen_ai/latest, 1 span(s), 3 tool call(s), owned)
  [ok] fixtures/raw/otel/ORD-1001.events.otlp.json → ad66fcf93d899366e10b923c366a42cb (gen_ai/latest, 1 span(s), 3 tool call(s), owned)
  [ok] fixtures/raw/otel/ORD-1002.events.otlp.json → ORD-1002 (gen_ai/latest, 1 span(s), 3 tool call(s), owned)
  [ok] fixtures/raw/otel/ORD-DEGRADED.otlp.json → ORD-DEGRADED (gen_ai/latest, 0 span(s), 0 tool call(s), probed)
  [ok] fixtures/raw/otel/degraded_v2.otlp.json → degraded_v2 (gen_ai/messages, 0 span(s), 0 tool call(s), probed)
  [ok] fixtures/raw/otel/messages/swebench_tool_call.otlp.json → swebench_tool_call (gen_ai/messages, 1 span(s), 1 tool call(s), owned)
  [REFUSED] fixtures/raw/otel/openinference/weather_tool_call.otlp.json: 2 sources in this batch resolve to trace id 'f026f026f026f026f026f026f026f026' ('fixtures/raw/otel/openinference/weather_tool_call.otlp.json', 'fixtures/raw/otel/openllmetry/weather_tool_call.otlp.json') and the store is keyed on it, so persisting them would overwrite one with the other; rename them so their first dotted filename segments differ, or tag each with a distinct calibrated_agent_evaluation.trace_id span attribute
  [REFUSED] fixtures/raw/otel/openllmetry/weather_tool_call.otlp.json: 2 sources in this batch resolve to trace id 'f026f026f026f026f026f026f026f026' ('fixtures/raw/otel/openinference/weather_tool_call.otlp.json', 'fixtures/raw/otel/openllmetry/weather_tool_call.otlp.json') and the store is keyed on it, so persisting them would overwrite one with the other; rename them so their first dotted filename segments differ, or tag each with a distinct calibrated_agent_evaluation.trace_id span attribute
  [skipped] fixtures/raw/otel/unknown/unrecognized.otlp.json: could not detect an OTel GenAI convention: no gen_ai.input/output.messages, openinference.span.kind/llm.*_messages, gen_ai.prompt/completion.*, invoke_agent/execute_tool, or legacy gen_ai.* message-event signal found. Declare `semconv=` on the source (`calibrated-agent-evaluation ingest --convention <name>`) to force one, or register a custom convention.

Every file ends in one of three states:

  • [ok] would be stored. Two of these come out probed because the files carry only the outer input and output, and the lines at the top say why.
  • [REFUSED] would lose data if stored. Here two files resolve to the same trace id, so storing both would overwrite one with the other. The reason names both files and two ways to fix it.
  • [skipped] could not be read as a trace at all. The reason lists the signals it looked for.

A dry run always exits 0. A real run exits 1 if any file was refused or skipped, so a script can tell a partial ingest from a complete one.

When files in one batch share the part of the name before the first dot (the three ORD-1001.* files above), each is stored under its OpenTelemetry trace id instead, which is why those lines show hex ids. To keep readable ids, give each file a distinct prefix or tag the spans at instrumentation time (below).

Draw a sample instead of taking everything

A failure rate describes production only when it is measured on traces that stand for production traffic. A directory of exports or a Langfuse project usually doesn't: whoever exported it chose what went in. --sample N --seed S draws a seeded random sample of the population, stores those traces tagged production-representative, and writes a record of the draw:

$ calibrated-agent-evaluation ingest --sample 3 --seed 7 fixtures/raw/strands
calibrated-agent-evaluation ingest: 3 ingested, 0 refused, 0 skipped
  sampled 3 of 6 trace(s) with seed 7, tagged production-representative (record sample-<id>)
  [ok] fixtures/raw/strands/CLM-4473.otlp.json → CLM-4473 (strands, 5 span(s), 4 tool call(s), owned)
  [ok] fixtures/raw/strands/CLM-4475.otlp.json → CLM-4475 (strands, 5 span(s), 4 tool call(s), owned)
  [ok] fixtures/raw/strands/CLM-4477.otlp.json → CLM-4477 (strands, 5 span(s), 4 tool call(s), owned)

The seed is required so anyone can repeat the draw and get the same traces. The record under <root>/records/sample/ keeps the seed, the population size, the ids drawn and the ids stored, and calibrated-agent-evaluation dump carries it with everything else. Asking for more traces than the population holds stops with an error that gives the population size.

For a Langfuse project, --cell KEY=VALUE draws within one taxonomy cell, which is a stratified sample: run it once per cell and each draw gets its own record with its own population size.

calibrated-agent-evaluation ingest --from-langfuse my-project --sample 200 --seed 7
calibrated-agent-evaluation ingest --from-langfuse my-project --sample 60 --seed 7 --cell channel=chat

A trace whose taxonomy cell is unknown cannot belong to a stratum, so it is left out of the population and counted on the record. Over files, --cell is refused: a file's cell is only known once it is normalized.

Refuse outcome-only traces

If you plan to analyze the traces step by step, a trace that carries only its outer input and output is of little use. --require-owned refuses it instead of storing it as probed:

$ calibrated-agent-evaluation ingest --require-owned fixtures/raw/otel/ORD-DEGRADED.otlp.json
trace ORD-DEGRADED degraded to ownership='probed': only a terminal input/output envelope was instrumented (no per-step tool spans and no per-step gen_ai message events)
calibrated-agent-evaluation ingest: 0 ingested, 1 refused, 0 skipped
  [REFUSED] fixtures/raw/otel/ORD-DEGRADED.otlp.json: trace 'ORD-DEGRADED' degraded to ownership='probed' and --require-owned was set; it carries terminal input/output only, with no per-step spans, so step-level attribution and per-agent rates have nothing to read; re-capture with tool/agent instrumentation, or drop --require-owned to accept outcome-only data

When a convention is ambiguous

If a file's spans match more than one OpenTelemetry convention, calibrated-agent-evaluation ingest skips it and says so rather than guess. The one exception: when the GenAI conventions' agent and tool spans are present, they win, because they carry the agent tree the others lack. Otherwise name the one to use:

calibrated-agent-evaluation ingest --convention openinference exports/phoenix/

The choices are gen_ai/latest, gen_ai/legacy, gen_ai/messages, openllmetry and openinference.

From Python

The CLI wraps a two-step library path: normalize() turns one source into a Trace, and a backend stores it.

from pathlib import Path

from calibrated_agent_evaluation.backends import get_backend
from calibrated_agent_evaluation.normalize import OTelSpans, normalize

trace = normalize(OTelSpans(path=Path("exports/session-8f2c.otlp.json")))

with get_backend() as backend:      # the local store at ./.calibrated-agent-evaluation
    backend.put_trace(trace)        # storing the same trace id again replaces it

normalize() returns a complete, validated Trace with every span, or raises NormalizationError. Note one difference from the CLI: the library takes the trace id from the spans (pass trace_id= to set one), while the CLI names a trace after its file.

Each source has its own descriptor:

from pathlib import Path

from calibrated_agent_evaluation.normalize import InspectEvalLog, SessionSpans, StrandsSession, normalize, normalize_all
from calibrated_agent_evaluation.normalize.datasets import JSONLDataset, load_traces

normalize(StrandsSession(path=Path("exports/claims-run.otlp.json")))

# An Inspect log holds one sample per dataset row; normalize_all returns one Trace per sample.
traces = normalize_all(InspectEvalLog(path=Path("logs/2026-07-23_orderbot.eval")))

# Spans you already hold in memory, for example from your own exporter.
normalize(SessionSpans(spans=my_span_dicts))

# A dataset streams one Trace per record.
for trace in load_traces(JSONLDataset(path=Path("captures/week-30.jsonl")), limit=500):
    ...

normalize() on a multi-sample Inspect log refuses rather than read only the first sample. Use normalize_all(), or pass sample_id= to pick one.

calibrated_agent_evaluation.normalize.datasets also has ParquetDataset, HFDataset and DatasetMapping, which names the columns a differently shaped dataset keeps its session id, spans and taxonomy fields in.

Name traces at instrumentation time

If you control the agent's OpenTelemetry setup, stamp three things on the root span so they survive ingest instead of being filled in with defaults:

  • a readable trace id, such as an order or ticket number;
  • the frame, which says why the trace was collected: production-representative for a sample of real traffic, discovery for traces gathered to find bugs. Rates are only ever estimated from the first kind;
  • the taxonomy cell, the combination of input dimensions the run exercised (channel, tier, feature), so failures can be broken down by it.
from calibrated_agent_evaluation.normalize.guidance import tag_span

tag_span(
    span.set_attribute,             # your OTel span's set_attribute
    trace_id="ORD-2026-07-0042",
    frame="production-representative",
    taxonomy_cell={"channel": "chat", "tier": "premium"},
)

This is optional. Without it, ingest uses the file name or OpenTelemetry trace id, and the discovery frame unless you pass --frame production-representative (or draw a sample with --sample N --seed S, which tags the drawn traces production-representative).

My framework is not listed

There are no adapters per framework. LangGraph, CrewAI, AutoGen and the rest reach Calibrated Agent Evaluation as OpenTelemetry spans. If your framework emits OpenTelemetry natively, or has an OpenLLMetry or OpenInference instrumentation you can turn on, export OTLP-JSON and run calibrated-agent-evaluation ingest.

For a dialect none of the built-in conventions reads, register_convention adds one from Python. It is library-only, since the CLI cannot load your module, and you reach it through SessionSpans:

from calibrated_agent_evaluation.normalize import SessionSpans, normalize
from calibrated_agent_evaluation.normalize.conventions.registry import register_convention
from calibrated_agent_evaluation.normalize.tree import AgentNode, Intermediate, ToolInvocation


def my_dialect(session: SessionSpans) -> Intermediate:
    attrs = session.spans[0]["attributes"]
    node = AgentNode(
        agent=attrs["myco.agent"],
        role="specialist",
        parent=None,
        input={"message": attrs["myco.prompt"]},
        output={"answer": attrs["myco.reply"]},
        tool_calls=[ToolInvocation(name=attrs["myco.tool"], args={}, result={})],
    )
    return Intermediate(
        trace_id=session.trace_id or "myco-trace",
        frame=session.frame,
        taxonomy_cell=dict(session.taxonomy_cell),
        trace_input=node.input,
        trace_output=node.output,
        nodes=[node],
    )


register_convention("myco", my_dialect)
trace = normalize(SessionSpans(spans=my_span_dicts, semconv="myco"))

Registering a built-in name, or one already registered, raises instead of replacing it. If your dialect only keeps content under different attribute keys, register_content_placements in calibrated_agent_evaluation.normalize.content_placements adds those keys to what the built-in readers try, with no new adapter.

Next