Skip to content

1. Make test inputs

The OrderBot team has two piles of traces. One is the escalations and complaints support forwarded last month: almost all of them are failures, which is why someone forwarded them. The other is a random day of live traffic. If the two piles get mixed, the escalations make OrderBot look several times worse than it is, and nobody can say by how much.

This step gets traces into the store and marks each one with why it was collected. Labeling and naming failures (steps 2 and 3) want the hunted pile. The failure rate (steps 5 and 7) is measured on the random pile only.

Mark What goes in it Used for
discovery traces picked to find failures: the frame's inputs, escalations, complaints, edge cases labeling, naming failures, checking the judge
production-representative a random draw from real traffic, nothing hand-picked the failure rate and the CI gate

calibrated-agent-evaluation run leaves discovery traces out of the rate and says how many it left out. Both kinds can sit in one store.

Bring the traces you have

calibrated-agent-evaluation ingest --dry-run traces/
$ calibrated-agent-evaluation ingest --dry-run fixtures/raw/otel
trace ORD-DEGRADED degraded to ownership='probed': only a terminal input/output envelope was instrumented (no per-step tool spans and no per-step gen_ai message events)
trace degraded_v2 degraded to ownership='probed': no gen_ai.input/output.messages content and no tool calls recovered from the gen_ai/messages session
calibrated-agent-evaluation ingest: 7 would ingest (2 probed), 2 refused, 1 skipped
  [ok] fixtures/raw/otel/ORD-1001.attrs.otlp.json → 411234653c440f74f9c7bd885663e509 (gen_ai/latest, 1 span(s), 3 tool call(s), owned)
  [ok] fixtures/raw/otel/ORD-1001.build2.otlp.json → 180fed21d2542a2cee4e99fc1f050396 (gen_ai/latest, 1 span(s), 3 tool call(s), owned)
  [ok] fixtures/raw/otel/ORD-1001.events.otlp.json → ad66fcf93d899366e10b923c366a42cb (gen_ai/latest, 1 span(s), 3 tool call(s), owned)
  [ok] fixtures/raw/otel/ORD-1002.events.otlp.json → ORD-1002 (gen_ai/latest, 1 span(s), 3 tool call(s), owned)
  [ok] fixtures/raw/otel/ORD-DEGRADED.otlp.json → ORD-DEGRADED (gen_ai/latest, 0 span(s), 0 tool call(s), probed)
  [ok] fixtures/raw/otel/degraded_v2.otlp.json → degraded_v2 (gen_ai/messages, 0 span(s), 0 tool call(s), probed)
  [ok] fixtures/raw/otel/messages/swebench_tool_call.otlp.json → swebench_tool_call (gen_ai/messages, 1 span(s), 1 tool call(s), owned)
  [REFUSED] fixtures/raw/otel/openinference/weather_tool_call.otlp.json: 2 sources in this batch resolve to trace id 'f026f026f026f026f026f026f026f026' ('fixtures/raw/otel/openinference/weather_tool_call.otlp.json', 'fixtures/raw/otel/openllmetry/weather_tool_call.otlp.json') and the store is keyed on it, so persisting them would overwrite one with the other; rename them so their first dotted filename segments differ, or tag each with a distinct calibrated_agent_evaluation.trace_id span attribute
  [REFUSED] fixtures/raw/otel/openllmetry/weather_tool_call.otlp.json: 2 sources in this batch resolve to trace id 'f026f026f026f026f026f026f026f026' ('fixtures/raw/otel/openinference/weather_tool_call.otlp.json', 'fixtures/raw/otel/openllmetry/weather_tool_call.otlp.json') and the store is keyed on it, so persisting them would overwrite one with the other; rename them so their first dotted filename segments differ, or tag each with a distinct calibrated_agent_evaluation.trace_id span attribute
  [skipped] fixtures/raw/otel/unknown/unrecognized.otlp.json: could not detect an OTel GenAI convention: no gen_ai.input/output.messages, openinference.span.kind/llm.*_messages, gen_ai.prompt/completion.*, invoke_agent/execute_tool, or legacy gen_ai.* message-event signal found. Declare `semconv=` on the source (`calibrated-agent-evaluation ingest --convention <name>`) to force one, or register a custom convention.

--dry-run reports what would be stored and what would be turned away, and writes nothing. Run it on every new source first, then again without --dry-run.

Ingest the traces in traces/ for OrderBot.

The run-error-analysis skill picks the route that matches what you have (trace files, an agent it can run in process, or an endpoint you are authorized to test) and runs the matching command. Ask it for a dry run first if you want one.

The output above is a folder of OpenTelemetry exports. Seven files convert. Two of those carry only the input and the final answer, so their traces can be graded on the outcome but cannot show which step went wrong. Two files would overwrite each other in the store, so neither is written until they get distinct ids, and one file is in a format the reader doesn't recognize. When a file is turned away, fix the export or pass --convention; don't edit the trace to make it fit.

Traces arrive marked discovery unless you pass --frame production-representative. For the random pile, let the toolkit do the drawing:

calibrated-agent-evaluation ingest traces/ --sample 200 --seed 7

That ingests a seeded random sample of 200 traces and marks it production-representative. The same seed over the same files draws the same sample, so the rate can be reproduced. Step 7 runs this on a schedule.

Connect your agent covers Langfuse, in-process capture, routing an agent through the toolkit, and the other formats.

Or generate inputs from the grid

When the team has few real traces for a situation, they turn the frame from step 0 into inputs:

calibrated-agent-evaluation generate --frame frame.json --out inputs.jsonl
$ calibrated-agent-evaluation generate --frame frame.json --out inputs.jsonl
synthesized 9 frame-tagged input(s) from a 9-row discovery frame (frame file frame.json) via the offline template renderer; wrote them to inputs.jsonl (JSONL, one input per line)

Generate inputs for every row of frame.json and run OrderBot on them.

The run-error-analysis skill runs calibrated-agent-evaluation generate, then generate --run against the agent. You give it the agent's module:callable.

Each input carries the grid cell it came from. The offline template writes a placeholder request per row, which is enough to check the plumbing. --model writes realistic customer messages with a model on Bedrock instead (the bedrock extra).

--run sends each input through the agent and stores the resulting traces, marked with the row's cell and discovery:

calibrated-agent-evaluation generate --run --frame frame.json \
  --executor strands --agent corpus.agents.orderbot_strands:build

--multi-turn holds a conversation per row, with a simulated customer writing the replies (Route your agent through).

Mark each labeling round

The team labels in rounds. Round 1 is for discovery (step 2). Round 2 is the set step 4 checks the judge on, and each later version of the judge needs a fresh round of its own. Give each batch of hunted traces a cell value that names its round, for example label_round=1, so later commands can pick one round with --cell label_round=1. The judge kit from quickstart --certify skips discovery: its two rounds are the test sets for its two judge versions.

If you can't see inside the agent

When the agent is someone else's endpoint, calibrated-agent-evaluation probe --endpoint URL --authorized collects its replies. Those traces show only what went in and what came out, so you can measure and gate the outcome but cannot say which step failed. Probe a black-box endpoint covers it.

What you have now

OrderBot's traces in one store, each marked discovery or production-representative, with hunted traces tagged by grid cell and labeling round.

Next: 2. Read and label traces.