1. Make test inputs¶
The OrderBot team has two piles of traces. One is the escalations and complaints support forwarded last month: almost all of them are failures, which is why someone forwarded them. The other is a random day of live traffic. If the two piles get mixed, the escalations make OrderBot look several times worse than it is, and nobody can say by how much.
This step gets traces into the store and marks each one with why it was collected. Labeling and naming failures (steps 2 and 3) want the hunted pile. The failure rate (steps 5 and 7) is measured on the random pile only.
| Mark | What goes in it | Used for |
|---|---|---|
discovery |
traces picked to find failures: the frame's inputs, escalations, complaints, edge cases | labeling, naming failures, checking the judge |
production-representative |
a random draw from real traffic, nothing hand-picked | the failure rate and the CI gate |
calibrated-agent-evaluation run leaves discovery traces out of the rate and says how many it
left out. Both kinds can sit in one store.
Bring the traces you have¶
$ calibrated-agent-evaluation ingest --dry-run fixtures/raw/otel
trace ORD-DEGRADED degraded to ownership='probed': only a terminal input/output envelope was instrumented (no per-step tool spans and no per-step gen_ai message events)
trace degraded_v2 degraded to ownership='probed': no gen_ai.input/output.messages content and no tool calls recovered from the gen_ai/messages session
calibrated-agent-evaluation ingest: 7 would ingest (2 probed), 2 refused, 1 skipped
[ok] fixtures/raw/otel/ORD-1001.attrs.otlp.json → 411234653c440f74f9c7bd885663e509 (gen_ai/latest, 1 span(s), 3 tool call(s), owned)
[ok] fixtures/raw/otel/ORD-1001.build2.otlp.json → 180fed21d2542a2cee4e99fc1f050396 (gen_ai/latest, 1 span(s), 3 tool call(s), owned)
[ok] fixtures/raw/otel/ORD-1001.events.otlp.json → ad66fcf93d899366e10b923c366a42cb (gen_ai/latest, 1 span(s), 3 tool call(s), owned)
[ok] fixtures/raw/otel/ORD-1002.events.otlp.json → ORD-1002 (gen_ai/latest, 1 span(s), 3 tool call(s), owned)
[ok] fixtures/raw/otel/ORD-DEGRADED.otlp.json → ORD-DEGRADED (gen_ai/latest, 0 span(s), 0 tool call(s), probed)
[ok] fixtures/raw/otel/degraded_v2.otlp.json → degraded_v2 (gen_ai/messages, 0 span(s), 0 tool call(s), probed)
[ok] fixtures/raw/otel/messages/swebench_tool_call.otlp.json → swebench_tool_call (gen_ai/messages, 1 span(s), 1 tool call(s), owned)
[REFUSED] fixtures/raw/otel/openinference/weather_tool_call.otlp.json: 2 sources in this batch resolve to trace id 'f026f026f026f026f026f026f026f026' ('fixtures/raw/otel/openinference/weather_tool_call.otlp.json', 'fixtures/raw/otel/openllmetry/weather_tool_call.otlp.json') and the store is keyed on it, so persisting them would overwrite one with the other; rename them so their first dotted filename segments differ, or tag each with a distinct calibrated_agent_evaluation.trace_id span attribute
[REFUSED] fixtures/raw/otel/openllmetry/weather_tool_call.otlp.json: 2 sources in this batch resolve to trace id 'f026f026f026f026f026f026f026f026' ('fixtures/raw/otel/openinference/weather_tool_call.otlp.json', 'fixtures/raw/otel/openllmetry/weather_tool_call.otlp.json') and the store is keyed on it, so persisting them would overwrite one with the other; rename them so their first dotted filename segments differ, or tag each with a distinct calibrated_agent_evaluation.trace_id span attribute
[skipped] fixtures/raw/otel/unknown/unrecognized.otlp.json: could not detect an OTel GenAI convention: no gen_ai.input/output.messages, openinference.span.kind/llm.*_messages, gen_ai.prompt/completion.*, invoke_agent/execute_tool, or legacy gen_ai.* message-event signal found. Declare `semconv=` on the source (`calibrated-agent-evaluation ingest --convention <name>`) to force one, or register a custom convention.
--dry-run reports what would be stored and what would be turned away, and writes nothing.
Run it on every new source first, then again without --dry-run.
Ingest the traces in traces/ for OrderBot.
The run-error-analysis skill picks the route that matches what you have (trace files, an
agent it can run in process, or an endpoint you are authorized to test) and runs the matching
command. Ask it for a dry run first if you want one.
The output above is a folder of OpenTelemetry exports. Seven files convert. Two of those carry only
the input and the final answer, so their traces can be graded on the outcome but cannot show which
step went wrong. Two files would overwrite each other in the store, so neither is written until
they get distinct ids, and one file is in a format the reader doesn't recognize. When a file is
turned away, fix the export or pass --convention; don't edit the trace to make it fit.
Traces arrive marked discovery unless you pass --frame production-representative. For the
random pile, let the toolkit do the drawing:
That ingests a seeded random sample of 200 traces and marks it production-representative. The
same seed over the same files draws the same sample, so the rate can be reproduced.
Step 7 runs this on a schedule.
Connect your agent covers Langfuse, in-process capture, routing an agent through the toolkit, and the other formats.
Or generate inputs from the grid¶
When the team has few real traces for a situation, they turn the frame from step 0 into inputs:
Generate inputs for every row of frame.json and run OrderBot on them.
The run-error-analysis skill runs calibrated-agent-evaluation generate, then
generate --run against the agent. You give it the agent's module:callable.
Each input carries the grid cell it came from. The offline template writes a placeholder request
per row, which is enough to check the plumbing. --model writes realistic customer messages with a
model on Bedrock instead (the bedrock extra).
--run sends each input through the agent and stores the resulting traces, marked with the row's
cell and discovery:
calibrated-agent-evaluation generate --run --frame frame.json \
--executor strands --agent corpus.agents.orderbot_strands:build
--multi-turn holds a conversation per row, with a simulated customer writing the replies
(Route your agent through).
Mark each labeling round¶
The team labels in rounds. Round 1 is for discovery (step 2). Round 2 is the set
step 4 checks the judge on, and each later version of the judge needs a fresh round of
its own. Give each batch of hunted traces a cell value that names its round, for example
label_round=1, so later commands can pick one round with --cell label_round=1. The judge kit
from quickstart --certify skips discovery: its two rounds are the test sets for its two judge
versions.
If you can't see inside the agent¶
When the agent is someone else's endpoint, calibrated-agent-evaluation probe --endpoint URL
--authorized collects its replies. Those traces show only what went in and what came out, so you
can measure and gate the outcome but cannot say which step failed. Probe a black-box
endpoint covers it.
What you have now¶
OrderBot's traces in one store, each marked discovery or production-representative, with hunted
traces tagged by grid cell and labeling round.
Next: 2. Read and label traces.