Capture in process¶
calibrated-agent-evaluation capture runs one instrumented run of your agent and stores the trace
it emitted. Use it when your agent emits OpenTelemetry spans but you have no exporter writing them
to files yet.
It needs the strands extra, which carries the OpenTelemetry SDK and the OTLP encoder the capture
uses:
uv sync --inexact --extra strands
source .venv/bin/activate # puts `calibrated-agent-evaluation` on your PATH
The agent¶
The agent is any Python function that takes no arguments and emits spans through the global
OpenTelemetry tracer. This is a complete one, orderbot.py, with one agent span and one tool span
using the OpenTelemetry GenAI attribute names:
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
# SPDX-License-Identifier: Apache-2.0
import json
from opentelemetry import trace
tracer = trace.get_tracer("orderbot")
def lookup_order(order_id: str) -> dict:
return {"order_id": order_id, "status": "delivered", "days_since_delivery": 41}
def run_once() -> None:
request = {"request": "Refund order A-1001, it arrived broken."}
with tracer.start_as_current_span(
"invoke_agent orderbot",
attributes={
"gen_ai.operation.name": "invoke_agent",
"gen_ai.agent.name": "orderbot",
"gen_ai.input.content": json.dumps(request),
},
) as agent_span:
args = {"order_id": "A-1001"}
with tracer.start_as_current_span(
"execute_tool lookup_order",
attributes={
"gen_ai.operation.name": "execute_tool",
"gen_ai.tool.name": "lookup_order",
"gen_ai.tool.call.arguments": json.dumps(args),
},
) as tool_span:
result = lookup_order(**args)
tool_span.set_attribute("gen_ai.tool.call.result", json.dumps(result))
answer = {"decision": "refund_approved", "amount": 59.0}
agent_span.set_attribute("gen_ai.output.content", json.dumps(answer))
An agent built on a framework that already emits GenAI spans (Strands, or anything under an OpenLLMetry or OpenInference instrumentor) needs no hand-written spans. The function just runs the agent once.
Capture it¶
These four commands were run from the directory holding orderbot.py, with PYTHONPATH=. so the
installed calibrated-agent-evaluation script can import it. An agent inside an installed package
needs no PYTHONPATH.
$ calibrated-agent-evaluation capture --agent orderbot:run_once --service orderbot --dry-run
calibrated-agent-evaluation capture: 1 would ingest, 0 refused, 0 skipped
[ok] <tmp>/calibrated-agent-evaluation-capture.otlp.json → <otel-trace-id> (gen_ai/latest, 1 span(s), 1 tool call(s), owned)
(exit 0)
$ calibrated-agent-evaluation capture --agent orderbot:run_once --service orderbot --trace-id ORD-A1001
calibrated-agent-evaluation capture: 1 ingested, 0 refused, 0 skipped
[ok] <tmp>/ORD-A1001.otlp.json → ORD-A1001 (gen_ai/latest, 1 span(s), 1 tool call(s), owned)
(exit 0)
$ calibrated-agent-evaluation capture --agent orderbot:run_once --service orderbot --trace-id ORD-A1001
calibrated-agent-evaluation capture: the store already holds trace 'ORD-A1001' and a capture cannot be re-run to recover what would be replaced; pass a different --trace-id, or --overwrite to replace it
(exit 1)
$ calibrated-agent-evaluation capture --agent os:getcwd
calibrated-agent-evaluation capture: captured 0 spans from 'os:getcwd': the agent emitted no OpenTelemetry spans (is it instrumented? did it exit before a span closed?), so nothing was stored
(exit 1)
In order, they show:
--dry-runruns the agent and reports what would be stored, without storing it. The id is the capture's OpenTelemetry trace id, which differs on every run.--trace-idstores the trace under a name you choose. The line reports the convention the spans were read with, the agent steps and tool calls recovered, and that the trace is owned (its steps are visible).- A second capture under the same id stops with an error and keeps the stored trace, since a
capture can't be re-run to get the old one back. Pass
--overwriteto replace it on purpose. - A function that emits no spans stops with an error, and nothing is stored.
os:getcwdis a real function that emits nothing.
If the function raises, calibrated-agent-evaluation capture prints the --agent value and the
exception, exits 1 and stores nothing.
Why it has to run in the same process¶
Capture works by installing an OpenTelemetry tracer provider in the process that runs the agent, so
calibrated-agent-evaluation capture imports your module and calls the function itself. An agent
that runs as a separate service, CLI or job cannot emit into that provider. For those, have the
agent's process export OTLP-JSON and ingest the file. Calibrated Agent
Evaluation's SpanCapture does the export in a few lines:
# in the agent's process, with the strands extra installed
from pathlib import Path
from calibrated_agent_evaluation.execute.strands_capture import SpanCapture, write_otlp_json
with SpanCapture("orderbot") as capture:
capture.register_global() # before the agent's first span
run_once() # your agent, emitting spans as usual
write_otlp_json(capture.finished_spans, Path("exports/run-1.otlp.json"))
Any exporter that writes one OTLP/HTTP JSON document (resourceSpans, scopeSpans, spans) per
file works.
From Python¶
capture_and_ingest is the function behind the command:
from orderbot import run_once
from calibrated_agent_evaluation.backends.local import LocalBackend
from calibrated_agent_evaluation.ingest import capture_and_ingest
backend = LocalBackend(root="./store")
report = capture_and_ingest(run_once, service_name="orderbot", backend=backend)
trace = backend.get_trace(report.results[0].trace_id)
call = trace.spans[0].tool_calls[0]
print(call.name, call.args, call.result)
lookup_order {'order_id': 'A-1001'} {'order_id': 'A-1001', 'status': 'delivered', 'days_since_delivery': 41}
The full program is docs/site/examples/integrate/capture_from_python.py.
You can call capture_and_ingest as many times as you like in one process; each call stores only
the spans its agent emitted. A span belongs to the capture it started inside, or to the capture
that holds its parent span, so two captures running on two threads keep their spans apart, and
host telemetry that started outside the capture (a web request, a database call) stays out of the
trace. If your application already installed its own OpenTelemetry SDK tracer provider, capture
attaches to it. While a capture is open, Strands' tracer emits only to the capture, so a Strands
run is not also exported raw by whatever exporter the application's provider carries. A provider
that is not the OpenTelemetry SDK's (some
vendor distributions) is refused with an error that names it, because capture cannot attach to
it; export OTLP-JSON from that process and ingest the file instead.
Next¶
With traces in the store, the next step is labeling. calibrated-agent-evaluation label without
--serve prints the command that opens the review surface on your store:
$ calibrated-agent-evaluation --root ./store label
review surface for the 'local' backend: run `calibrated-agent-evaluation --backend local --root store label --serve --rater RATER_ID` with RATER_ID replaced by your rater id, then open http://127.0.0.1:5001 (needs the UI extra: uv sync --inexact --extra ui)
See Review traces in the UI for the labeling session itself.