Skip to content

Capture in process

calibrated-agent-evaluation capture runs one instrumented run of your agent and stores the trace it emitted. Use it when your agent emits OpenTelemetry spans but you have no exporter writing them to files yet.

It needs the strands extra, which carries the OpenTelemetry SDK and the OTLP encoder the capture uses:

uv sync --inexact --extra strands
source .venv/bin/activate    # puts `calibrated-agent-evaluation` on your PATH

The agent

The agent is any Python function that takes no arguments and emits spans through the global OpenTelemetry tracer. This is a complete one, orderbot.py, with one agent span and one tool span using the OpenTelemetry GenAI attribute names:

# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
# SPDX-License-Identifier: Apache-2.0

import json

from opentelemetry import trace

tracer = trace.get_tracer("orderbot")


def lookup_order(order_id: str) -> dict:
    return {"order_id": order_id, "status": "delivered", "days_since_delivery": 41}


def run_once() -> None:
    request = {"request": "Refund order A-1001, it arrived broken."}
    with tracer.start_as_current_span(
        "invoke_agent orderbot",
        attributes={
            "gen_ai.operation.name": "invoke_agent",
            "gen_ai.agent.name": "orderbot",
            "gen_ai.input.content": json.dumps(request),
        },
    ) as agent_span:
        args = {"order_id": "A-1001"}
        with tracer.start_as_current_span(
            "execute_tool lookup_order",
            attributes={
                "gen_ai.operation.name": "execute_tool",
                "gen_ai.tool.name": "lookup_order",
                "gen_ai.tool.call.arguments": json.dumps(args),
            },
        ) as tool_span:
            result = lookup_order(**args)
            tool_span.set_attribute("gen_ai.tool.call.result", json.dumps(result))
        answer = {"decision": "refund_approved", "amount": 59.0}
        agent_span.set_attribute("gen_ai.output.content", json.dumps(answer))

An agent built on a framework that already emits GenAI spans (Strands, or anything under an OpenLLMetry or OpenInference instrumentor) needs no hand-written spans. The function just runs the agent once.

Capture it

These four commands were run from the directory holding orderbot.py, with PYTHONPATH=. so the installed calibrated-agent-evaluation script can import it. An agent inside an installed package needs no PYTHONPATH.

$ calibrated-agent-evaluation capture --agent orderbot:run_once --service orderbot --dry-run
calibrated-agent-evaluation capture: 1 would ingest, 0 refused, 0 skipped
  [ok] <tmp>/calibrated-agent-evaluation-capture.otlp.json → <otel-trace-id> (gen_ai/latest, 1 span(s), 1 tool call(s), owned)
(exit 0)

$ calibrated-agent-evaluation capture --agent orderbot:run_once --service orderbot --trace-id ORD-A1001
calibrated-agent-evaluation capture: 1 ingested, 0 refused, 0 skipped
  [ok] <tmp>/ORD-A1001.otlp.json → ORD-A1001 (gen_ai/latest, 1 span(s), 1 tool call(s), owned)
(exit 0)

$ calibrated-agent-evaluation capture --agent orderbot:run_once --service orderbot --trace-id ORD-A1001
calibrated-agent-evaluation capture: the store already holds trace 'ORD-A1001' and a capture cannot be re-run to recover what would be replaced; pass a different --trace-id, or --overwrite to replace it
(exit 1)

$ calibrated-agent-evaluation capture --agent os:getcwd
calibrated-agent-evaluation capture: captured 0 spans from 'os:getcwd': the agent emitted no OpenTelemetry spans (is it instrumented? did it exit before a span closed?), so nothing was stored
(exit 1)

In order, they show:

  1. --dry-run runs the agent and reports what would be stored, without storing it. The id is the capture's OpenTelemetry trace id, which differs on every run.
  2. --trace-id stores the trace under a name you choose. The line reports the convention the spans were read with, the agent steps and tool calls recovered, and that the trace is owned (its steps are visible).
  3. A second capture under the same id stops with an error and keeps the stored trace, since a capture can't be re-run to get the old one back. Pass --overwrite to replace it on purpose.
  4. A function that emits no spans stops with an error, and nothing is stored. os:getcwd is a real function that emits nothing.

If the function raises, calibrated-agent-evaluation capture prints the --agent value and the exception, exits 1 and stores nothing.

Why it has to run in the same process

Capture works by installing an OpenTelemetry tracer provider in the process that runs the agent, so calibrated-agent-evaluation capture imports your module and calls the function itself. An agent that runs as a separate service, CLI or job cannot emit into that provider. For those, have the agent's process export OTLP-JSON and ingest the file. Calibrated Agent Evaluation's SpanCapture does the export in a few lines:

# in the agent's process, with the strands extra installed
from pathlib import Path

from calibrated_agent_evaluation.execute.strands_capture import SpanCapture, write_otlp_json

with SpanCapture("orderbot") as capture:
    capture.register_global()      # before the agent's first span
    run_once()                     # your agent, emitting spans as usual
write_otlp_json(capture.finished_spans, Path("exports/run-1.otlp.json"))
calibrated-agent-evaluation ingest exports/run-1.otlp.json

Any exporter that writes one OTLP/HTTP JSON document (resourceSpans, scopeSpans, spans) per file works.

From Python

capture_and_ingest is the function behind the command:

from orderbot import run_once

from calibrated_agent_evaluation.backends.local import LocalBackend
from calibrated_agent_evaluation.ingest import capture_and_ingest

backend = LocalBackend(root="./store")
report = capture_and_ingest(run_once, service_name="orderbot", backend=backend)
trace = backend.get_trace(report.results[0].trace_id)
call = trace.spans[0].tool_calls[0]
print(call.name, call.args, call.result)
lookup_order {'order_id': 'A-1001'} {'order_id': 'A-1001', 'status': 'delivered', 'days_since_delivery': 41}

The full program is docs/site/examples/integrate/capture_from_python.py.

You can call capture_and_ingest as many times as you like in one process; each call stores only the spans its agent emitted. A span belongs to the capture it started inside, or to the capture that holds its parent span, so two captures running on two threads keep their spans apart, and host telemetry that started outside the capture (a web request, a database call) stays out of the trace. If your application already installed its own OpenTelemetry SDK tracer provider, capture attaches to it. While a capture is open, Strands' tracer emits only to the capture, so a Strands run is not also exported raw by whatever exporter the application's provider carries. A provider that is not the OpenTelemetry SDK's (some vendor distributions) is refused with an error that names it, because capture cannot attach to it; export OTLP-JSON from that process and ingest the file instead.

Next

With traces in the store, the next step is labeling. calibrated-agent-evaluation label without --serve prints the command that opens the review surface on your store:

$ calibrated-agent-evaluation --root ./store label
review surface for the 'local' backend: run `calibrated-agent-evaluation --backend local --root store label --serve --rater RATER_ID` with RATER_ID replaced by your rater id, then open http://127.0.0.1:5001 (needs the UI extra: uv sync --inexact --extra ui)

See Review traces in the UI for the labeling session itself.