Route your agent through Calibrated Agent Evaluation¶
On this path Calibrated Agent Evaluation runs your agent and records each run as a trace. You hand it a function that builds your agent, or name an agent you have deployed to Amazon Bedrock AgentCore Runtime. Use it when you want the library to drive the agent over inputs it generates, rather than evaluate runs you captured elsewhere. The traces are your agent's real behavior: the library supplies the inputs and the recording, never the outputs.
Three kinds of agent are supported:
| Your agent is | Executor | You provide | Extra |
|---|---|---|---|
A Strands Agent, Graph or Swarm |
StrandsExecutor |
build(model) returning the topology |
strands |
| An Inspect AI task | InspectExecutor |
the task, with your agent as its solver | inspect |
| Deployed to AgentCore Runtime | AgentCoreRuntimeExecutor |
the runtime ARN and the request body its agent reads | agentcore |
If your agent is on none of these, use Capture in process or export its spans and ingest the files.
Replay a recorded run first¶
Executors record every model call to a cassette file and can replay it later, so a run needs no model and no credentials. That is also how you keep CI offline. This replays the repository's ClaimsCrew, a Strands graph with a supervisor and four specialists, from its committed cassette:
scenario = load_scenario("CLM-4471")
claim = scenario.trace_input
task = StrandsTask(
build=lambda model: build_claims_graph(model, scenario),
input=(
f"Adjudicate claim {claim['claim_id']} for ${claim['amount_requested_usd']}. "
f"Narrative: {claim['narrative']}"
),
mode="legacy",
trace_id="CLM-4471",
taxonomy_cell=dict(scenario.taxonomy_cell),
)
with (
StrandsExecutor(
cassette_path="fixtures/raw/cassettes/claimscrew_strands.cassette.json",
cassette_mode=Mode.REPLAY,
) as executor,
contextlib.redirect_stdout(io.StringIO()),
):
# Strands agents stream their replies to stdout by default; keep the page output to the trace.
(trace,) = executor.run(task)
print(f"trace {trace.trace_id}: ownership={trace.ownership}, {len(trace.spans)} spans")
print(f"taxonomy cell: {trace.taxonomy_cell}")
# The specialists run in parallel, so their order varies between runs; list them by name.
for span in sorted(trace.spans, key=lambda s: (s.parent is not None, s.agent)):
tools = ", ".join(call.name for call in span.tool_calls) or "no tools"
parent = span.parent.split("/", 1)[-1] if span.parent else "root"
print(f" {span.agent:<15} parent={parent:<12} {tools}")
trace CLM-4471: ownership=owned, 5 spans
taxonomy cell: {'feature': 'claim_adjudication', 'scenario': 'S2_vacancy_exclusion', 'persona': 'P2_homeowner_absent'}
supervisor parent=root no tools
fraud-scorer parent=supervisor score_fraud_risk
intake parent=supervisor extract_claim_fields
payout parent=supervisor no tools
policy-checker parent=supervisor lookup_policy_exclusions, lookup_policy_exclusions
The full program is docs/site/examples/integrate/route_strands_replay.py. The Inspect equivalent
replays the OrderBot task:
with InspectExecutor(
model="cassette/orderbot",
cassette_path="fixtures/raw/cassettes/inspect_ORD-1001.cassette.json",
cassette_mode=Mode.REPLAY,
) as executor:
(trace,) = executor.run(orderbot_fail)
print(f"trace {trace.trace_id}: ownership={trace.ownership}, {len(trace.spans)} span(s)")
print(f"input: {trace.input}")
for span in trace.spans:
for call in span.tool_calls:
print(f" {span.agent} called {call.name}({call.args})")
print(f"output: {trace.output}")
trace ORD-1001: ownership=owned, 1 span(s)
input: {'message': "I want a refund for order A-7781, the blender just isn't for me."}
orderbot called lookup_order({'order_id': 'A-7781'})
orderbot called check_refund_policy({'order_id': 'A-7781', 'today': '2025-01-30'})
orderbot called issue_refund({'amount_usd': 89.99, 'order_id': 'A-7781'})
output: {'answer': 'I processed a full refund of $89.99 for order A-7781 (Countertop Blender). The refund was within the return window and was successfully issued with Refund ID RF-30022.'}
cassette_mode is Mode.RECORD, Mode.REPLAY or Mode.AUTO. AUTO, the default, replays a call
the cassette holds and records one it does not. The CALIBRATED_AGENT_EVALUATION_CASSETTE_MODE environment variable
sets the default, so CALIBRATED_AGENT_EVALUATION_CASSETTE_MODE=replay in CI makes any unrecorded call fail instead of
reaching a model.
Your Strands agent¶
Write a function that takes the model and returns your topology. The executor passes in the model so it can wrap it for recording and replay; you only assemble the agents. This is a two-agent graph:
# needs the strands extra; the first run calls Bedrock and records the cassette
from calibrated_agent_evaluation.execute import Mode, StrandsExecutor, StrandsTask
def build_claims_crew(model):
from strands import Agent
from strands.multiagent import GraphBuilder
intake = Agent(model=model, name="intake", system_prompt="Extract the claim fields.")
checker = Agent(model=model, name="checker", system_prompt="Apply the policy exclusions.")
builder = GraphBuilder()
builder.add_node(intake, "intake")
builder.add_node(checker, "checker")
builder.add_edge("intake", "checker")
builder.set_entry_point("intake")
return builder.build()
with StrandsExecutor(
model_id="global.anthropic.claude-sonnet-4-6", # the default
region="us-west-2", # the default
cassette_path="cassettes/claims.json",
cassette_mode=Mode.RECORD,
timeout_seconds=60.0,
) as executor:
traces = executor.run(
StrandsTask(
build=build_claims_crew,
input="Claim 4471: water damage, policyholder away in Florida last month.",
taxonomy_cell={"line": "property", "peril": "water"},
)
)
A live run goes through Bedrock, so set your AWS credentials the usual way (for example
AWS_PROFILE). Commit the cassette and later runs replay it. taxonomy_cell is copied onto the
trace, so failures can later be broken down by line and peril.
The default global. model ID is a cross-Region inference profile, so Bedrock may process the
agent's prompts in other Regions. If you have data-residency requirements, pass an in-Region model
ID or a geographic profile; see
Bedrock model IDs and Regions.
The executor sends temperature=0.0 by default. The newest Anthropic models reject an explicit
temperature, so pass temperature=None to send none.
Your Inspect task¶
Your agent is the task's solver. InspectExecutor takes an Inspect model string:
# needs the inspect extra
from inspect_ai import Task
from inspect_ai.dataset import Sample
from calibrated_agent_evaluation.execute import InspectExecutor
task = Task(dataset=[Sample(input="I want a refund for order 1001.")], solver=my_solver())
with InspectExecutor(model="bedrock/global.anthropic.claude-sonnet-4-6") as executor:
traces = executor.run(task)
For recording and replay, use a cassette/<name> model with cassette_path, as in the replay
example above. cassette_inner_model names the real model a recording calls.
Generate the inputs too: calibrated-agent-evaluation generate --run¶
calibrated-agent-evaluation generate builds a discovery frame from your input dimensions, a
covering array in which every pair of dimension values appears, writes one input per row, and with
--run sends each through your agent and stores the traces:
uv sync --inexact --extra sampling --extra stats --extra strands
calibrated-agent-evaluation generate --run --taxonomy taxonomy.json \
--executor strands --agent mypkg.agents:build_claims_crew
uv sync --inexact --extra sampling --extra stats --extra inspect
calibrated-agent-evaluation generate --run --taxonomy taxonomy.json \
--executor inspect --agent mypkg.agents:my_solver \
--agent-model bedrock/global.anthropic.claude-sonnet-4-6
--taxonomy FILEis a JSON object of dimension to values, such as{"line": ["property", "auto"], "peril": ["water", "fire"]}. Without it the dimensions come from the stored traces' taxonomy cells, and a store with none is refused.--frame FILEtakes a frame written bycalibrated-agent-evaluation define --outinstead.--agent module:callableis your agent: a topology builder for Strands, a solver for Inspect (the generated inputs are wrapped into the task). It is required with--run, except with--executor agentcore, where the agent is the deployed runtime (below).--agent-modelis required for Inspect: the model the agent runs on, eitherbedrock/<model-id>orcassette/<name>together with--cassette. Strands uses its own default model.--cassette PATHrecords and replays the agent's model calls.--model <id>writes the inputs with an LLM (needs thebedrockextra). Without it, inputs come from an offline template.
The traces generate --run stores carry the discovery frame: they were made to find failures,
so they never feed a production rate estimate.
From Python, the same pieces are calibrated_agent_evaluation.synth.generate_and_run with an
executor and a task builder from calibrated_agent_evaluation.execute:
StrandsTaskBuilder(build_topology=...) or InspectTaskBuilder(solver=...), or
AgentCoreTaskBuilder() from calibrated_agent_evaluation.agentcore. It refuses to run without an
executor, since the outputs must come from your agent.
Your agent on AgentCore Runtime¶
--executor agentcore sends each generated input to an agent deployed to Amazon Bedrock
AgentCore Runtime and stores one trace per input. This replays the repository's recorded run of a
synthetic runtime, so it needs no AWS account:
$ CALIBRATED_AGENT_EVALUATION_CASSETTE_MODE=replay calibrated-agent-evaluation generate \
--run \
--executor agentcore \
--runtime-arn arn:aws:bedrock-agentcore:us-west-2:111122223333:runtime/calibrated_agent_evaluation_fixture-AbCdEf1234 \
--payload-format agentcore_sse \
--cassette fixtures/raw/cassettes/agentcore_runtime_CLM-4471.cassette.json \
--frame fixtures/raw/frames/agentcore_CLM-4471.frame.json
routed 1 synthesized input(s) (frame file fixtures/raw/frames/agentcore_CLM-4471.frame.json) through the agentcore executor (runtime arn:aws:bedrock-agentcore:us-west-2:111122223333:runtime/calibrated_agent_evaluation_fixture-AbCdEf1234): 1 discovery-frame, taxonomy-attributable trace(s) persisted; the run manifest, one record per run, is $CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT/records/agentcore/agentcore_run_record.json
The frame file holds one combination of input dimensions, that of the ClaimsCrew claim CLM-4471, and
the cassette holds the recorded answer to the input generated for it. Each run, replayed or live,
adds one record to the run manifest: the session ID, the runtime version, the qualifier, the
configuration bundle, the timing, the attempts and how the session was stopped, keyed by the trace
ID. The manifest is a pair of files, agentcore_run_record.json (the one the library reads back)
and agentcore_run_record.csv, with one record per run. Each file is rewritten atomically while it
holds a lock on their directory, so two runs writing at once both keep their record. The pair goes
to <root>/records/agentcore/ unless you pass --log-dir. calibrated-agent-evaluation dump and
calibrated-agent-evaluation load do not carry that directory, so copy it yourself to keep the
provenance.
In a live run, the executor makes a new runtime session ID and W3C trace context for each input, invokes the runtime with them, reads the final answer from the response, reads the run's span tree, and stops the session. Two options are required:
--runtime-arn ARNis the runtime, by its own ARN (not an endpoint's). Every AWS call goes to the ARN's region.--payload-format NAMEis the request body the agent reads:generic_message({"message": ...}),openai({"messages": [...]}),agentcore_sse({"prompt": ..., "session_id": ...}) ornested_agentcore. It is never guessed, since each guess would be a billed call.
The others are optional:
--qualifier NAMEis the endpoint to invoke (defaultDEFAULT).--config-bundle-arn ARNand--config-bundle-version VERSION, given together, name a configuration bundle for the agent to run with. They are sent in thebaggageheader, named in the plan, and kept in the run manifest and the cassette. An empty value for either is refused, since the agent would otherwise run without the bundle.--read-timeout SECONDSis how long an invocation waits for each read of the response, not for the whole answer (default 300). It must be a positive, finite number of seconds.--profile NAMEis the AWS profile (defaultAWS_PROFILE).--cassette PATHrecords runs into this file and replays them from it.--log-dir DIRis where the run manifest goes instead.--yesgoes ahead with the billed calls.
These flags are refused with another executor or without --run, and --agent and
--agent-model are refused with this one. The frame must hold discovery rows only.
A run that can call AWS, in any cassette mode but replay, first checks
CALIBRATED_AGENT_EVALUATION_AGENTCORE_MAX_BILLED_CALLS and that a span reader is installed, then that the trace store
can take the traces (with --backend langfuse, that the host answers and accepts the keys), since
they are stored only once the last input has run. It then prints what it will call and stops unless
you pass --yes:
$ CALIBRATED_AGENT_EVALUATION_CASSETTE_MODE=auto calibrated-agent-evaluation generate \
--run \
--executor agentcore \
--runtime-arn arn:aws:bedrock-agentcore:us-west-2:111122223333:runtime/calibrated_agent_evaluation_fixture-AbCdEf1234 \
--payload-format agentcore_sse \
--cassette fixtures/raw/cassettes/agentcore_runtime_CLM-4471.cassette.json \
--frame fixtures/raw/frames/agentcore_CLM-4471.frame.json
calibrated-agent-evaluation generate: this run can call AWS. It sends up to 1 input to the AgentCore Runtime agent arn:aws:bedrock-agentcore:us-west-2:111122223333:runtime/calibrated_agent_evaluation_fixture-AbCdEf1234 (qualifier DEFAULT), each a billed invocation (retried when the session is busy or the runtime throttles), and the span reader reads each run's spans. It stops at the first input that fails, and the traces of the inputs before it are then not stored: they stay in the cassette fixtures/raw/cassettes/agentcore_runtime_CLM-4471.cassette.json.
calibrated-agent-evaluation generate: nothing was sent: this run can make the billed AWS calls listed above, so it needs --yes. Pass --yes to go ahead; offline runs need a cassette in replay mode: set CALIBRATED_AGENT_EVALUATION_CASSETTE_MODE=replay and pass --cassette PATH
(exit 1)
Before the first invocation, read-only calls check the runtime. One that only accepts OAuth (it
has a custom JWT authorizer) or that does not serve HTTP (MCP, A2A or AG-UI) is refused before
anything billed is sent; the executor signs its calls with SigV4. A busy session or throttling is retried
in the same session, up to three attempts. Any other failure, an error event in the agent's
response included, stops the command at that input, and the traces of the inputs before it are
not stored: with --cassette they stay in the cassette, without it only their records in the
run manifest are kept. CALIBRATED_AGENT_EVALUATION_AGENTCORE_MAX_BILLED_CALLS=N caps the billed calls a command
makes, retries included. It does not count CloudWatch Logs Insights queries (logs:StartQuery),
which a span reader may make and which Logs Insights bills by the data each one scans.
Live runs read each run's span tree from AgentCore Observability with the per-run span reader,
calibrated_agent_evaluation.agentcore.AgentCoreSpanReader, which calibrated-agent-evaluation
generate builds from the same clients as the executor. It finds the run by its trace ID, waits for
the telemetry to settle, and returns the whole tree or refuses the run; a run whose trace context
was not sampled records no spans, so it is refused, naming that cause. The window ingest,
calibrated-agent-evaluation ingest --source agentcore, is a separate reader for a time window.
From Python, build an AgentCoreRuntimeExecutor from a RuntimeTarget and run
AgentCoreRuntimeTasks, or pass it to generate_and_run with AgentCoreTaskBuilder(). Its
span_reader takes any object that implements RunSpanReader, its run_records holds the
RunRecord of every run it finished, and its manifest_errors says why any of those records could
not be added to the run manifest; the calibrated_agent_evaluation.agentcore
reference describes all of them. The
executor builds no AWS client until its first live call, so a replay builds none and needs no
credentials.
Hold a conversation instead of one shot: --multi-turn¶
Many failures only appear on the third turn. With --multi-turn, each generated input opens a
conversation: a simulated person writes the user turns, your agent answers them, and the
conversation is stored as one trace.
uv sync --inexact --extra sampling --extra stats --extra strands --extra synth
calibrated-agent-evaluation generate --run --taxonomy taxonomy.json --executor strands --multi-turn --max-turns 4 \
--agent mypkg.agents:build_support_agent
Every agent turn is a fresh run of your agent over the conversation so far, which is what a
multi-turn agent gets in production. The conversation is stored as one trace that keeps every
turn's spans, in turn order and prefixed t1:, t2: and so on. Its input is the whole transcript,
with the goal and why the conversation stopped (goal_completed, actor_stopped or max_turns),
so a labeler or a judge reads the conversation rather than its opening line.
--max-turns Nstops the conversation after N agent turns (default 6). The simulated person can end it earlier, when it decides its goal is met.--persona FILEis a JSON object withtraits,contextandactor_goal. Without it, each input gets a persona built from its own taxonomy cell.--simulator strands|agentcorepicks who plays the customer: thestrands-agents-evalsactor (the default), or AgentCore's actor simulator (theagentcoreextra).--simulator-modelsets its model.--multi-turnworks with--executor strandsand--executor agentcore. On an AgentCore Runtime agent, every turn of a conversation is sent on one runtime session, so the deployed agent keeps its own memory of the earlier turns, and the session is stopped once the conversation ends. The Inspect route evaluates one input per sample, so it has no turns.
{
"traits": {"tone": "impatient", "expertise": "none"},
"context": "A customer on their third call about the same order.",
"actor_goal": "Get the refund issued today, or a clear reason why not."
}
The simulated person is strands_evals's ActorSimulator (the synth extra), and it writes only
the user turns. The library never writes an agent turn: each one comes back from your agent through the
executor, and a turn that produces no text stops the run with an error.
Next¶
- Read and label traces takes stored traces through labeling, judging and the gate.
- The
calibrated_agent_evaluation.executereference lists every executor parameter, and thecalibrated_agent_evaluation.agentcorereference the AgentCore Runtime executor's.