calibrated-agent-evaluation generate¶
synthesize frame-tagged inputs; with --run, route them through your agent
usage: calibrated-agent-evaluation generate [-h] [--trace TRACE_ID] [--run]
[--executor {inspect,strands,agentcore}]
[--agent AGENT] [--model MODEL] [--agent-model MODEL]
[--cassette PATH] [--multi-turn] [--max-turns N]
[--persona FILE] [--simulator {strands,agentcore}]
[--simulator-model MODEL_ID] [--taxonomy FILE]
[--frame FILE] [--out FILE] [--runtime-arn ARN]
[--qualifier NAME]
[--payload-format {generic_message,openai,agentcore_sse,nested_agentcore}]
[--config-bundle-arn ARN]
[--config-bundle-version VERSION]
[--read-timeout SECONDS] [--profile NAME]
[--log-dir DIR] [--yes] [--backend {local,langfuse}]
[--root PATH] [--explain]
Render one natural-language input per frame row (offline template, or an LLM with --model). With
--run the inputs go through your agent and its traces are stored. --multi-turn turns each input
into a conversation: a simulated person writes the user turns, your agent answers them, and the
last turn's trace is the stored conversation.
options:
-h, --help show this help message and exit
--trace TRACE_ID operate on one stored trace (default: every stored trace); an unknown id
is refused
--run route synthesized inputs through a real agent (needs an executor)
--executor {inspect,strands,agentcore}
executor for --run (inspect, strands or agentcore; needs the matching
extra)
--agent AGENT the agent under test for --run, as 'module:callable' (an Inspect solver,
or a Strands topology builder); required with --run since the agent is
yours, not the library's (with --executor agentcore the agent is the
runtime, so it is refused). Imported with the current directory first on
the path, as `python -m` does
--model MODEL model id for the LLM renderer (needs [bedrock]); omit for the offline
template
--agent-model MODEL model the agent under test runs on with --run (required for inspect;
strands defaults to its own; refused with agentcore)
--cassette PATH record/replay the agent's model calls for --run through this cassette file
(with --executor agentcore, the runtime's runs)
--multi-turn hold a simulated conversation per input instead of one shot: a simulated
person writes each user turn and the agent answers it (needs --run with
--executor strands or agentcore, and the synth extra)
--max-turns N stop a --multi-turn conversation after this many agent turns (default: 6)
--persona FILE JSON persona for --multi-turn (traits, context, actor_goal); the default
persona is built from each input's taxonomy cell
--simulator {strands,agentcore}
who plays the customer in each conversation: strands (the strands-agents-
evals actor, the default) or agentcore (AgentCore's actor simulator; needs
the agentcore extra)
--simulator-model MODEL_ID
the simulated customer's model id (default: the simulator's own)
--taxonomy FILE JSON object of dimension -> list of values (default: the stored traces'
taxonomy cells)
--frame FILE a frame written by `calibrated-agent-evaluation define --out` (instead of
building one)
--out FILE write the synthesized inputs as JSONL, one per line (without --run)
--runtime-arn ARN with --executor agentcore: the AgentCore Runtime agent to send to, by its
runtime ARN; every AWS call goes to the ARN's region
--qualifier NAME with --executor agentcore: the runtime endpoint to invoke (default:
DEFAULT)
--payload-format {generic_message,openai,agentcore_sse,nested_agentcore}
with --executor agentcore: the request body the agent reads (required: it
is never guessed, since each guess would be a billed call)
--config-bundle-arn ARN
with --executor agentcore: a configuration bundle for the agent to run
with (needs --config-bundle-version)
--config-bundle-version VERSION
with --executor agentcore: the version of --config-bundle-arn
--read-timeout SECONDS
with --executor agentcore: how long to wait for each read of the agent's
response, in seconds, not the whole answer (default: 300)
--profile NAME with --executor agentcore: the AWS profile to call AWS with (default:
AWS_PROFILE)
--log-dir DIR with --executor agentcore: where generate and probe keep the run manifest,
agentcore_run_record.json and .csv with one record per run (default:
<root>/records/agentcore, shared by both)
--yes with --executor agentcore: go ahead with the billed calls the printed plan
lists (needed for every run that can call AWS; a cassette replay calls
none)
--backend {local,langfuse}
trace-store backend (default: local, offline)
--root PATH local root for the trace store and the stage records (default:
$CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, else ./.calibrated-agent-
evaluation; quickstart defaults to a temp dir)
--explain after the plain summary, print the statistics behind it (bounds, methods,
thresholds, record paths); CALIBRATED_AGENT_EVALUATION_EXPLAIN=1 turns it
on for every command