Skip to content

calibrated-agent-evaluation generate

synthesize frame-tagged inputs; with --run, route them through your agent

usage: calibrated-agent-evaluation generate [-h] [--trace TRACE_ID] [--run]
                                            [--executor {inspect,strands,agentcore}]
                                            [--agent AGENT] [--model MODEL] [--agent-model MODEL]
                                            [--cassette PATH] [--multi-turn] [--max-turns N]
                                            [--persona FILE] [--simulator {strands,agentcore}]
                                            [--simulator-model MODEL_ID] [--taxonomy FILE]
                                            [--frame FILE] [--out FILE] [--runtime-arn ARN]
                                            [--qualifier NAME]
                                            [--payload-format {generic_message,openai,agentcore_sse,nested_agentcore}]
                                            [--config-bundle-arn ARN]
                                            [--config-bundle-version VERSION]
                                            [--read-timeout SECONDS] [--profile NAME]
                                            [--log-dir DIR] [--yes] [--backend {local,langfuse}]
                                            [--root PATH] [--explain]

Render one natural-language input per frame row (offline template, or an LLM with --model). With
--run the inputs go through your agent and its traces are stored. --multi-turn turns each input
into a conversation: a simulated person writes the user turns, your agent answers them, and the
last turn's trace is the stored conversation.

options:
  -h, --help            show this help message and exit
  --trace TRACE_ID      operate on one stored trace (default: every stored trace); an unknown id
                        is refused
  --run                 route synthesized inputs through a real agent (needs an executor)
  --executor {inspect,strands,agentcore}
                        executor for --run (inspect, strands or agentcore; needs the matching
                        extra)
  --agent AGENT         the agent under test for --run, as 'module:callable' (an Inspect solver,
                        or a Strands topology builder); required with --run since the agent is
                        yours, not the library's (with --executor agentcore the agent is the
                        runtime, so it is refused). Imported with the current directory first on
                        the path, as `python -m` does
  --model MODEL         model id for the LLM renderer (needs [bedrock]); omit for the offline
                        template
  --agent-model MODEL   model the agent under test runs on with --run (required for inspect;
                        strands defaults to its own; refused with agentcore)
  --cassette PATH       record/replay the agent's model calls for --run through this cassette file
                        (with --executor agentcore, the runtime's runs)
  --multi-turn          hold a simulated conversation per input instead of one shot: a simulated
                        person writes each user turn and the agent answers it (needs --run with
                        --executor strands or agentcore, and the synth extra)
  --max-turns N         stop a --multi-turn conversation after this many agent turns (default: 6)
  --persona FILE        JSON persona for --multi-turn (traits, context, actor_goal); the default
                        persona is built from each input's taxonomy cell
  --simulator {strands,agentcore}
                        who plays the customer in each conversation: strands (the strands-agents-
                        evals actor, the default) or agentcore (AgentCore's actor simulator; needs
                        the agentcore extra)
  --simulator-model MODEL_ID
                        the simulated customer's model id (default: the simulator's own)
  --taxonomy FILE       JSON object of dimension -> list of values (default: the stored traces'
                        taxonomy cells)
  --frame FILE          a frame written by `calibrated-agent-evaluation define --out` (instead of
                        building one)
  --out FILE            write the synthesized inputs as JSONL, one per line (without --run)
  --runtime-arn ARN     with --executor agentcore: the AgentCore Runtime agent to send to, by its
                        runtime ARN; every AWS call goes to the ARN's region
  --qualifier NAME      with --executor agentcore: the runtime endpoint to invoke (default:
                        DEFAULT)
  --payload-format {generic_message,openai,agentcore_sse,nested_agentcore}
                        with --executor agentcore: the request body the agent reads (required: it
                        is never guessed, since each guess would be a billed call)
  --config-bundle-arn ARN
                        with --executor agentcore: a configuration bundle for the agent to run
                        with (needs --config-bundle-version)
  --config-bundle-version VERSION
                        with --executor agentcore: the version of --config-bundle-arn
  --read-timeout SECONDS
                        with --executor agentcore: how long to wait for each read of the agent's
                        response, in seconds, not the whole answer (default: 300)
  --profile NAME        with --executor agentcore: the AWS profile to call AWS with (default:
                        AWS_PROFILE)
  --log-dir DIR         with --executor agentcore: where generate and probe keep the run manifest,
                        agentcore_run_record.json and .csv with one record per run (default:
                        <root>/records/agentcore, shared by both)
  --yes                 with --executor agentcore: go ahead with the billed calls the printed plan
                        lists (needed for every run that can call AWS; a cassette replay calls
                        none)
  --backend {local,langfuse}
                        trace-store backend (default: local, offline)
  --root PATH           local root for the trace store and the stage records (default:
                        $CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, else ./.calibrated-agent-
                        evaluation; quickstart defaults to a temp dir)
  --explain             after the plain summary, print the statistics behind it (bounds, methods,
                        thresholds, record paths); CALIBRATED_AGENT_EVALUATION_EXPLAIN=1 turns it
                        on for every command