Skip to content

calibrated-agent-evaluation dataset

turn labeled failures into test scenarios; publish and pull them as AgentCore dataset versions

usage: calibrated-agent-evaluation dataset [-h] [--trace TRACE_ID] [--out FILE] [--force]
                                           [--answers FILE] [--failure-mode NAME] [--name NAME]
                                           [--yes] [--dataset-id ID] [--version N]
                                           [--profile PROFILE] [--region REGION]
                                           [--executor {strands,agentcore}] [--agent AGENT]
                                           [--agent-model AGENT_MODEL] [--cassette PATH]
                                           [--judge FILE] [--simulator {strands,agentcore}]
                                           [--simulator-model MODEL_ID] [--runtime-arn ARN]
                                           [--qualifier NAME]
                                           [--payload-format {generic_message,openai,agentcore_sse,nested_agentcore}]
                                           [--config-bundle-arn ARN]
                                           [--config-bundle-version VERSION]
                                           [--read-timeout SECONDS] [--log-dir DIR]
                                           [--backend {local,langfuse}] [--root PATH] [--explain]
                                           {answers,build,publish,pull,run} [SLICE]

answers writes a starter answer key listing the failure modes on the human FAIL labels; build
turns each trace every rater labeled FAIL into a single-turn scenario, with the expected tool
order and assertions from the answer key; publish shows its plan, and with --yes creates or
versions an AgentCore dataset (needs the agentcore extra); pull reads one published version back
into a slice file.

positional arguments:
  {answers,build,publish,pull,run}
                        answers: write a starter answer key; build: build a scenario slice from
                        the human FAIL labels; publish: publish a slice as an AgentCore dataset
                        version; pull: read a published version back into a slice file; run: hold
                        one simulated conversation per scenario of a simulated slice and store
                        each as a trace
  SLICE                 publish: the slice file to publish; run: the simulated slice to run

options:
  -h, --help            show this help message and exit
  --trace TRACE_ID      operate on one stored trace (default: every stored trace); an unknown id
                        is refused
  --out FILE            answers, build, pull: file to write
  --force               answers, build, pull: replace --out if it exists (build keeps its
                        published version)
  --answers FILE        build: the answer key giving expected tool orders and assertions per
                        failure mode
  --failure-mode NAME   answers, build: only the human FAIL labels with this failure mode
  --name NAME           publish: the AgentCore dataset name (letters, digits and underscores,
                        starting with a letter)
  --yes                 publish: create or version the dataset; run: make the billed calls its
                        plan lists. Without it, both only print their plan
  --dataset-id ID       pull: the dataset id
  --version N           pull: the published version number (the editable draft can't be pulled)
  --profile PROFILE     publish, pull, run: AWS profile (default: AWS_PROFILE)
  --region REGION       publish, pull, run --judge with --executor strands: AWS region (default:
                        the profile's; with --executor agentcore the runtime ARN's region is used)
  --executor {strands,agentcore}
                        run: the agent under test, a local Strands agent (the default) or an
                        AgentCore Runtime agent
  --agent AGENT         run with --executor strands: the agent under test, as 'module:callable' (a
                        Strands topology builder)
  --agent-model AGENT_MODEL
                        run with --executor strands: the agent's model (default: the executor's)
  --cassette PATH       run: record or replay the agent's turns through this cassette, and the
                        simulated user's turns through <stem>.actor.json beside it
  --judge FILE          run: an agentcore judge file with level session; each conversation is
                        judged after every one has run (a live, billed call)
  --simulator {strands,agentcore}
                        who plays the customer in each conversation: strands (the strands-agents-
                        evals actor, the default) or agentcore (AgentCore's actor simulator; needs
                        the agentcore extra)
  --simulator-model MODEL_ID
                        the simulated customer's model id (default: the simulator's own)
  --runtime-arn ARN     with --executor agentcore: the AgentCore Runtime agent to send to, by its
                        runtime ARN; every AWS call goes to the ARN's region
  --qualifier NAME      with --executor agentcore: the runtime endpoint to invoke (default:
                        DEFAULT)
  --payload-format {generic_message,openai,agentcore_sse,nested_agentcore}
                        with --executor agentcore: the request body the agent reads (required: it
                        is never guessed, since each guess would be a billed call)
  --config-bundle-arn ARN
                        with --executor agentcore: a configuration bundle for the agent to run
                        with (needs --config-bundle-version)
  --config-bundle-version VERSION
                        with --executor agentcore: the version of --config-bundle-arn
  --read-timeout SECONDS
                        with --executor agentcore: how long to wait for each read of the agent's
                        response, in seconds, not the whole answer (default: 300)
  --log-dir DIR         with --executor agentcore: where generate and probe keep the run manifest,
                        agentcore_run_record.json and .csv with one record per run (default:
                        <root>/records/agentcore, shared by both)
  --backend {local,langfuse}
                        trace-store backend (default: local, offline)
  --root PATH           local root for the trace store and the stage records (default:
                        $CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, else ./.calibrated-agent-
                        evaluation; quickstart defaults to a temp dir)
  --explain             after the plain summary, print the statistics behind it (bounds, methods,
                        thresholds, record paths); CALIBRATED_AGENT_EVALUATION_EXPLAIN=1 turns it
                        on for every command