calibrated-agent-evaluation dataset¶
turn labeled failures into test scenarios; publish and pull them as AgentCore dataset versions
usage: calibrated-agent-evaluation dataset [-h] [--trace TRACE_ID] [--out FILE] [--force]
[--answers FILE] [--failure-mode NAME] [--name NAME]
[--yes] [--dataset-id ID] [--version N]
[--profile PROFILE] [--region REGION]
[--executor {strands,agentcore}] [--agent AGENT]
[--agent-model AGENT_MODEL] [--cassette PATH]
[--judge FILE] [--simulator {strands,agentcore}]
[--simulator-model MODEL_ID] [--runtime-arn ARN]
[--qualifier NAME]
[--payload-format {generic_message,openai,agentcore_sse,nested_agentcore}]
[--config-bundle-arn ARN]
[--config-bundle-version VERSION]
[--read-timeout SECONDS] [--log-dir DIR]
[--backend {local,langfuse}] [--root PATH] [--explain]
{answers,build,publish,pull,run} [SLICE]
answers writes a starter answer key listing the failure modes on the human FAIL labels; build
turns each trace every rater labeled FAIL into a single-turn scenario, with the expected tool
order and assertions from the answer key; publish shows its plan, and with --yes creates or
versions an AgentCore dataset (needs the agentcore extra); pull reads one published version back
into a slice file.
positional arguments:
{answers,build,publish,pull,run}
answers: write a starter answer key; build: build a scenario slice from
the human FAIL labels; publish: publish a slice as an AgentCore dataset
version; pull: read a published version back into a slice file; run: hold
one simulated conversation per scenario of a simulated slice and store
each as a trace
SLICE publish: the slice file to publish; run: the simulated slice to run
options:
-h, --help show this help message and exit
--trace TRACE_ID operate on one stored trace (default: every stored trace); an unknown id
is refused
--out FILE answers, build, pull: file to write
--force answers, build, pull: replace --out if it exists (build keeps its
published version)
--answers FILE build: the answer key giving expected tool orders and assertions per
failure mode
--failure-mode NAME answers, build: only the human FAIL labels with this failure mode
--name NAME publish: the AgentCore dataset name (letters, digits and underscores,
starting with a letter)
--yes publish: create or version the dataset; run: make the billed calls its
plan lists. Without it, both only print their plan
--dataset-id ID pull: the dataset id
--version N pull: the published version number (the editable draft can't be pulled)
--profile PROFILE publish, pull, run: AWS profile (default: AWS_PROFILE)
--region REGION publish, pull, run --judge with --executor strands: AWS region (default:
the profile's; with --executor agentcore the runtime ARN's region is used)
--executor {strands,agentcore}
run: the agent under test, a local Strands agent (the default) or an
AgentCore Runtime agent
--agent AGENT run with --executor strands: the agent under test, as 'module:callable' (a
Strands topology builder)
--agent-model AGENT_MODEL
run with --executor strands: the agent's model (default: the executor's)
--cassette PATH run: record or replay the agent's turns through this cassette, and the
simulated user's turns through <stem>.actor.json beside it
--judge FILE run: an agentcore judge file with level session; each conversation is
judged after every one has run (a live, billed call)
--simulator {strands,agentcore}
who plays the customer in each conversation: strands (the strands-agents-
evals actor, the default) or agentcore (AgentCore's actor simulator; needs
the agentcore extra)
--simulator-model MODEL_ID
the simulated customer's model id (default: the simulator's own)
--runtime-arn ARN with --executor agentcore: the AgentCore Runtime agent to send to, by its
runtime ARN; every AWS call goes to the ARN's region
--qualifier NAME with --executor agentcore: the runtime endpoint to invoke (default:
DEFAULT)
--payload-format {generic_message,openai,agentcore_sse,nested_agentcore}
with --executor agentcore: the request body the agent reads (required: it
is never guessed, since each guess would be a billed call)
--config-bundle-arn ARN
with --executor agentcore: a configuration bundle for the agent to run
with (needs --config-bundle-version)
--config-bundle-version VERSION
with --executor agentcore: the version of --config-bundle-arn
--read-timeout SECONDS
with --executor agentcore: how long to wait for each read of the agent's
response, in seconds, not the whole answer (default: 300)
--log-dir DIR with --executor agentcore: where generate and probe keep the run manifest,
agentcore_run_record.json and .csv with one record per run (default:
<root>/records/agentcore, shared by both)
--backend {local,langfuse}
trace-store backend (default: local, offline)
--root PATH local root for the trace store and the stage records (default:
$CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, else ./.calibrated-agent-
evaluation; quickstart defaults to a temp dir)
--explain after the plain summary, print the statistics behind it (bounds, methods,
thresholds, record paths); CALIBRATED_AGENT_EVALUATION_EXPLAIN=1 turns it
on for every command