calibrated-agent-evaluation probe¶
black-box probe an agent endpoint and store what it shows (needs --authorized)
usage: calibrated-agent-evaluation probe [-h] [--endpoint ENDPOINT] [--authorized]
[--auth-bearer-env ENV_VAR] [--auth-api-key-env ENV_VAR]
[--auth-api-key-header HEADER]
[--auth-header-env ENV_VAR]
[--auth-subprotocol-env ENV_VAR] [--prompt TEXT]
[--timeout TIMEOUT] [--battery [BENCHMARK ...]]
[--trials K] [--allow-use-restricted]
[--grader-model MODEL] [--executor {agentcore}]
[--runtime-arn ARN] [--qualifier NAME]
[--payload-format {generic_message,openai,agentcore_sse,nested_agentcore}]
[--config-bundle-arn ARN]
[--config-bundle-version VERSION]
[--read-timeout SECONDS] [--profile NAME] [--log-dir DIR]
[--yes] [--policy-cases PATH]
[--backend {local,langfuse}] [--root PATH] [--explain]
Send prompts to an agent endpoint you are authorized to test, infer its observable tools and
failure hypotheses, and store the probed traces for the later stages.
options:
-h, --help show this help message and exit
--endpoint ENDPOINT target endpoint to black-box probe (http(s):// or ws(s)://); omit for
guided interactive setup when run in a terminal
--authorized attest you are authorized to probe the target (the --endpoint, or with
--executor agentcore the --runtime-arn); required before anything is sent
--auth-bearer-env ENV_VAR
name of the env var holding a bearer token (value never inlined/logged)
--auth-api-key-env ENV_VAR
name of the env var holding an API key (paired with --auth-api-key-header)
--auth-api-key-header HEADER
header name to carry the API key (e.g. 'x-api-key')
--auth-header-env ENV_VAR
name of the env var holding a JSON object of raw header name->value pairs
--auth-subprotocol-env ENV_VAR
WebSocket only: env var holding a token sent in Sec-WebSocket-Protocol
--prompt TEXT a prompt to send (repeatable); omit to use the built-in default seeds
--timeout TIMEOUT URL probe only: per-request timeout in seconds (default: 30); an AgentCore
probe takes --read-timeout
--battery [BENCHMARK ...]
URL probe only: run published safety benchmarks against the endpoint
instead of the seed prompts (strong_reject, xstest; no names runs both).
One benchmark sample is one trial, and the result is an upper bound on the
break rate, never a point estimate
--trials K URL probe only: how many samples of each --battery benchmark to run; more
trials tighten the upper bound (0 breaks in 59 trials bounds it at 0.05)
--allow-use-restricted
URL probe only: include --battery benchmarks whose dataset carries a use
restriction (AgentHarm)
--grader-model MODEL URL probe only: the model a --battery benchmark grades replies with (its
own default is an OpenAI model); e.g. bedrock/global.anthropic.claude-
sonnet-4-6
--executor {agentcore}
send the prompts to an agent hosted on Amazon Bedrock AgentCore Runtime,
named by --runtime-arn, instead of an --endpoint URL (agentcore; needs the
agentcore extra)
--runtime-arn ARN with --executor agentcore: the AgentCore Runtime agent to send to, by its
runtime ARN; every AWS call goes to the ARN's region
--qualifier NAME with --executor agentcore: the runtime endpoint to invoke (default:
DEFAULT)
--payload-format {generic_message,openai,agentcore_sse,nested_agentcore}
with --executor agentcore: the request body the agent reads (required: it
is never guessed, since each guess would be a billed call)
--config-bundle-arn ARN
with --executor agentcore: a configuration bundle for the agent to run
with (needs --config-bundle-version)
--config-bundle-version VERSION
with --executor agentcore: the version of --config-bundle-arn
--read-timeout SECONDS
with --executor agentcore: how long to wait for each read of the agent's
response, in seconds, not the whole answer (default: 300)
--profile NAME with --executor agentcore: the AWS profile to call AWS with (default:
AWS_PROFILE)
--log-dir DIR with --executor agentcore: where generate and probe keep the run manifest,
agentcore_run_record.json and .csv with one record per run (default:
<root>/records/agentcore, shared by both)
--yes with --executor agentcore: go ahead with the billed calls the printed plan
lists (needed for every run that can call AWS; a cassette replay calls
none)
--policy-cases PATH with --executor agentcore: run the policy cases in PATH (a JSON case file)
and score each case from the Gateway Policy engine's recorded decisions,
instead of probing with --prompt
--backend {local,langfuse}
trace-store backend (default: local, offline)
--root PATH local root for the trace store and the stage records (default:
$CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, else ./.calibrated-agent-
evaluation; quickstart defaults to a temp dir)
--explain after the plain summary, print the statistics behind it (bounds, methods,
thresholds, record paths); CALIBRATED_AGENT_EVALUATION_EXPLAIN=1 turns it
on for every command