Skip to content

calibrated-agent-evaluation probe

black-box probe an agent endpoint and store what it shows (needs --authorized)

usage: calibrated-agent-evaluation probe [-h] [--endpoint ENDPOINT] [--authorized]
                                         [--auth-bearer-env ENV_VAR] [--auth-api-key-env ENV_VAR]
                                         [--auth-api-key-header HEADER]
                                         [--auth-header-env ENV_VAR]
                                         [--auth-subprotocol-env ENV_VAR] [--prompt TEXT]
                                         [--timeout TIMEOUT] [--battery [BENCHMARK ...]]
                                         [--trials K] [--allow-use-restricted]
                                         [--grader-model MODEL] [--executor {agentcore}]
                                         [--runtime-arn ARN] [--qualifier NAME]
                                         [--payload-format {generic_message,openai,agentcore_sse,nested_agentcore}]
                                         [--config-bundle-arn ARN]
                                         [--config-bundle-version VERSION]
                                         [--read-timeout SECONDS] [--profile NAME] [--log-dir DIR]
                                         [--yes] [--policy-cases PATH]
                                         [--backend {local,langfuse}] [--root PATH] [--explain]

Send prompts to an agent endpoint you are authorized to test, infer its observable tools and
failure hypotheses, and store the probed traces for the later stages.

options:
  -h, --help            show this help message and exit
  --endpoint ENDPOINT   target endpoint to black-box probe (http(s):// or ws(s)://); omit for
                        guided interactive setup when run in a terminal
  --authorized          attest you are authorized to probe the target (the --endpoint, or with
                        --executor agentcore the --runtime-arn); required before anything is sent
  --auth-bearer-env ENV_VAR
                        name of the env var holding a bearer token (value never inlined/logged)
  --auth-api-key-env ENV_VAR
                        name of the env var holding an API key (paired with --auth-api-key-header)
  --auth-api-key-header HEADER
                        header name to carry the API key (e.g. 'x-api-key')
  --auth-header-env ENV_VAR
                        name of the env var holding a JSON object of raw header name->value pairs
  --auth-subprotocol-env ENV_VAR
                        WebSocket only: env var holding a token sent in Sec-WebSocket-Protocol
  --prompt TEXT         a prompt to send (repeatable); omit to use the built-in default seeds
  --timeout TIMEOUT     URL probe only: per-request timeout in seconds (default: 30); an AgentCore
                        probe takes --read-timeout
  --battery [BENCHMARK ...]
                        URL probe only: run published safety benchmarks against the endpoint
                        instead of the seed prompts (strong_reject, xstest; no names runs both).
                        One benchmark sample is one trial, and the result is an upper bound on the
                        break rate, never a point estimate
  --trials K            URL probe only: how many samples of each --battery benchmark to run; more
                        trials tighten the upper bound (0 breaks in 59 trials bounds it at 0.05)
  --allow-use-restricted
                        URL probe only: include --battery benchmarks whose dataset carries a use
                        restriction (AgentHarm)
  --grader-model MODEL  URL probe only: the model a --battery benchmark grades replies with (its
                        own default is an OpenAI model); e.g. bedrock/global.anthropic.claude-
                        sonnet-4-6
  --executor {agentcore}
                        send the prompts to an agent hosted on Amazon Bedrock AgentCore Runtime,
                        named by --runtime-arn, instead of an --endpoint URL (agentcore; needs the
                        agentcore extra)
  --runtime-arn ARN     with --executor agentcore: the AgentCore Runtime agent to send to, by its
                        runtime ARN; every AWS call goes to the ARN's region
  --qualifier NAME      with --executor agentcore: the runtime endpoint to invoke (default:
                        DEFAULT)
  --payload-format {generic_message,openai,agentcore_sse,nested_agentcore}
                        with --executor agentcore: the request body the agent reads (required: it
                        is never guessed, since each guess would be a billed call)
  --config-bundle-arn ARN
                        with --executor agentcore: a configuration bundle for the agent to run
                        with (needs --config-bundle-version)
  --config-bundle-version VERSION
                        with --executor agentcore: the version of --config-bundle-arn
  --read-timeout SECONDS
                        with --executor agentcore: how long to wait for each read of the agent's
                        response, in seconds, not the whole answer (default: 300)
  --profile NAME        with --executor agentcore: the AWS profile to call AWS with (default:
                        AWS_PROFILE)
  --log-dir DIR         with --executor agentcore: where generate and probe keep the run manifest,
                        agentcore_run_record.json and .csv with one record per run (default:
                        <root>/records/agentcore, shared by both)
  --yes                 with --executor agentcore: go ahead with the billed calls the printed plan
                        lists (needed for every run that can call AWS; a cassette replay calls
                        none)
  --policy-cases PATH   with --executor agentcore: run the policy cases in PATH (a JSON case file)
                        and score each case from the Gateway Policy engine's recorded decisions,
                        instead of probing with --prompt
  --backend {local,langfuse}
                        trace-store backend (default: local, offline)
  --root PATH           local root for the trace store and the stage records (default:
                        $CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, else ./.calibrated-agent-
                        evaluation; quickstart defaults to a temp dir)
  --explain             after the plain summary, print the statistics behind it (bounds, methods,
                        thresholds, record paths); CALIBRATED_AGENT_EVALUATION_EXPLAIN=1 turns it
                        on for every command