Skip to content

calibrated-agent-evaluation online

put a calibrated Calibrated Agent Evaluation judge into AgentCore online evaluation

usage: calibrated-agent-evaluation online [-h] [--judge FILE] [--name NAME] [--out FILE]
                                          [--runtime-arn ARN] [--endpoint NAME] [--role-arn ARN]
                                          [--sampling PCT] [--session-timeout MINUTES]
                                          [--filter KEY OPERATOR VALUE] [--wait SECONDS] [--yes]
                                          [--config CONFIG_ID] [--since ISO] [--until ISO]
                                          [--omni] [--wait-minutes N] [--drift-baseline STORE]
                                          [--drift-feature {input-chars,turns,tool-calls}]
                                          [--seed SEED] [--profile PROFILE] [--region REGION]
                                          [--backend {local,langfuse}] [--root PATH] [--explain]
                                          {create-evaluator,attach,read,publish,cost}

create-evaluator creates a workflow-level prompt judge as an AgentCore LLM-as-judge evaluator at
level TRACE (the judge's prompt with AgentCore's trace placeholders, a 0.0 FAIL / 1.0 PASS scale
and the judge's Bedrock model), waits for it to become ACTIVE and writes an agentcore judge file
naming it. Without --yes it shows the exact request and sends nothing. Needs the agentcore extra.

positional arguments:
  {create-evaluator,attach,read,publish,cost}
                        create-evaluator: create a prompt judge as an AgentCore evaluator and
                        write an agentcore judge file naming it; attach: attach certified, bias-
                        cleared agentcore judges to a new online evaluation config on a deployed
                        runtime; read: read the config's verdicts back into the store and check
                        the judges for staleness (and, when asked, the traffic for drift);
                        publish: publish each judge's corrected pass rate to CloudWatch; cost:
                        count and price the run's evaluations

options:
  -h, --help            show this help message and exit
  --judge FILE          create-evaluator: the prompt judge file; attach, read, publish: an
                        agentcore judge file, one per attached evaluator (repeatable)
  --name NAME           create-evaluator: the evaluator name (default: the judge id and prompt
                        version, for example orderbot_refund_policy_v2); attach: the online
                        config's name
  --out FILE            create-evaluator: the agentcore judge file to write (default:
                        <judge>.agentcore.json in the current directory); it must not exist
  --runtime-arn ARN     attach: the deployed runtime's ARN (default:
                        CALIBRATED_AGENT_EVALUATION_AGENTCORE_RUNTIME_ARN)
  --endpoint NAME       attach: the runtime endpoint whose sessions are scored (default: DEFAULT)
  --role-arn ARN        attach: the role AgentCore assumes to run the config (default:
                        CALIBRATED_AGENT_EVALUATION_AGENTCORE_EVAL_ROLE_ARN)
  --sampling PCT        attach: the percentage of sessions scored, 0.01 to 100 (required)
  --session-timeout MINUTES
                        attach: how long a session is idle before it is scored, 1 to 1440
                        (default: 15)
  --filter KEY OPERATOR VALUE
                        attach: score only sessions whose KEY compares true to VALUE (repeatable,
                        at most five); VALUE is sent as a number or true/false when it reads as
                        one
  --wait SECONDS        how long to wait for the evaluator (create-evaluator, default 120) or the
                        config (attach, default 300) to become ACTIVE
  --yes                 send the request; without it the command only shows it
  --config CONFIG_ID    read, publish, cost: the online config (may be left out when the store
                        holds one)
  --since ISO           read: the window's start, with a timezone (default: when the config was
                        created)
  --until ISO           read: the window's end, with a timezone (default: now)
  --omni                read: read the traces through CloudWatch Omni instead of Logs Insights
  --wait-minutes N      read: how long past a session's timeout its result may still come before
                        the session counts as unscored (default: 15)
  --drift-baseline STORE
                        read: a local store whose production-representative traces are the
                        baseline; compares --drift-feature with the traces read back (optional, as
                        on calibrated-agent-evaluation run)
  --drift-feature {input-chars,turns,tool-calls}
                        read, with --drift-baseline: the input feature to compare (input-chars:
                        characters of the trace input; turns: agent steps; tool-calls: tool calls)
  --seed SEED           read: the seed of the drift check's permutation test (default: 0)
  --profile PROFILE     AWS profile (default: AWS_PROFILE)
  --region REGION       AWS region (default: the profile's)
  --backend {local,langfuse}
                        trace-store backend (default: local, offline)
  --root PATH           local root for the trace store and the stage records (default:
                        $CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, else ./.calibrated-agent-
                        evaluation; quickstart defaults to a temp dir)
  --explain             after the plain summary, print the statistics behind it (bounds, methods,
                        thresholds, record paths); CALIBRATED_AGENT_EVALUATION_EXPLAIN=1 turns it
                        on for every command