calibrated-agent-evaluation online¶
put a calibrated Calibrated Agent Evaluation judge into AgentCore online evaluation
usage: calibrated-agent-evaluation online [-h] [--judge FILE] [--name NAME] [--out FILE]
[--runtime-arn ARN] [--endpoint NAME] [--role-arn ARN]
[--sampling PCT] [--session-timeout MINUTES]
[--filter KEY OPERATOR VALUE] [--wait SECONDS] [--yes]
[--config CONFIG_ID] [--since ISO] [--until ISO]
[--omni] [--wait-minutes N] [--drift-baseline STORE]
[--drift-feature {input-chars,turns,tool-calls}]
[--seed SEED] [--profile PROFILE] [--region REGION]
[--backend {local,langfuse}] [--root PATH] [--explain]
{create-evaluator,attach,read,publish,cost}
create-evaluator creates a workflow-level prompt judge as an AgentCore LLM-as-judge evaluator at
level TRACE (the judge's prompt with AgentCore's trace placeholders, a 0.0 FAIL / 1.0 PASS scale
and the judge's Bedrock model), waits for it to become ACTIVE and writes an agentcore judge file
naming it. Without --yes it shows the exact request and sends nothing. Needs the agentcore extra.
positional arguments:
{create-evaluator,attach,read,publish,cost}
create-evaluator: create a prompt judge as an AgentCore evaluator and
write an agentcore judge file naming it; attach: attach certified, bias-
cleared agentcore judges to a new online evaluation config on a deployed
runtime; read: read the config's verdicts back into the store and check
the judges for staleness (and, when asked, the traffic for drift);
publish: publish each judge's corrected pass rate to CloudWatch; cost:
count and price the run's evaluations
options:
-h, --help show this help message and exit
--judge FILE create-evaluator: the prompt judge file; attach, read, publish: an
agentcore judge file, one per attached evaluator (repeatable)
--name NAME create-evaluator: the evaluator name (default: the judge id and prompt
version, for example orderbot_refund_policy_v2); attach: the online
config's name
--out FILE create-evaluator: the agentcore judge file to write (default:
<judge>.agentcore.json in the current directory); it must not exist
--runtime-arn ARN attach: the deployed runtime's ARN (default:
CALIBRATED_AGENT_EVALUATION_AGENTCORE_RUNTIME_ARN)
--endpoint NAME attach: the runtime endpoint whose sessions are scored (default: DEFAULT)
--role-arn ARN attach: the role AgentCore assumes to run the config (default:
CALIBRATED_AGENT_EVALUATION_AGENTCORE_EVAL_ROLE_ARN)
--sampling PCT attach: the percentage of sessions scored, 0.01 to 100 (required)
--session-timeout MINUTES
attach: how long a session is idle before it is scored, 1 to 1440
(default: 15)
--filter KEY OPERATOR VALUE
attach: score only sessions whose KEY compares true to VALUE (repeatable,
at most five); VALUE is sent as a number or true/false when it reads as
one
--wait SECONDS how long to wait for the evaluator (create-evaluator, default 120) or the
config (attach, default 300) to become ACTIVE
--yes send the request; without it the command only shows it
--config CONFIG_ID read, publish, cost: the online config (may be left out when the store
holds one)
--since ISO read: the window's start, with a timezone (default: when the config was
created)
--until ISO read: the window's end, with a timezone (default: now)
--omni read: read the traces through CloudWatch Omni instead of Logs Insights
--wait-minutes N read: how long past a session's timeout its result may still come before
the session counts as unscored (default: 15)
--drift-baseline STORE
read: a local store whose production-representative traces are the
baseline; compares --drift-feature with the traces read back (optional, as
on calibrated-agent-evaluation run)
--drift-feature {input-chars,turns,tool-calls}
read, with --drift-baseline: the input feature to compare (input-chars:
characters of the trace input; turns: agent steps; tool-calls: tool calls)
--seed SEED read: the seed of the drift check's permutation test (default: 0)
--profile PROFILE AWS profile (default: AWS_PROFILE)
--region REGION AWS region (default: the profile's)
--backend {local,langfuse}
trace-store backend (default: local, offline)
--root PATH local root for the trace store and the stage records (default:
$CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, else ./.calibrated-agent-
evaluation; quickstart defaults to a temp dir)
--explain after the plain summary, print the statistics behind it (bounds, methods,
thresholds, record paths); CALIBRATED_AGENT_EVALUATION_EXPLAIN=1 turns it
on for every command