calibrated-agent-evaluation audit¶
audit an AgentCore A/B test or recommendation: its claim, its arms, a re-test
usage: calibrated-agent-evaluation audit [-h] [--recommendation-id ID] [--arms-read ID]
[--audit-id ID] [--before-root PATH] [--after-root PATH]
[--development-root PATH]
[--verdicts {judge,scorer,human}] [--ab-test-id ID]
[--claim-record ID] [--runtime RUNTIME] [--since SINCE]
[--until UNTIL] [--omni] [--rekey-sessions]
[--applied-bundle-attribute KEY]
[--applied-bundle-require KEY=VALUE] [--profile PROFILE]
[--region REGION] [--backend {local,langfuse}]
[--root PATH] [--explain]
{abtest,arms,recommendation,retest,report} [METRIC]
positional arguments:
{abtest,arms,recommendation,retest,report}
abtest: record what AgentCore reports about an A/B test (a vendor claim,
not evidence); arms: read which arm each trace in a window ran in, from
its span stamps; recommendation: record what AgentCore reports about a
recommendation; retest: decide the change on a held-out replay of both
bundle versions; report: print the library's verdict beside AgentCore's
claim
METRIC retest: the pre-registered target failure mode
options:
-h, --help show this help message and exit
--recommendation-id ID
recommendation, retest, report: the recommendation id
--arms-read ID retest, report: the stored arms read to use when the test has several
--audit-id ID report: the stored re-test to print when the subject has several
--before-root PATH retest: the replay store under the control (or input) bundle version
--after-root PATH retest: the replay store under the treatment (or result) bundle version
--development-root PATH
retest: the store of the traces the change was developed on
--verdicts {judge,scorer,human}
retest: which verdicts decide, as `calibrated-agent-evaluation improve
--evaluate` reads them (default: judge)
--ab-test-id ID the A/B test id
--claim-record ID arms, retest, report: the stored claim record to use when the subject has
several
--runtime RUNTIME arms: the AgentCore Runtime name to read, as at ingest
--since SINCE arms: start of the read window as ISO-8601, as at ingest
--until UNTIL arms: end of the read window as ISO-8601, as at ingest
--omni arms: read spans through CloudWatch Omni, as at ingest (default: Logs
Insights)
--rekey-sessions arms: re-key mixed-session traces to the runtime session id, as at ingest
--applied-bundle-attribute KEY
arms: a span attribute that must equal the arm's bundle version
--applied-bundle-require KEY=VALUE
arms: a span attribute value every assigned trace must carry (repeatable)
--profile PROFILE AWS profile (default: AWS_PROFILE)
--region REGION AWS region (default: the profile's)
--backend {local,langfuse}
trace-store backend (default: local, offline)
--root PATH local root for the trace store and the stage records (default:
$CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, else ./.calibrated-agent-
evaluation; quickstart defaults to a temp dir)
--explain after the plain summary, print the statistics behind it (bounds, methods,
thresholds, record paths); CALIBRATED_AGENT_EVALUATION_EXPLAIN=1 turns it
on for every command