Skip to content

calibrated-agent-evaluation audit

audit an AgentCore A/B test or recommendation: its claim, its arms, a re-test

usage: calibrated-agent-evaluation audit [-h] [--recommendation-id ID] [--arms-read ID]
                                         [--audit-id ID] [--before-root PATH] [--after-root PATH]
                                         [--development-root PATH]
                                         [--verdicts {judge,scorer,human}] [--ab-test-id ID]
                                         [--claim-record ID] [--runtime RUNTIME] [--since SINCE]
                                         [--until UNTIL] [--omni] [--rekey-sessions]
                                         [--applied-bundle-attribute KEY]
                                         [--applied-bundle-require KEY=VALUE] [--profile PROFILE]
                                         [--region REGION] [--backend {local,langfuse}]
                                         [--root PATH] [--explain]
                                         {abtest,arms,recommendation,retest,report} [METRIC]

positional arguments:
  {abtest,arms,recommendation,retest,report}
                        abtest: record what AgentCore reports about an A/B test (a vendor claim,
                        not evidence); arms: read which arm each trace in a window ran in, from
                        its span stamps; recommendation: record what AgentCore reports about a
                        recommendation; retest: decide the change on a held-out replay of both
                        bundle versions; report: print the library's verdict beside AgentCore's
                        claim
  METRIC                retest: the pre-registered target failure mode

options:
  -h, --help            show this help message and exit
  --recommendation-id ID
                        recommendation, retest, report: the recommendation id
  --arms-read ID        retest, report: the stored arms read to use when the test has several
  --audit-id ID         report: the stored re-test to print when the subject has several
  --before-root PATH    retest: the replay store under the control (or input) bundle version
  --after-root PATH     retest: the replay store under the treatment (or result) bundle version
  --development-root PATH
                        retest: the store of the traces the change was developed on
  --verdicts {judge,scorer,human}
                        retest: which verdicts decide, as `calibrated-agent-evaluation improve
                        --evaluate` reads them (default: judge)
  --ab-test-id ID       the A/B test id
  --claim-record ID     arms, retest, report: the stored claim record to use when the subject has
                        several
  --runtime RUNTIME     arms: the AgentCore Runtime name to read, as at ingest
  --since SINCE         arms: start of the read window as ISO-8601, as at ingest
  --until UNTIL         arms: end of the read window as ISO-8601, as at ingest
  --omni                arms: read spans through CloudWatch Omni, as at ingest (default: Logs
                        Insights)
  --rekey-sessions      arms: re-key mixed-session traces to the runtime session id, as at ingest
  --applied-bundle-attribute KEY
                        arms: a span attribute that must equal the arm's bundle version
  --applied-bundle-require KEY=VALUE
                        arms: a span attribute value every assigned trace must carry (repeatable)
  --profile PROFILE     AWS profile (default: AWS_PROFILE)
  --region REGION       AWS region (default: the profile's)
  --backend {local,langfuse}
                        trace-store backend (default: local, offline)
  --root PATH           local root for the trace store and the stage records (default:
                        $CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, else ./.calibrated-agent-
                        evaluation; quickstart defaults to a temp dir)
  --explain             after the plain summary, print the statistics behind it (bounds, methods,
                        thresholds, record paths); CALIBRATED_AGENT_EVALUATION_EXPLAIN=1 turns it
                        on for every command