Skip to content

calibrated-agent-evaluation calibrate

certify a judge against the human gold in the store (the first trust gate)

usage: calibrated-agent-evaluation calibrate [-h] --judge FILE [--cassette PATH]
                                             [--cassette-mode {replay,record}]
                                             [--expected-accuracy P] [--expected-tnr P]
                                             [--rate-safe] [--raters N] [--cell KEY=VALUE]
                                             [--check] [--yes] [--profile PROFILE]
                                             [--region REGION] [--backend {local,langfuse}]
                                             [--root PATH] [--explain]

Run the judge over the stored traces that carry human gold for its failure mode, then certify it:
CERTIFIED only when the 95% lower bounds of TPR and TNR both reach 0.80. Reports how many gold
labels per class the plan needs and whether the set has them; --rate-safe plans enough for
`calibrated-agent-evaluation run`'s corrected rate at any observed accuracy; gold that two raters
double-code (--raters 2, or two raters already in it) is planned at 65 per class or more so their
agreement can be measured on it. The record goes to <root>/records/calibration/ either way; a
REFUSED judge exits 1.

options:
  -h, --help            show this help message and exit
  --judge FILE          the judge file, one binary judge as JSON (`calibrated-agent-evaluation
                        judge --check FILE` validates one; `calibrated-agent-evaluation quickstart
                        --certify` writes examples)
  --cassette PATH       a cassette of judge-model replies, replayed instead of calling the model
                        or recorded into with --cassette-mode record (default: the judge's live
                        model, which needs [inspect])
  --cassette-mode {replay,record}
                        replay (default): answer from the cassette only, calling no model; record:
                        call the judge's live model (needs [inspect] and its credentials) for
                        every judge call and write each reply to the cassette, overwriting a reply
                        it already holds
  --expected-accuracy P
                        the judge accuracy the gold-set plan assumes (default: 0.90); with
                        --expected-tnr it is the expected TPR
  --expected-tnr P      the expected TNR, for a judge expected to be better on one class
  --rate-safe           plan for a corrected rate: the per-class size at which any accuracy that
                        certifies leaves the record rate-safe (162 at tau 0.80); without it the
                        plan only certifies the judge and covers the rate at the expected accuracy
  --raters N            how many raters will label this gold (default: the raters already in it).
                        From 2 the gold is also what they double-code for the agreement check, so
                        the plan is at least 65 per class (130 shared traces)
  --cell KEY=VALUE      calibrate only on traces whose taxonomy cell has KEY=VALUE (repeatable),
                        e.g. the agent the judge was written for
  --check               re-check a certified judge on fresh gold (traces no calibration of its
                        mode used) and record the change in TPR and TNR as a judge-check record;
                        never writes or alters a calibration record. The fresh gold needed per
                        class is derived from the certifying record
  --yes                 an AgentCore judge file: make the billed Evaluate calls; without it the
                        command shows the estimate and sends nothing
  --profile PROFILE     an AgentCore judge file: AWS profile (default: AWS_PROFILE)
  --region REGION       an AgentCore judge file: AWS region (default: the profile's)
  --backend {local,langfuse}
                        trace-store backend (default: local, offline)
  --root PATH           local root for the trace store and the stage records (default:
                        $CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, else ./.calibrated-agent-
                        evaluation; quickstart defaults to a temp dir)
  --explain             after the plain summary, print the statistics behind it (bounds, methods,
                        thresholds, record paths); CALIBRATED_AGENT_EVALUATION_EXPLAIN=1 turns it
                        on for every command