calibrated-agent-evaluation calibrate¶
certify a judge against the human gold in the store (the first trust gate)
usage: calibrated-agent-evaluation calibrate [-h] --judge FILE [--cassette PATH]
[--cassette-mode {replay,record}]
[--expected-accuracy P] [--expected-tnr P]
[--rate-safe] [--raters N] [--cell KEY=VALUE]
[--check] [--yes] [--profile PROFILE]
[--region REGION] [--backend {local,langfuse}]
[--root PATH] [--explain]
Run the judge over the stored traces that carry human gold for its failure mode, then certify it:
CERTIFIED only when the 95% lower bounds of TPR and TNR both reach 0.80. Reports how many gold
labels per class the plan needs and whether the set has them; --rate-safe plans enough for
`calibrated-agent-evaluation run`'s corrected rate at any observed accuracy; gold that two raters
double-code (--raters 2, or two raters already in it) is planned at 65 per class or more so their
agreement can be measured on it. The record goes to <root>/records/calibration/ either way; a
REFUSED judge exits 1.
options:
-h, --help show this help message and exit
--judge FILE the judge file, one binary judge as JSON (`calibrated-agent-evaluation
judge --check FILE` validates one; `calibrated-agent-evaluation quickstart
--certify` writes examples)
--cassette PATH a cassette of judge-model replies, replayed instead of calling the model
or recorded into with --cassette-mode record (default: the judge's live
model, which needs [inspect])
--cassette-mode {replay,record}
replay (default): answer from the cassette only, calling no model; record:
call the judge's live model (needs [inspect] and its credentials) for
every judge call and write each reply to the cassette, overwriting a reply
it already holds
--expected-accuracy P
the judge accuracy the gold-set plan assumes (default: 0.90); with
--expected-tnr it is the expected TPR
--expected-tnr P the expected TNR, for a judge expected to be better on one class
--rate-safe plan for a corrected rate: the per-class size at which any accuracy that
certifies leaves the record rate-safe (162 at tau 0.80); without it the
plan only certifies the judge and covers the rate at the expected accuracy
--raters N how many raters will label this gold (default: the raters already in it).
From 2 the gold is also what they double-code for the agreement check, so
the plan is at least 65 per class (130 shared traces)
--cell KEY=VALUE calibrate only on traces whose taxonomy cell has KEY=VALUE (repeatable),
e.g. the agent the judge was written for
--check re-check a certified judge on fresh gold (traces no calibration of its
mode used) and record the change in TPR and TNR as a judge-check record;
never writes or alters a calibration record. The fresh gold needed per
class is derived from the certifying record
--yes an AgentCore judge file: make the billed Evaluate calls; without it the
command shows the estimate and sends nothing
--profile PROFILE an AgentCore judge file: AWS profile (default: AWS_PROFILE)
--region REGION an AgentCore judge file: AWS region (default: the profile's)
--backend {local,langfuse}
trace-store backend (default: local, offline)
--root PATH local root for the trace store and the stage records (default:
$CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, else ./.calibrated-agent-
evaluation; quickstart defaults to a temp dir)
--explain after the plain summary, print the statistics behind it (bounds, methods,
thresholds, record paths); CALIBRATED_AGENT_EVALUATION_EXPLAIN=1 turns it
on for every command