calibrated-agent-evaluation judge¶
score traces with a judge that has a certifying calibration and a CLEAR bias run
usage: calibrated-agent-evaluation judge [-h] [--trace TRACE_ID] [--judge FILE] [--check FILE]
[--cassette PATH] [--cassette-mode {replay,record}]
[--frame {discovery,production-representative}] [--yes]
[--profile PROFILE] [--region REGION]
[--backend {local,langfuse}] [--root PATH] [--explain]
Admit a judge through calibration, the bias gate and the score gate (records under
<root>/records/), then score the traces in scope and store the verdicts. No calibration, no score.
--check FILE only checks a judge file, with no store and no model.
options:
-h, --help show this help message and exit
--trace TRACE_ID operate on one stored trace (default: every stored trace); an unknown id
is refused
--judge FILE judge file to score with; it must have a certifying calibration record and
a CLEAR bias battery under <root>/records/
--check FILE check a judge file the way the scoring stages load it, then exit: reads no
store, calls no model, and prints 'judge file ok' or the error
--cassette PATH a cassette of judge-model replies, replayed instead of calling the model
or recorded into with --cassette-mode record (default: the judge's live
model, which needs [inspect])
--cassette-mode {replay,record}
replay (default): answer from the cassette only, calling no model; record:
call the judge's live model (needs [inspect] and its credentials) for
every judge call and write each reply to the cassette, overwriting a reply
it already holds
--frame {discovery,production-representative}
score only traces in this frame (production-representative: the random
sample the rate and the gate are read from)
--yes an AgentCore judge file: make the billed Evaluate calls; without it the
command shows the estimate and sends nothing
--profile PROFILE an AgentCore judge file: AWS profile (default: AWS_PROFILE)
--region REGION an AgentCore judge file: AWS region (default: the profile's)
--backend {local,langfuse}
trace-store backend (default: local, offline)
--root PATH local root for the trace store and the stage records (default:
$CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, else ./.calibrated-agent-
evaluation; quickstart defaults to a temp dir)
--explain after the plain summary, print the statistics behind it (bounds, methods,
thresholds, record paths); CALIBRATED_AGENT_EVALUATION_EXPLAIN=1 turns it
on for every command