calibrated-agent-evaluation bias¶
run a certified judge's bias battery (the second trust gate)
usage: calibrated-agent-evaluation bias [-h] --judge FILE [--cassette PATH]
[--cassette-mode {replay,record}] --probes DIR
--thresholds FILE [--yes] [--profile PROFILE]
[--region REGION] [--battery-id ID]
[--backend {local,langfuse}] [--root PATH] [--explain]
Run the CALM bias battery for a judge that holds a certifying calibration record, through the
judge's own model (or --cassette). The record goes to <root>/records/bias/; a BIASED judge exits
1.
options:
-h, --help show this help message and exit
--judge FILE the judge file, one binary judge as JSON (`calibrated-agent-evaluation
judge --check FILE` validates one; `calibrated-agent-evaluation quickstart
--certify` writes examples)
--cassette PATH a cassette of judge-model replies, replayed instead of calling the model
or recorded into with --cassette-mode record (default: the judge's live
model, which needs [inspect])
--cassette-mode {replay,record}
replay (default): answer from the cassette only, calling no model; record:
call the judge's live model (needs [inspect] and its credentials) for
every judge call and write each reply to the cassette, overwriting a reply
it already holds
--probes DIR directory of bias probe files, one *.json per bias (`calibrated-agent-
evaluation quickstart --certify` writes an example set)
--thresholds FILE per-model bias ceilings as JSON (`calibrated-agent-evaluation quickstart
--certify` writes an example)
--yes an AgentCore judge file: make the billed Evaluate calls; without it the
command shows the estimate and sends nothing
--profile PROFILE an AgentCore judge file: AWS profile (default: AWS_PROFILE)
--region REGION an AgentCore judge file: AWS region (default: the profile's)
--battery-id ID id for the battery record (default: a random one)
--backend {local,langfuse}
trace-store backend (default: local, offline)
--root PATH local root for the trace store and the stage records (default:
$CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, else ./.calibrated-agent-
evaluation; quickstart defaults to a temp dir)
--explain after the plain summary, print the statistics behind it (bounds, methods,
thresholds, record paths); CALIBRATED_AGENT_EVALUATION_EXPLAIN=1 turns it
on for every command