Skip to content

calibrated-agent-evaluation bias

run a certified judge's bias battery (the second trust gate)

usage: calibrated-agent-evaluation bias [-h] --judge FILE [--cassette PATH]
                                        [--cassette-mode {replay,record}] --probes DIR
                                        --thresholds FILE [--yes] [--profile PROFILE]
                                        [--region REGION] [--battery-id ID]
                                        [--backend {local,langfuse}] [--root PATH] [--explain]

Run the CALM bias battery for a judge that holds a certifying calibration record, through the
judge's own model (or --cassette). The record goes to <root>/records/bias/; a BIASED judge exits
1.

options:
  -h, --help            show this help message and exit
  --judge FILE          the judge file, one binary judge as JSON (`calibrated-agent-evaluation
                        judge --check FILE` validates one; `calibrated-agent-evaluation quickstart
                        --certify` writes examples)
  --cassette PATH       a cassette of judge-model replies, replayed instead of calling the model
                        or recorded into with --cassette-mode record (default: the judge's live
                        model, which needs [inspect])
  --cassette-mode {replay,record}
                        replay (default): answer from the cassette only, calling no model; record:
                        call the judge's live model (needs [inspect] and its credentials) for
                        every judge call and write each reply to the cassette, overwriting a reply
                        it already holds
  --probes DIR          directory of bias probe files, one *.json per bias (`calibrated-agent-
                        evaluation quickstart --certify` writes an example set)
  --thresholds FILE     per-model bias ceilings as JSON (`calibrated-agent-evaluation quickstart
                        --certify` writes an example)
  --yes                 an AgentCore judge file: make the billed Evaluate calls; without it the
                        command shows the estimate and sends nothing
  --profile PROFILE     an AgentCore judge file: AWS profile (default: AWS_PROFILE)
  --region REGION       an AgentCore judge file: AWS region (default: the profile's)
  --battery-id ID       id for the battery record (default: a random one)
  --backend {local,langfuse}
                        trace-store backend (default: local, offline)
  --root PATH           local root for the trace store and the stage records (default:
                        $CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, else ./.calibrated-agent-
                        evaluation; quickstart defaults to a temp dir)
  --explain             after the plain summary, print the statistics behind it (bounds, methods,
                        thresholds, record paths); CALIBRATED_AGENT_EVALUATION_EXPLAIN=1 turns it
                        on for every command