Skip to content

calibrated-agent-evaluation judge

score traces with a judge that has a certifying calibration and a CLEAR bias run

usage: calibrated-agent-evaluation judge [-h] [--trace TRACE_ID] [--judge FILE] [--check FILE]
                                         [--cassette PATH] [--cassette-mode {replay,record}]
                                         [--frame {discovery,production-representative}] [--yes]
                                         [--profile PROFILE] [--region REGION]
                                         [--backend {local,langfuse}] [--root PATH] [--explain]

Admit a judge through calibration, the bias gate and the score gate (records under
<root>/records/), then score the traces in scope and store the verdicts. No calibration, no score.
--check FILE only checks a judge file, with no store and no model.

options:
  -h, --help            show this help message and exit
  --trace TRACE_ID      operate on one stored trace (default: every stored trace); an unknown id
                        is refused
  --judge FILE          judge file to score with; it must have a certifying calibration record and
                        a CLEAR bias battery under <root>/records/
  --check FILE          check a judge file the way the scoring stages load it, then exit: reads no
                        store, calls no model, and prints 'judge file ok' or the error
  --cassette PATH       a cassette of judge-model replies, replayed instead of calling the model
                        or recorded into with --cassette-mode record (default: the judge's live
                        model, which needs [inspect])
  --cassette-mode {replay,record}
                        replay (default): answer from the cassette only, calling no model; record:
                        call the judge's live model (needs [inspect] and its credentials) for
                        every judge call and write each reply to the cassette, overwriting a reply
                        it already holds
  --frame {discovery,production-representative}
                        score only traces in this frame (production-representative: the random
                        sample the rate and the gate are read from)
  --yes                 an AgentCore judge file: make the billed Evaluate calls; without it the
                        command shows the estimate and sends nothing
  --profile PROFILE     an AgentCore judge file: AWS profile (default: AWS_PROFILE)
  --region REGION       an AgentCore judge file: AWS region (default: the profile's)
  --backend {local,langfuse}
                        trace-store backend (default: local, offline)
  --root PATH           local root for the trace store and the stage records (default:
                        $CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, else ./.calibrated-agent-
                        evaluation; quickstart defaults to a temp dir)
  --explain             after the plain summary, print the statistics behind it (bounds, methods,
                        thresholds, record paths); CALIBRATED_AGENT_EVALUATION_EXPLAIN=1 turns it
                        on for every command