Skip to content

calibrated-agent-evaluation run

gate the stored verdicts, correct each judged mode's rate, record the run

usage: calibrated-agent-evaluation run [-h] [--trace TRACE_ID] [--ci] [--threshold THRESHOLD]
                                       [--verdicts {judge,scorer,human}] [--judge FILE]
                                       [--profile PROFILE] [--region REGION] [--seed SEED]
                                       [--n-resamples N_RESAMPLES] [--publish {local,langfuse}]
                                       [--drift-baseline STORE]
                                       [--drift-feature {input-chars,turns,tool-calls}]
                                       [--fail-on-drift] [--atoms FILE]
                                       [--backend {local,langfuse}] [--root PATH] [--explain]

The CI gate: every failure mode's pass-rate 95% lower bound must clear --threshold (--ci makes a
failure exit 1), over the LLM-judge verdicts or, with --verdicts scorer or --verdicts human, the
code-scorer verdicts or the human labels. LLM-judge verdicts are gated only when the judge that
made them is admitted (a certifying calibration and a CLEAR bias battery under <root>/records/);
verdicts from any other judge are refused. With --judge, each judged mode's rate is corrected for
the judge's error. Each run is recorded under <root>/records/run/. Exit codes: 0 the run finished
(and, with --ci, the gate passed); 1 the gate failed under --ci, a refusal, or a failed --publish;
2 a usage error; 3 with --fail-on-drift, input drift or a stale judge was flagged while the gate
did not fail.

options:
  -h, --help            show this help message and exit
  --trace TRACE_ID      operate on one stored trace (default: every stored trace); an unknown id
                        is refused
  --ci                  exit non-zero if the CI gate fails (without it a failing gate is reported
                        only)
  --threshold THRESHOLD
                        the bar each mode's pass-rate 95% lower bound must clear (default: 0.80)
  --verdicts {judge,scorer,human}
                        which verdicts the gate reads: judge (the LLM-judge verdicts, the
                        default), scorer (the code-scorer verdicts) or human (the human labels); a
                        source with none in scope is refused, never swapped
  --judge FILE          judge file whose verdicts get a corrected-rate trend (repeatable; its
                        calibration is read from <root>/records/calibration)
  --profile PROFILE     an AgentCore judge file: AWS profile (default: AWS_PROFILE)
  --region REGION       an AgentCore judge file: AWS region (default: the profile's)
  --seed SEED           seed of the corrected-rate bootstrap, recorded on the run record (default:
                        0)
  --n-resamples N_RESAMPLES
                        bootstrap resamples for the corrected-rate interval (default: 9999)
  --publish {local,langfuse}
                        after the run is recorded, publish its rates and drift to a dashboard:
                        local writes <root>/dashboard/<run_id>/, langfuse also sends them as
                        scores to the LANGFUSE_* project (whatever --backend is); a failed publish
                        exits 1
  --drift-baseline STORE
                        a local store whose production-representative traces are the baseline;
                        compares --drift-feature and the taxonomy-cell mix with the traces in
                        scope (reported, never changes the gate)
  --drift-feature {input-chars,turns,tool-calls}
                        with --drift-baseline: the input feature to compare (input-chars:
                        characters of the trace input; turns: agent steps; tool-calls: tool
                        calls); required, no default
  --fail-on-drift       with --drift-baseline: exit 3 when input drift is flagged or an admitted
                        judge looks stale (a failing gate under --ci still exits 1)
  --atoms FILE          an AtomSet JSON file; every gated failure mode must be one of its
                        objective atoms, and a subjective dimension is refused (it is tracked,
                        never gated)
  --backend {local,langfuse}
                        trace-store backend (default: local, offline)
  --root PATH           local root for the trace store and the stage records (default:
                        $CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, else ./.calibrated-agent-
                        evaluation; quickstart defaults to a temp dir)
  --explain             after the plain summary, print the statistics behind it (bounds, methods,
                        thresholds, record paths); CALIBRATED_AGENT_EVALUATION_EXPLAIN=1 turns it
                        on for every command