calibrated-agent-evaluation run¶
gate the stored verdicts, correct each judged mode's rate, record the run
usage: calibrated-agent-evaluation run [-h] [--trace TRACE_ID] [--ci] [--threshold THRESHOLD]
[--verdicts {judge,scorer,human}] [--judge FILE]
[--profile PROFILE] [--region REGION] [--seed SEED]
[--n-resamples N_RESAMPLES] [--publish {local,langfuse}]
[--drift-baseline STORE]
[--drift-feature {input-chars,turns,tool-calls}]
[--fail-on-drift] [--atoms FILE]
[--backend {local,langfuse}] [--root PATH] [--explain]
The CI gate: every failure mode's pass-rate 95% lower bound must clear --threshold (--ci makes a
failure exit 1), over the LLM-judge verdicts or, with --verdicts scorer or --verdicts human, the
code-scorer verdicts or the human labels. LLM-judge verdicts are gated only when the judge that
made them is admitted (a certifying calibration and a CLEAR bias battery under <root>/records/);
verdicts from any other judge are refused. With --judge, each judged mode's rate is corrected for
the judge's error. Each run is recorded under <root>/records/run/. Exit codes: 0 the run finished
(and, with --ci, the gate passed); 1 the gate failed under --ci, a refusal, or a failed --publish;
2 a usage error; 3 with --fail-on-drift, input drift or a stale judge was flagged while the gate
did not fail.
options:
-h, --help show this help message and exit
--trace TRACE_ID operate on one stored trace (default: every stored trace); an unknown id
is refused
--ci exit non-zero if the CI gate fails (without it a failing gate is reported
only)
--threshold THRESHOLD
the bar each mode's pass-rate 95% lower bound must clear (default: 0.80)
--verdicts {judge,scorer,human}
which verdicts the gate reads: judge (the LLM-judge verdicts, the
default), scorer (the code-scorer verdicts) or human (the human labels); a
source with none in scope is refused, never swapped
--judge FILE judge file whose verdicts get a corrected-rate trend (repeatable; its
calibration is read from <root>/records/calibration)
--profile PROFILE an AgentCore judge file: AWS profile (default: AWS_PROFILE)
--region REGION an AgentCore judge file: AWS region (default: the profile's)
--seed SEED seed of the corrected-rate bootstrap, recorded on the run record (default:
0)
--n-resamples N_RESAMPLES
bootstrap resamples for the corrected-rate interval (default: 9999)
--publish {local,langfuse}
after the run is recorded, publish its rates and drift to a dashboard:
local writes <root>/dashboard/<run_id>/, langfuse also sends them as
scores to the LANGFUSE_* project (whatever --backend is); a failed publish
exits 1
--drift-baseline STORE
a local store whose production-representative traces are the baseline;
compares --drift-feature and the taxonomy-cell mix with the traces in
scope (reported, never changes the gate)
--drift-feature {input-chars,turns,tool-calls}
with --drift-baseline: the input feature to compare (input-chars:
characters of the trace input; turns: agent steps; tool-calls: tool
calls); required, no default
--fail-on-drift with --drift-baseline: exit 3 when input drift is flagged or an admitted
judge looks stale (a failing gate under --ci still exits 1)
--atoms FILE an AtomSet JSON file; every gated failure mode must be one of its
objective atoms, and a subjective dimension is refused (it is tracked,
never gated)
--backend {local,langfuse}
trace-store backend (default: local, offline)
--root PATH local root for the trace store and the stage records (default:
$CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, else ./.calibrated-agent-
evaluation; quickstart defaults to a temp dir)
--explain after the plain summary, print the statistics behind it (bounds, methods,
thresholds, record paths); CALIBRATED_AGENT_EVALUATION_EXPLAIN=1 turns it
on for every command