CLI reference¶
The calibrated-agent-evaluation command runs the whole evaluation pipeline. These pages are generated from the argparse parser when the docs are built, so they always match the installed CLI.
| Command | What it does |
|---|---|
calibrated-agent-evaluation quickstart |
run the bundled sample corpus end-to-end, offline (no setup, no keys) |
calibrated-agent-evaluation doctor |
report the install: Python, which optional extras are present, backend reachability |
calibrated-agent-evaluation ingest |
normalize trace dumps (OTLP / Inspect .eval / JSONL / parquet) into the store; refuses lossy ones |
calibrated-agent-evaluation capture |
run an instrumented in-process agent, capture its OTel trace, and ingest it |
calibrated-agent-evaluation dataset |
turn labeled failures into test scenarios; publish and pull them as AgentCore dataset versions |
calibrated-agent-evaluation audit |
audit an AgentCore A/B test or recommendation: its claim, its arms, a re-test |
calibrated-agent-evaluation online |
put a calibrated Calibrated Agent Evaluation judge into AgentCore online evaluation |
calibrated-agent-evaluation probe |
black-box probe an agent endpoint and store what it shows (needs --authorized) |
calibrated-agent-evaluation define |
build the discovery frame (a covering array) from a taxonomy |
calibrated-agent-evaluation generate |
synthesize frame-tagged inputs; with --run, route them through your agent |
calibrated-agent-evaluation label |
open the review surface where a human labels traces PASS/FAIL with a critique |
calibrated-agent-evaluation cluster |
check failure-mode saturation and taxonomy coverage; draft modes with --embedder |
calibrated-agent-evaluation calibrate |
certify a judge against the human gold in the store (the first trust gate) |
calibrated-agent-evaluation bias |
run a certified judge's bias battery (the second trust gate) |
calibrated-agent-evaluation judge |
score traces with a judge that has a certifying calibration and a CLEAR bias run |
calibrated-agent-evaluation score |
grade traces with a deterministic code scorer (exact, levenshtein, numeric, ...) |
calibrated-agent-evaluation run |
gate the stored verdicts, correct each judged mode's rate, record the run |
calibrated-agent-evaluation prefer |
import pairwise preferences on subjective dimensions and rank traces (tracked, never gated) |
calibrated-agent-evaluation improve |
pre-register a fix's target, then accept or reject the fix against it |
calibrated-agent-evaluation dump |
write the store's traces and labels, and the local root's records, to a directory (--judges-only: just what another store needs to score with given judges) |
calibrated-agent-evaluation load |
read a dump directory back into the store and the local root's records |
calibrated-agent-evaluation plan |
size a sample before labeling it: --gate gives the traces per mode calibrated-agent-evaluation run needs |
calibrated-agent-evaluation results |
write every stored estimate (intervals, bounds, refusals) to one CSV table |
calibrated-agent-evaluation¶
Global options apply to every subcommand.
usage: calibrated-agent-evaluation [-h] [--version] [--backend {local,langfuse}] [--root PATH]
[--explain]
<command> ...
Calibrated Agent Evaluation: error analysis and judge calibration for AI agents.
positional arguments:
<command>
quickstart run the bundled sample corpus end-to-end, offline (no setup, no keys)
doctor report the install: Python, which optional extras are present, backend
reachability
ingest normalize trace dumps (OTLP / Inspect .eval / JSONL / parquet) into the
store; refuses lossy ones
capture run an instrumented in-process agent, capture its OTel trace, and ingest
it
dataset turn labeled failures into test scenarios; publish and pull them as
AgentCore dataset versions
audit audit an AgentCore A/B test or recommendation: its claim, its arms, a re-
test
online put a calibrated Calibrated Agent Evaluation judge into AgentCore online
evaluation
probe black-box probe an agent endpoint and store what it shows (needs
--authorized)
define build the discovery frame (a covering array) from a taxonomy
generate synthesize frame-tagged inputs; with --run, route them through your agent
label open the review surface where a human labels traces PASS/FAIL with a
critique
cluster check failure-mode saturation and taxonomy coverage; draft modes with
--embedder
calibrate certify a judge against the human gold in the store (the first trust gate)
bias run a certified judge's bias battery (the second trust gate)
judge score traces with a judge that has a certifying calibration and a CLEAR
bias run
score grade traces with a deterministic code scorer (exact, levenshtein,
numeric, ...)
run gate the stored verdicts, correct each judged mode's rate, record the run
prefer import pairwise preferences on subjective dimensions and rank traces
(tracked, never gated)
improve pre-register a fix's target, then accept or reject the fix against it
dump write the store's traces and labels, and the local root's records, to a
directory (--judges-only: just what another store needs to score with
given judges)
load read a dump directory back into the store and the local root's records
plan size a sample before labeling it: --gate gives the traces per mode
`calibrated-agent-evaluation run` needs
results write every stored estimate (intervals, bounds, refusals) to one CSV table
options:
-h, --help show this help message and exit
--version show program's version number and exit
--backend {local,langfuse}
trace-store backend (default: local, offline)
--root PATH local root for the trace store and the stage records (default:
$CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, else ./.calibrated-agent-
evaluation; quickstart defaults to a temp dir)
--explain after the plain summary, print the statistics behind it (bounds, methods,
thresholds, record paths); CALIBRATED_AGENT_EVALUATION_EXPLAIN=1 turns it
on for every command