Skip to content

CLI reference

The calibrated-agent-evaluation command runs the whole evaluation pipeline. These pages are generated from the argparse parser when the docs are built, so they always match the installed CLI.

Command What it does
calibrated-agent-evaluation quickstart run the bundled sample corpus end-to-end, offline (no setup, no keys)
calibrated-agent-evaluation doctor report the install: Python, which optional extras are present, backend reachability
calibrated-agent-evaluation ingest normalize trace dumps (OTLP / Inspect .eval / JSONL / parquet) into the store; refuses lossy ones
calibrated-agent-evaluation capture run an instrumented in-process agent, capture its OTel trace, and ingest it
calibrated-agent-evaluation dataset turn labeled failures into test scenarios; publish and pull them as AgentCore dataset versions
calibrated-agent-evaluation audit audit an AgentCore A/B test or recommendation: its claim, its arms, a re-test
calibrated-agent-evaluation online put a calibrated Calibrated Agent Evaluation judge into AgentCore online evaluation
calibrated-agent-evaluation probe black-box probe an agent endpoint and store what it shows (needs --authorized)
calibrated-agent-evaluation define build the discovery frame (a covering array) from a taxonomy
calibrated-agent-evaluation generate synthesize frame-tagged inputs; with --run, route them through your agent
calibrated-agent-evaluation label open the review surface where a human labels traces PASS/FAIL with a critique
calibrated-agent-evaluation cluster check failure-mode saturation and taxonomy coverage; draft modes with --embedder
calibrated-agent-evaluation calibrate certify a judge against the human gold in the store (the first trust gate)
calibrated-agent-evaluation bias run a certified judge's bias battery (the second trust gate)
calibrated-agent-evaluation judge score traces with a judge that has a certifying calibration and a CLEAR bias run
calibrated-agent-evaluation score grade traces with a deterministic code scorer (exact, levenshtein, numeric, ...)
calibrated-agent-evaluation run gate the stored verdicts, correct each judged mode's rate, record the run
calibrated-agent-evaluation prefer import pairwise preferences on subjective dimensions and rank traces (tracked, never gated)
calibrated-agent-evaluation improve pre-register a fix's target, then accept or reject the fix against it
calibrated-agent-evaluation dump write the store's traces and labels, and the local root's records, to a directory (--judges-only: just what another store needs to score with given judges)
calibrated-agent-evaluation load read a dump directory back into the store and the local root's records
calibrated-agent-evaluation plan size a sample before labeling it: --gate gives the traces per mode calibrated-agent-evaluation run needs
calibrated-agent-evaluation results write every stored estimate (intervals, bounds, refusals) to one CSV table

calibrated-agent-evaluation

Global options apply to every subcommand.

usage: calibrated-agent-evaluation [-h] [--version] [--backend {local,langfuse}] [--root PATH]
                                   [--explain]
                                   <command> ...

Calibrated Agent Evaluation: error analysis and judge calibration for AI agents.

positional arguments:
  <command>
    quickstart          run the bundled sample corpus end-to-end, offline (no setup, no keys)
    doctor              report the install: Python, which optional extras are present, backend
                        reachability
    ingest              normalize trace dumps (OTLP / Inspect .eval / JSONL / parquet) into the
                        store; refuses lossy ones
    capture             run an instrumented in-process agent, capture its OTel trace, and ingest
                        it
    dataset             turn labeled failures into test scenarios; publish and pull them as
                        AgentCore dataset versions
    audit               audit an AgentCore A/B test or recommendation: its claim, its arms, a re-
                        test
    online              put a calibrated Calibrated Agent Evaluation judge into AgentCore online
                        evaluation
    probe               black-box probe an agent endpoint and store what it shows (needs
                        --authorized)
    define              build the discovery frame (a covering array) from a taxonomy
    generate            synthesize frame-tagged inputs; with --run, route them through your agent
    label               open the review surface where a human labels traces PASS/FAIL with a
                        critique
    cluster             check failure-mode saturation and taxonomy coverage; draft modes with
                        --embedder
    calibrate           certify a judge against the human gold in the store (the first trust gate)
    bias                run a certified judge's bias battery (the second trust gate)
    judge               score traces with a judge that has a certifying calibration and a CLEAR
                        bias run
    score               grade traces with a deterministic code scorer (exact, levenshtein,
                        numeric, ...)
    run                 gate the stored verdicts, correct each judged mode's rate, record the run
    prefer              import pairwise preferences on subjective dimensions and rank traces
                        (tracked, never gated)
    improve             pre-register a fix's target, then accept or reject the fix against it
    dump                write the store's traces and labels, and the local root's records, to a
                        directory (--judges-only: just what another store needs to score with
                        given judges)
    load                read a dump directory back into the store and the local root's records
    plan                size a sample before labeling it: --gate gives the traces per mode
                        `calibrated-agent-evaluation run` needs
    results             write every stored estimate (intervals, bounds, refusals) to one CSV table

options:
  -h, --help            show this help message and exit
  --version             show program's version number and exit
  --backend {local,langfuse}
                        trace-store backend (default: local, offline)
  --root PATH           local root for the trace store and the stage records (default:
                        $CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, else ./.calibrated-agent-
                        evaluation; quickstart defaults to a temp dir)
  --explain             after the plain summary, print the statistics behind it (bounds, methods,
                        thresholds, record paths); CALIBRATED_AGENT_EVALUATION_EXPLAIN=1 turns it
                        on for every command