Skip to content

calibrated-agent-evaluation cluster

check failure-mode saturation and taxonomy coverage; draft modes with --embedder

usage: calibrated-agent-evaluation cluster [-h] [--trace TRACE_ID] [--taxonomy FILE]
                                           [--embedder {bedrock}] [--embed-model ID]
                                           [--threshold THRESHOLD] [--out FILE] [--ratify MAP]
                                           [--backend {local,langfuse}] [--root PATH] [--explain]

Over the human FAIL labels: whether the failure modes are saturated (Good-Turing coverage lower
bound), which taxonomy cells have no failure yet, and with an embedder a draft codebook for a
human to ratify.

options:
  -h, --help            show this help message and exit
  --trace TRACE_ID      operate on one stored trace (default: every stored trace); an unknown id
                        is refused
  --taxonomy FILE       grid for the coverage report (default: the stored traces' taxonomy cells)
  --embedder {bedrock}  embed the FAIL critiques to draft failure modes (Bedrock Titan, needs
                        [bedrock]; a paid call, so opt-in)
  --embed-model ID      embedding model id for --embedder bedrock (default: Titan text v2)
  --threshold THRESHOLD
                        Ward-linkage distance at which draft clusters are cut (default: 1.0)
  --out FILE            write the draft codebook as JSON (with --embedder)
  --ratify MAP          stamp human-ratified failure modes onto the stored FAIL labels from a JSON
                        map {"codebook_version": ..., "ratified_by": ..., "assign": [{"trace_id":
                        ..., "mode": ...}]} and store the codebook
  --backend {local,langfuse}
                        trace-store backend (default: local, offline)
  --root PATH           local root for the trace store and the stage records (default:
                        $CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, else ./.calibrated-agent-
                        evaluation; quickstart defaults to a temp dir)
  --explain             after the plain summary, print the statistics behind it (bounds, methods,
                        thresholds, record paths); CALIBRATED_AGENT_EVALUATION_EXPLAIN=1 turns it
                        on for every command