calibrated-agent-evaluation cluster¶
check failure-mode saturation and taxonomy coverage; draft modes with --embedder
usage: calibrated-agent-evaluation cluster [-h] [--trace TRACE_ID] [--taxonomy FILE]
[--embedder {bedrock}] [--embed-model ID]
[--threshold THRESHOLD] [--out FILE] [--ratify MAP]
[--backend {local,langfuse}] [--root PATH] [--explain]
Over the human FAIL labels: whether the failure modes are saturated (Good-Turing coverage lower
bound), which taxonomy cells have no failure yet, and with an embedder a draft codebook for a
human to ratify.
options:
-h, --help show this help message and exit
--trace TRACE_ID operate on one stored trace (default: every stored trace); an unknown id
is refused
--taxonomy FILE grid for the coverage report (default: the stored traces' taxonomy cells)
--embedder {bedrock} embed the FAIL critiques to draft failure modes (Bedrock Titan, needs
[bedrock]; a paid call, so opt-in)
--embed-model ID embedding model id for --embedder bedrock (default: Titan text v2)
--threshold THRESHOLD
Ward-linkage distance at which draft clusters are cut (default: 1.0)
--out FILE write the draft codebook as JSON (with --embedder)
--ratify MAP stamp human-ratified failure modes onto the stored FAIL labels from a JSON
map {"codebook_version": ..., "ratified_by": ..., "assign": [{"trace_id":
..., "mode": ...}]} and store the codebook
--backend {local,langfuse}
trace-store backend (default: local, offline)
--root PATH local root for the trace store and the stage records (default:
$CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, else ./.calibrated-agent-
evaluation; quickstart defaults to a temp dir)
--explain after the plain summary, print the statistics behind it (bounds, methods,
thresholds, record paths); CALIBRATED_AGENT_EVALUATION_EXPLAIN=1 turns it
on for every command