Skip to content

calibrated-agent-evaluation define

build the discovery frame (a covering array) from a taxonomy

usage: calibrated-agent-evaluation define [-h] [--trace TRACE_ID] [--taxonomy FILE]
                                          [--strength STRENGTH] [--out FILE]
                                          [--backend {local,langfuse}] [--root PATH] [--explain]

Turn a taxonomy (a --taxonomy file, or the stored traces' cells) into a discovery frame in which
every pair of dimension values appears, and write it with --out.

options:
  -h, --help            show this help message and exit
  --trace TRACE_ID      operate on one stored trace (default: every stored trace); an unknown id
                        is refused
  --taxonomy FILE       JSON object of dimension -> list of values (default: the stored traces'
                        taxonomy cells)
  --strength STRENGTH   cover every combination of this many dimensions (default: 2, pairwise)
  --out FILE            write the frame as JSON, for `calibrated-agent-evaluation generate
                        --frame`
  --backend {local,langfuse}
                        trace-store backend (default: local, offline)
  --root PATH           local root for the trace store and the stage records (default:
                        $CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, else ./.calibrated-agent-
                        evaluation; quickstart defaults to a temp dir)
  --explain             after the plain summary, print the statistics behind it (bounds, methods,
                        thresholds, record paths); CALIBRATED_AGENT_EVALUATION_EXPLAIN=1 turns it
                        on for every command