Skip to content

calibrated-agent-evaluation plan

size a sample before labeling it: --gate gives the traces per mode calibrated-agent-evaluation run needs

usage: calibrated-agent-evaluation plan [-h] [--gate] [--threshold T] --expected-pass P
                                        [--backend {local,langfuse}] [--root PATH] [--explain]

With --gate, the number of production-representative traces each failure mode needs for
`calibrated-agent-evaluation run --threshold T` to pass a suite expected to pass P of the time, at
that size and every larger one.

options:
  -h, --help            show this help message and exit
  --gate                size the sample `calibrated-agent-evaluation run` gates on: traces per
                        failure mode
  --threshold T         the gate's threshold, as passed to `calibrated-agent-evaluation run
                        --threshold` (default: 0.80)
  --expected-pass P     the pass rate you expect each mode to have, e.g. 0.95
  --backend {local,langfuse}
                        trace-store backend (default: local, offline)
  --root PATH           local root for the trace store and the stage records (default:
                        $CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, else ./.calibrated-agent-
                        evaluation; quickstart defaults to a temp dir)
  --explain             after the plain summary, print the statistics behind it (bounds, methods,
                        thresholds, record paths); CALIBRATED_AGENT_EVALUATION_EXPLAIN=1 turns it
                        on for every command