calibrated-agent-evaluation plan¶
size a sample before labeling it: --gate gives the traces per mode calibrated-agent-evaluation run needs
usage: calibrated-agent-evaluation plan [-h] [--gate] [--threshold T] --expected-pass P
[--backend {local,langfuse}] [--root PATH] [--explain]
With --gate, the number of production-representative traces each failure mode needs for
`calibrated-agent-evaluation run --threshold T` to pass a suite expected to pass P of the time, at
that size and every larger one.
options:
-h, --help show this help message and exit
--gate size the sample `calibrated-agent-evaluation run` gates on: traces per
failure mode
--threshold T the gate's threshold, as passed to `calibrated-agent-evaluation run
--threshold` (default: 0.80)
--expected-pass P the pass rate you expect each mode to have, e.g. 0.95
--backend {local,langfuse}
trace-store backend (default: local, offline)
--root PATH local root for the trace store and the stage records (default:
$CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, else ./.calibrated-agent-
evaluation; quickstart defaults to a temp dir)
--explain after the plain summary, print the statistics behind it (bounds, methods,
thresholds, record paths); CALIBRATED_AGENT_EVALUATION_EXPLAIN=1 turns it
on for every command