calibrated-agent-evaluation quickstart¶
run the bundled sample corpus end-to-end, offline (no setup, no keys)
usage: calibrated-agent-evaluation quickstart [-h] [--certify] [--force]
[--backend {local,langfuse}] [--root PATH]
[--explain]
options:
-h, --help show this help message and exit
--certify seed a set big enough to certify a judge (real agent runs, synthetic gold)
and write a judge kit whose recorded replies take calibrate, bias, judge
and run offline
--force seed into a store directory that already holds data (the sample's traces
and gold labels are added to what is there, and the judge kit files are
overwritten)
--backend {local,langfuse}
trace-store backend (default: local, offline)
--root PATH local root for the trace store and the stage records (default:
$CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, else ./.calibrated-agent-
evaluation; quickstart defaults to a temp dir)
--explain after the plain summary, print the statistics behind it (bounds, methods,
thresholds, record paths); CALIBRATED_AGENT_EVALUATION_EXPLAIN=1 turns it
on for every command