Skip to content

calibrated-agent-evaluation quickstart

run the bundled sample corpus end-to-end, offline (no setup, no keys)

usage: calibrated-agent-evaluation quickstart [-h] [--certify] [--force]
                                              [--backend {local,langfuse}] [--root PATH]
                                              [--explain]

options:
  -h, --help            show this help message and exit
  --certify             seed a set big enough to certify a judge (real agent runs, synthetic gold)
                        and write a judge kit whose recorded replies take calibrate, bias, judge
                        and run offline
  --force               seed into a store directory that already holds data (the sample's traces
                        and gold labels are added to what is there, and the judge kit files are
                        overwritten)
  --backend {local,langfuse}
                        trace-store backend (default: local, offline)
  --root PATH           local root for the trace store and the stage records (default:
                        $CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, else ./.calibrated-agent-
                        evaluation; quickstart defaults to a temp dir)
  --explain             after the plain summary, print the statistics behind it (bounds, methods,
                        thresholds, record paths); CALIBRATED_AGENT_EVALUATION_EXPLAIN=1 turns it
                        on for every command