Skip to content

calibrated-agent-evaluation improve

pre-register a fix's target, then accept or reject the fix against it

usage: calibrated-agent-evaluation improve [-h] [--register METRIC] [--min-reduction X]
                                           [--code-ref REF] [--crew] [--evaluate METRIC]
                                           [--before-root PATH] [--after-root PATH]
                                           [--development FILE] [--verdicts {judge,scorer,human}]
                                           [--crew-file CREW.json] [--touched-agent NAME]
                                           [--request FILE] [--improve-id ID]
                                           [--backend {local,langfuse}] [--root PATH] [--explain]

--register freezes a target metric and its minimum fail-rate reduction before the fix; --evaluate
pairs the verdicts of the run before the fix and the run after it (two local stores) per trace,
guards every other failure mode, and records the verdict. --verdicts picks the LLM-judge verdicts
(default), the code-scorer verdicts or the human labels. For a crew metric (--register --crew),
--crew-file and --touched-agent add a guard slice per other agent and per handoff and check that
the failures did not just move. --request takes the pairs from a file instead.

options:
  -h, --help            show this help message and exit
  --register METRIC     pre-register the failure mode a fix targets, before the fix exists
  --min-reduction X     with --register: the absolute fail-rate drop the fix must show, in (0, 1)
  --code-ref REF        with --register: the commit of the agent the target is frozen against
  --crew                with --register: the metric is a crew's workflow outcome rate
  --evaluate METRIC     evaluate a fix for a pre-registered metric from the verdicts in --before-
                        root and --after-root, paired by trace id
  --before-root PATH    with --evaluate: the local store of the run before the fix
  --after-root PATH     with --evaluate: the local store of the same inputs run after the fix
  --development FILE    with --evaluate: the trace ids the fix was developed on, one per line
                        (they must not be among the paired traces)
  --verdicts {judge,scorer,human}
                        with --evaluate: pair the LLM-judge verdicts (default), the code-scorer
                        verdicts or the human labels, never pooled; with human, a trace whose only
                        label is a FAIL for another mode is left out of that mode's pairs (the
                        target's and each guard's) and named
  --crew-file CREW.json
                        with --evaluate: the crew's agents and handoffs, for a metric registered
                        with --crew; each other agent and each handoff becomes a guard slice
  --touched-agent NAME  with --crew-file: the agent the fix changed, whose failures the relocation
                        check follows
  --request FILE        evaluate a fix from a JSON file of paired before/after verdicts and guard
                        slices measured elsewhere (the fallback to --evaluate)
  --improve-id ID       id for the improve record (default: a timestamped one)
  --backend {local,langfuse}
                        trace-store backend (default: local, offline)
  --root PATH           local root for the trace store and the stage records (default:
                        $CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, else ./.calibrated-agent-
                        evaluation; quickstart defaults to a temp dir)
  --explain             after the plain summary, print the statistics behind it (bounds, methods,
                        thresholds, record paths); CALIBRATED_AGENT_EVALUATION_EXPLAIN=1 turns it
                        on for every command