calibrated-agent-evaluation improve¶
pre-register a fix's target, then accept or reject the fix against it
usage: calibrated-agent-evaluation improve [-h] [--register METRIC] [--min-reduction X]
[--code-ref REF] [--crew] [--evaluate METRIC]
[--before-root PATH] [--after-root PATH]
[--development FILE] [--verdicts {judge,scorer,human}]
[--crew-file CREW.json] [--touched-agent NAME]
[--request FILE] [--improve-id ID]
[--backend {local,langfuse}] [--root PATH] [--explain]
--register freezes a target metric and its minimum fail-rate reduction before the fix; --evaluate
pairs the verdicts of the run before the fix and the run after it (two local stores) per trace,
guards every other failure mode, and records the verdict. --verdicts picks the LLM-judge verdicts
(default), the code-scorer verdicts or the human labels. For a crew metric (--register --crew),
--crew-file and --touched-agent add a guard slice per other agent and per handoff and check that
the failures did not just move. --request takes the pairs from a file instead.
options:
-h, --help show this help message and exit
--register METRIC pre-register the failure mode a fix targets, before the fix exists
--min-reduction X with --register: the absolute fail-rate drop the fix must show, in (0, 1)
--code-ref REF with --register: the commit of the agent the target is frozen against
--crew with --register: the metric is a crew's workflow outcome rate
--evaluate METRIC evaluate a fix for a pre-registered metric from the verdicts in --before-
root and --after-root, paired by trace id
--before-root PATH with --evaluate: the local store of the run before the fix
--after-root PATH with --evaluate: the local store of the same inputs run after the fix
--development FILE with --evaluate: the trace ids the fix was developed on, one per line
(they must not be among the paired traces)
--verdicts {judge,scorer,human}
with --evaluate: pair the LLM-judge verdicts (default), the code-scorer
verdicts or the human labels, never pooled; with human, a trace whose only
label is a FAIL for another mode is left out of that mode's pairs (the
target's and each guard's) and named
--crew-file CREW.json
with --evaluate: the crew's agents and handoffs, for a metric registered
with --crew; each other agent and each handoff becomes a guard slice
--touched-agent NAME with --crew-file: the agent the fix changed, whose failures the relocation
check follows
--request FILE evaluate a fix from a JSON file of paired before/after verdicts and guard
slices measured elsewhere (the fallback to --evaluate)
--improve-id ID id for the improve record (default: a timestamped one)
--backend {local,langfuse}
trace-store backend (default: local, offline)
--root PATH local root for the trace store and the stage records (default:
$CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, else ./.calibrated-agent-
evaluation; quickstart defaults to a temp dir)
--explain after the plain summary, print the statistics behind it (bounds, methods,
thresholds, record paths); CALIBRATED_AGENT_EVALUATION_EXPLAIN=1 turns it
on for every command