calibrated-agent-evaluation prefer¶
import pairwise preferences on subjective dimensions and rank traces (tracked, never gated)
usage: calibrated-agent-evaluation prefer [-h] [--atoms FILE] [--pairs FILE] [--rank DIMENSION]
[--backend {local,langfuse}] [--root PATH] [--explain]
--atoms FILE --pairs FILE imports preferences (trace A or B was better on a subjective dimension),
each checked against the atom set: an objective or undeclared dimension refuses the file. --rank
DIMENSION fits Bradley-Terry ratings with 95% intervals over the stored preferences, per criterion
and judge kind, never pooled. The ratings are tracked, never gated; sequential Elo appears only
under --explain.
options:
-h, --help show this help message and exit
--atoms FILE the atom set (JSON) that declares which dimensions are objective and which
subjective; required with --pairs, narrows --rank to its criterion
--pairs FILE import preferences from a JSON Lines file (trace_id_a, trace_id_b,
dimension, winner a|b, judge_kind); one bad line refuses the file
--rank DIMENSION rank the stored preferences on one subjective dimension (Bradley-Terry
with 95% intervals, tracked, never gated; sequential Elo under --explain)
--backend {local,langfuse}
trace-store backend (default: local, offline)
--root PATH local root for the trace store and the stage records (default:
$CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, else ./.calibrated-agent-
evaluation; quickstart defaults to a temp dir)
--explain after the plain summary, print the statistics behind it (bounds, methods,
thresholds, record paths); CALIBRATED_AGENT_EVALUATION_EXPLAIN=1 turns it
on for every command