Skip to content

calibrated-agent-evaluation prefer

import pairwise preferences on subjective dimensions and rank traces (tracked, never gated)

usage: calibrated-agent-evaluation prefer [-h] [--atoms FILE] [--pairs FILE] [--rank DIMENSION]
                                          [--backend {local,langfuse}] [--root PATH] [--explain]

--atoms FILE --pairs FILE imports preferences (trace A or B was better on a subjective dimension),
each checked against the atom set: an objective or undeclared dimension refuses the file. --rank
DIMENSION fits Bradley-Terry ratings with 95% intervals over the stored preferences, per criterion
and judge kind, never pooled. The ratings are tracked, never gated; sequential Elo appears only
under --explain.

options:
  -h, --help            show this help message and exit
  --atoms FILE          the atom set (JSON) that declares which dimensions are objective and which
                        subjective; required with --pairs, narrows --rank to its criterion
  --pairs FILE          import preferences from a JSON Lines file (trace_id_a, trace_id_b,
                        dimension, winner a|b, judge_kind); one bad line refuses the file
  --rank DIMENSION      rank the stored preferences on one subjective dimension (Bradley-Terry
                        with 95% intervals, tracked, never gated; sequential Elo under --explain)
  --backend {local,langfuse}
                        trace-store backend (default: local, offline)
  --root PATH           local root for the trace store and the stage records (default:
                        $CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, else ./.calibrated-agent-
                        evaluation; quickstart defaults to a temp dir)
  --explain             after the plain summary, print the statistics behind it (bounds, methods,
                        thresholds, record paths); CALIBRATED_AGENT_EVALUATION_EXPLAIN=1 turns it
                        on for every command