Skip to content

Rank subjective qualities

Some qualities of an answer have no PASS or FAIL, such as whether a refund denial reads as courteous. You declare which parts of a criterion are objective (judged PASS or FAIL, checked, and gated with calibrated-agent-evaluation run) and which are subjective (compared two traces at a time and ranked). Rankings are tracked over time; no command fails a build on them.

Declare the split

The split is one JSON file per criterion, passed with --atoms (the toolkit calls it an atom set):

{
  "criterion": "claim decision letter",
  "objective": [
    {"name": "cites_exclusion", "description": "names the policy exclusion the decision rests on"}
  ],
  "subjective": [
    {"name": "tone", "description": "courteous and plain to a claimant who is not getting paid"}
  ]
}

objective lists the pass/fail parts, subjective the dimensions you compare on. A name may appear in only one of the two lists, and unknown keys are rejected.

Import preferences

Each line of a pairs file says which of two stored traces was better on one subjective dimension:

{"trace_id_a": "CLM-4473", "trace_id_b": "CLM-4471", "dimension": "tone", "winner": "b", "judge_kind": "human"}

winner is a or b; judge_kind is human, llm-judge or code. Import the file:

$ calibrated-agent-evaluation prefer --atoms atoms.json --pairs pairs.jsonl
Imported 9 preference(s) on 'tone' for 'claim decision letter'.
Next: rank them with `calibrated-agent-evaluation prefer --rank tone`.
Run with --explain for the numbers behind this.

The preferences go to <root>/records/preference/ and travel with calibrated-agent-evaluation dump and calibrated-agent-evaluation load. Importing the same file again adds nothing.

Every line is checked against the --atoms file, and one bad line rejects the whole file. A preference on an objective part is the common mistake:

$ calibrated-agent-evaluation prefer --atoms atoms.json --pairs pairs-objective.jsonl
calibrated-agent-evaluation prefer: 1 line(s) of the pairs file docs/site/examples/guides/prefer-pairs-objective.jsonl were refused, so nothing was imported: line 2: Dimension 'cites_exclusion' is objective; use binary PASS/FAIL labels, not preferences

A trace compared with itself, a dimension the --atoms file does not declare, and a trace id the store does not hold are rejected the same way.

Record preferences in the review UI

calibrated-agent-evaluation label --serve --pairwise --atoms atoms.json --rater <your-id> --cell feature=refund_handling

The A/B page shows two traces side by side and asks which is better on one dimension. Press a (or 1) for the left trace, b (or 2) for the right, s to skip a pair you cannot call. The queue is every pair of traces in scope on every subjective dimension, in a fixed shuffled order with the sides shuffled too. Your choices are stored as human preferences in the same records calibrated-agent-evaluation prefer --rank reads, and a restarted session leaves out the pairs you already did.

Narrow the scope with --cell, --frame or --limit so both traces answer the same kind of input. Like the labeling page, it is served on this machine only (see Review traces in the UI).

Rank a dimension

$ calibrated-agent-evaluation prefer --rank tone
'tone' for 'claim decision letter', human preferences (9 comparison(s)), tracked, never gated:
  1. CLM-4471  1677 (95% interval 1388 to 1965), 5 comparison(s)
  2. CLM-4475  1490 (95% interval 1216 to 1765), 4 comparison(s)
  3. CLM-4473  1462 (95% interval 1213 to 1712), 5 comparison(s)
  4. CLM-4477  1371 (95% interval 1067 to 1675), 4 comparison(s)
These ratings are tracked, never gated. Gate on the objective atoms with `calibrated-agent-evaluation run`; collect more comparisons to narrow the intervals.
Run with --explain for the numbers behind this.

Each trace gets a rating from all the stored comparisons on the dimension, with a range. The scale reads like chess ratings: a 400-point gap means the higher trace wins about 10 times out of 11. When two traces' ranges overlap, the comparisons don't separate them yet; collect more.

Preferences from different criteria or different judge kinds are ranked separately, so a human's choices and an LLM judge's never share a ranking.

--explain adds the method and a running chess-style rating for comparison:

$ calibrated-agent-evaluation prefer --rank tone --explain
'tone' for 'claim decision letter', human preferences (9 comparison(s)), tracked, never gated:
  1. CLM-4471  1677 (95% interval 1388 to 1965), 5 comparison(s)
  2. CLM-4475  1490 (95% interval 1216 to 1765), 4 comparison(s)
  3. CLM-4473  1462 (95% interval 1213 to 1712), 5 comparison(s)
  4. CLM-4477  1371 (95% interval 1067 to 1675), 4 comparison(s)
These ratings are tracked, never gated. Gate on the objective atoms with `calibrated-agent-evaluation run`; collect more comparisons to narrow the intervals.

'claim decision letter' / human: Bradley-Terry maximum likelihood (bradley_terry_mle_wald), 95% Wald intervals from the Fisher information under a sum-to-zero constraint, on the Elo scale 1500 + 400 * log-strength / ln 10 (a 400-point gap is 10:1 odds); order-invariant
  sequential Elo, display only (K=32, start 1500, in stored order; it changes with the order and has no interval):
    CLM-4471: 1540 (5 comparison(s))
    CLM-4475: 1502 (4 comparison(s))
    CLM-4473: 1483 (5 comparison(s))
    CLM-4477: 1474 (4 comparison(s))

The running rating is for display only: it changes with the order the preferences were stored in and has no range. The fitted rating is the one stored. Methods names both.

A trace that never lost is not ranked

When a trace won (or lost) every comparison it was in, there is no finite rating for it. calibrated-agent-evaluation prefer --rank stops with exit 1 for that group and names it. Compare it against more traces.

Next