Rank subjective qualities¶
Some qualities of an answer have no PASS or FAIL, such as whether a refund denial reads as
courteous. You declare which parts of a criterion are objective (judged PASS or FAIL, checked, and
gated with calibrated-agent-evaluation run) and which are subjective (compared two traces at a
time and ranked). Rankings are tracked over time; no command fails a build on them.
Declare the split¶
The split is one JSON file per criterion, passed with --atoms (the toolkit calls it an atom set):
{
"criterion": "claim decision letter",
"objective": [
{"name": "cites_exclusion", "description": "names the policy exclusion the decision rests on"}
],
"subjective": [
{"name": "tone", "description": "courteous and plain to a claimant who is not getting paid"}
]
}
objective lists the pass/fail parts, subjective the dimensions you compare on. A name may
appear in only one of the two lists, and unknown keys are rejected.
Import preferences¶
Each line of a pairs file says which of two stored traces was better on one subjective dimension:
{"trace_id_a": "CLM-4473", "trace_id_b": "CLM-4471", "dimension": "tone", "winner": "b", "judge_kind": "human"}
winner is a or b; judge_kind is human, llm-judge or code. Import the file:
$ calibrated-agent-evaluation prefer --atoms atoms.json --pairs pairs.jsonl
Imported 9 preference(s) on 'tone' for 'claim decision letter'.
Next: rank them with `calibrated-agent-evaluation prefer --rank tone`.
Run with --explain for the numbers behind this.
The preferences go to <root>/records/preference/ and travel with calibrated-agent-evaluation
dump and calibrated-agent-evaluation load. Importing the same file again adds nothing.
Every line is checked against the --atoms file, and one bad line rejects the whole file. A preference
on an objective part is the common mistake:
$ calibrated-agent-evaluation prefer --atoms atoms.json --pairs pairs-objective.jsonl
calibrated-agent-evaluation prefer: 1 line(s) of the pairs file docs/site/examples/guides/prefer-pairs-objective.jsonl were refused, so nothing was imported: line 2: Dimension 'cites_exclusion' is objective; use binary PASS/FAIL labels, not preferences
A trace compared with itself, a dimension the --atoms file does not declare, and a trace id the store
does not hold are rejected the same way.
Record preferences in the review UI¶
calibrated-agent-evaluation label --serve --pairwise --atoms atoms.json --rater <your-id> --cell feature=refund_handling
The A/B page shows two traces side by side and asks which is better on one dimension. Press a (or
1) for the left trace, b (or 2) for the right, s to skip a pair you cannot call. The queue
is every pair of traces in scope on every subjective dimension, in a fixed shuffled order with the
sides shuffled too. Your choices are stored as human preferences in the same records
calibrated-agent-evaluation prefer --rank reads, and a restarted session leaves out the pairs you
already did.
Narrow the scope with --cell, --frame or --limit so both traces answer the same kind of
input. Like the labeling page, it is served on this machine only (see
Review traces in the UI).
Rank a dimension¶
$ calibrated-agent-evaluation prefer --rank tone
'tone' for 'claim decision letter', human preferences (9 comparison(s)), tracked, never gated:
1. CLM-4471 1677 (95% interval 1388 to 1965), 5 comparison(s)
2. CLM-4475 1490 (95% interval 1216 to 1765), 4 comparison(s)
3. CLM-4473 1462 (95% interval 1213 to 1712), 5 comparison(s)
4. CLM-4477 1371 (95% interval 1067 to 1675), 4 comparison(s)
These ratings are tracked, never gated. Gate on the objective atoms with `calibrated-agent-evaluation run`; collect more comparisons to narrow the intervals.
Run with --explain for the numbers behind this.
Each trace gets a rating from all the stored comparisons on the dimension, with a range. The scale reads like chess ratings: a 400-point gap means the higher trace wins about 10 times out of 11. When two traces' ranges overlap, the comparisons don't separate them yet; collect more.
Preferences from different criteria or different judge kinds are ranked separately, so a human's choices and an LLM judge's never share a ranking.
--explain adds the method and a running chess-style rating for comparison:
$ calibrated-agent-evaluation prefer --rank tone --explain
'tone' for 'claim decision letter', human preferences (9 comparison(s)), tracked, never gated:
1. CLM-4471 1677 (95% interval 1388 to 1965), 5 comparison(s)
2. CLM-4475 1490 (95% interval 1216 to 1765), 4 comparison(s)
3. CLM-4473 1462 (95% interval 1213 to 1712), 5 comparison(s)
4. CLM-4477 1371 (95% interval 1067 to 1675), 4 comparison(s)
These ratings are tracked, never gated. Gate on the objective atoms with `calibrated-agent-evaluation run`; collect more comparisons to narrow the intervals.
'claim decision letter' / human: Bradley-Terry maximum likelihood (bradley_terry_mle_wald), 95% Wald intervals from the Fisher information under a sum-to-zero constraint, on the Elo scale 1500 + 400 * log-strength / ln 10 (a 400-point gap is 10:1 odds); order-invariant
sequential Elo, display only (K=32, start 1500, in stored order; it changes with the order and has no interval):
CLM-4471: 1540 (5 comparison(s))
CLM-4475: 1502 (4 comparison(s))
CLM-4473: 1483 (5 comparison(s))
CLM-4477: 1474 (4 comparison(s))
The running rating is for display only: it changes with the order the preferences were stored in and has no range. The fitted rating is the one stored. Methods names both.
A trace that never lost is not ranked
When a trace won (or lost) every comparison it was in, there is no finite rating for it.
calibrated-agent-evaluation prefer --rank stops with exit 1 for that group and names it.
Compare it against more traces.
Next¶
- Gate the objective parts the usual way: Put a judge on probation, then Gate your pull requests.
- The flags:
calibrated-agent-evaluation preferandcalibrated-agent-evaluation label.