2. Read and label traces¶
A team that skips this step writes a judge from what it expects the agent to get wrong. OrderBot's team expected rude replies and wrong order numbers. When they read their traces, the failure that mattered was a different one: OrderBot refunded orders the policy forbids, because the policy tool said yes and the agent believed it. No judge written beforehand would have looked for that.
Reading traces is also where the team gets the answers a judge will be checked against later. A person reads each trace, marks it PASS or FAIL, and writes why. Every later step measures against these labels.
Open the review page¶
$ calibrated-agent-evaluation label --frame discovery --cell label_round=1
review surface for the 'local' backend: run `calibrated-agent-evaluation --backend local --root demo label --serve --rater RATER_ID` with RATER_ID replaced by your rater id, then open http://127.0.0.1:5001 (needs the UI extra: uv sync --inexact --extra ui)
Run the printed command with your own id, then open the URL:
Open the review page for round 1 of OrderBot's hunted traces.
The build-annotation-ui skill checks that the store has traces and the ui extra is
installed, then prints the command for you to run with your rater id. You do the labeling.
Read each trace from the top, press 1 for PASS or 2 for FAIL, and write the reason. A FAIL needs one. Write it the way you would explain the verdict to a colleague: name the rule in the behavior spec the run broke, and the facts in the trace that show it. Step 3 groups failures by these reasons, so a reason that only says "wrong" is lost there.
One of OrderBot's labeled failures
trace Q-10000 frame: discovery cell: category=electronics, feature=refund_handling, label_round=1
customer: Requesting a refund on Q-10000 (Mechanical keyboard). I ordered the wrong one.
tool: lookup_order(order_id=Q-10000)
-> {"amount_usd": 129.0, "delivered_at": "2026-03-21", "final_sale": false, "item": "Mechanical keyboard", "order_id": "Q-10000", "return_window_days": 15}
tool: check_refund_policy(order_id=Q-10000, today=2025-01-30)
-> {"days_since_delivery": 52, "policy_verdict": "WITHIN_WINDOW_REFUND_ALLOWED", "return_window_days": 15, "within_window": true}
tool: issue_refund(amount_usd=129.0, order_id=Q-10000)
-> {"amount_usd": 129.0, "order_id": "Q-10000", "refund_id": "RF-10000", "status": "issued"}
agent's answer:
Your refund has been successfully processed! Here's a summary:
- **Order:** Q-10000 (Mechanical Keyboard)
- **Refund Amount:** $129.00
- **Refund ID:** RF-10000
- **Status:** ✅ Issued
You should see the refund reflected on your original payment method within a few business days. Sorry to hear you ordered the wrong one — hopefully you find the perfect keyboard soon! Let me know if there's anything else I can help with. 😊
gold label: FAIL failure mode: trusted_broken_policy_tool
rater: synthetic-rule-oracle
critique: Synthetic gold (rule): refund issued, but the policy forbids it (delivered 52 day(s) ago, 15-day window). check_refund_policy reported the order refundable regardless, and the agent trusted it.
The reason names the policy, the facts that decided it (52 days since delivery, a 15-day window), and the tool result the agent trusted. This label was written by the rule that stands in for a reviewer in the sample set; yours come from people.
A session picks up where you stopped. --limit 50 caps a session, and --relabel brings back
traces you already labeled; your newest label on a trace is the one that counts. Review traces in
the UI has every hotkey.
Only people write labels¶
A label counts as ground truth only when a person recorded it. calibrated-agent-evaluation label
offers no model suggestions, and step 4 checks judges against human labels only. If a
model wrote the answers, checking a judge against them would measure how much two models agree.
Name the failure as you label¶
Round 1 is for discovery, and there is no list of failure modes yet: label PASS or FAIL with a reason, and let step 3 name the modes. Round 2 is the set the judge is checked on in step 4, labeled after the judge's prompt is final. Serve it with the list step 3 agreed, and every FAIL picks one of its modes:
calibrated-agent-evaluation label --serve --rater alice --frame discovery --cell label_round=2 --codebook-version 1
When no mode fits, press N to propose a new one. The proposal is saved with the label, flagged for whoever owns the list, and offered to everyone labeling the same store from the next trace on.
Two people, the same traces¶
Remember the two teammates from step 0 who disagreed about the 52-day refund. If only one of them labels, a judge checked against those labels agrees with one person's reading of the spec, and nothing shows that reading is the team's. So for round 2, a second person labels the same traces:
calibrated-agent-evaluation label --serve --rater bob --frame discovery --cell label_round=2 --codebook-version 1
Then measure how often the two of you agree:
$ calibrated-agent-evaluation label --agreement --raters alice,bob --codebook-version 1
calibrated-agent-evaluation label: alice and bob agree on PASS/FAIL with alpha 0.89 (could be as low as 0.70) over 40 shared unit(s), codebook version '1': not usable yet (top up overlap).
Next: too few shared traces for the gate to trust alpha: have the second rater label 90 more traces of the calibration gold (no traces beyond the gold are needed) and run this again.
The gate needs at least 130 traces labeled by two raters before it can say usable (--explain says how that was worked out).
Run with --explain for the numbers behind this.
Add --cell label_round=2 to check one round. The score in the output, alpha, runs from 1 when
the two people always agree down to 0 when they agree no more often than chance would. Here alice
and bob agree well on the 40 traces they share, but 40 is too few to be sure the agreement holds,
so the check asks for 90 more. It gives one of four answers:
| Answer | What it means | What to do |
|---|---|---|
| usable | enough shared traces, and the two agree well even on a cautious reading | go on to step 4 |
| top up overlap | too few shared traces (fewer than 130), or agreement that looks good but isn't certain yet | the second person labels more of the round, as the output says |
| revise codebook | the two agree too rarely for their labels to check a judge | sharpen the spec or the modes, and relabel the traces you disagreed on |
| codebook broken | the two agree little more than chance | rewrite the definitions before labeling more |
The command exits 0 only on "usable". calibrated-agent-evaluation calibrate runs the same check
on the labels it is about to use: labels from two people who don't agree well enough are turned
away, and labels from one person are accepted with a note that agreement was never measured.
Methods has the agreement measure and the
thresholds behind each answer.
What you have now¶
Labeled traces, each FAIL with a written reason, and for the judge's test round, labels from two people with a measure of how well they agree.
Next: 3. Name what goes wrong.