calibrated-agent-evaluation label¶
open the review surface where a human labels traces PASS/FAIL with a critique
usage: calibrated-agent-evaluation label [-h] [--trace TRACE_ID] [--serve] [--rater RATER_ID]
[--frame {discovery,production-representative}]
[--cell KEY=VALUE] [--limit LIMIT] [--relabel]
[--session-log PATH] [--pairwise] [--atoms FILE]
[--host HOST] [--port PORT] [--codebook-version V]
[--codebook FILE] [--agreement] [--raters A,B[,C]]
[--axis {outcome,failure_mode}]
[--backend {local,langfuse}] [--root PATH] [--explain]
Print the command that serves the review surface, or serve it with --serve. Every label carries
the rater id; --trace/--frame/--cell/--limit narrow the work list.
options:
-h, --help show this help message and exit
--trace TRACE_ID operate on one stored trace (default: every stored trace); an unknown id
is refused
--serve launch the review server (needs [ui]); omit to print the launch command +
URL
--rater RATER_ID your rater id, stamped on every label you write (required with --serve)
--frame {discovery,production-representative}
review only traces in this frame
--cell KEY=VALUE review only traces whose taxonomy cell has KEY=VALUE (repeatable)
--limit LIMIT review at most this many traces still to label
--relabel include traces you already labeled (default: resume, skipping them)
--session-log PATH JSONL file that gets one line per verdict with its time on trace (default:
<root>/review-sessions.jsonl)
--pairwise with --serve: serve the A/B page instead, where you pick the better of two
traces on each subjective dimension of --atoms; choices go to
<root>/records/preference/
--atoms FILE with --pairwise: the atom set whose subjective dimensions you compare on
--host HOST loopback interface the review server binds (default: 127.0.0.1; ::1 and
localhost also work). Other addresses are refused: label from another
machine over an SSH port-forward
--port PORT port the review server listens on (default: 5001)
--codebook-version V codebook version stamped on every label you write, and the version
--agreement measures (default: none, the unversioned bucket)
--codebook FILE failure-mode list for the review form, JSON {"version": ..., "modes":
[{"name": ...}]} (default: the modes `calibrated-agent-evaluation cluster
--ratify` stored for --codebook-version)
--agreement measure inter-rater agreement on the gold set (Krippendorff's alpha)
instead of serving; needs --raters
--raters A,B[,C] with --agreement: the rater ids whose double-coded labels are compared
--axis {outcome,failure_mode}
with --agreement: compare PASS/FAIL (default) or the failure mode
--backend {local,langfuse}
trace-store backend (default: local, offline)
--root PATH local root for the trace store and the stage records (default:
$CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, else ./.calibrated-agent-
evaluation; quickstart defaults to a temp dir)
--explain after the plain summary, print the statistics behind it (bounds, methods,
thresholds, record paths); CALIBRATED_AGENT_EVALUATION_EXPLAIN=1 turns it
on for every command