Skip to content

calibrated-agent-evaluation label

open the review surface where a human labels traces PASS/FAIL with a critique

usage: calibrated-agent-evaluation label [-h] [--trace TRACE_ID] [--serve] [--rater RATER_ID]
                                         [--frame {discovery,production-representative}]
                                         [--cell KEY=VALUE] [--limit LIMIT] [--relabel]
                                         [--session-log PATH] [--pairwise] [--atoms FILE]
                                         [--host HOST] [--port PORT] [--codebook-version V]
                                         [--codebook FILE] [--agreement] [--raters A,B[,C]]
                                         [--axis {outcome,failure_mode}]
                                         [--backend {local,langfuse}] [--root PATH] [--explain]

Print the command that serves the review surface, or serve it with --serve. Every label carries
the rater id; --trace/--frame/--cell/--limit narrow the work list.

options:
  -h, --help            show this help message and exit
  --trace TRACE_ID      operate on one stored trace (default: every stored trace); an unknown id
                        is refused
  --serve               launch the review server (needs [ui]); omit to print the launch command +
                        URL
  --rater RATER_ID      your rater id, stamped on every label you write (required with --serve)
  --frame {discovery,production-representative}
                        review only traces in this frame
  --cell KEY=VALUE      review only traces whose taxonomy cell has KEY=VALUE (repeatable)
  --limit LIMIT         review at most this many traces still to label
  --relabel             include traces you already labeled (default: resume, skipping them)
  --session-log PATH    JSONL file that gets one line per verdict with its time on trace (default:
                        <root>/review-sessions.jsonl)
  --pairwise            with --serve: serve the A/B page instead, where you pick the better of two
                        traces on each subjective dimension of --atoms; choices go to
                        <root>/records/preference/
  --atoms FILE          with --pairwise: the atom set whose subjective dimensions you compare on
  --host HOST           loopback interface the review server binds (default: 127.0.0.1; ::1 and
                        localhost also work). Other addresses are refused: label from another
                        machine over an SSH port-forward
  --port PORT           port the review server listens on (default: 5001)
  --codebook-version V  codebook version stamped on every label you write, and the version
                        --agreement measures (default: none, the unversioned bucket)
  --codebook FILE       failure-mode list for the review form, JSON {"version": ..., "modes":
                        [{"name": ...}]} (default: the modes `calibrated-agent-evaluation cluster
                        --ratify` stored for --codebook-version)
  --agreement           measure inter-rater agreement on the gold set (Krippendorff's alpha)
                        instead of serving; needs --raters
  --raters A,B[,C]      with --agreement: the rater ids whose double-coded labels are compared
  --axis {outcome,failure_mode}
                        with --agreement: compare PASS/FAIL (default) or the failure mode
  --backend {local,langfuse}
                        trace-store backend (default: local, offline)
  --root PATH           local root for the trace store and the stage records (default:
                        $CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, else ./.calibrated-agent-
                        evaluation; quickstart defaults to a temp dir)
  --explain             after the plain summary, print the statistics behind it (bounds, methods,
                        thresholds, record paths); CALIBRATED_AGENT_EVALUATION_EXPLAIN=1 turns it
                        on for every command