Skip to content

calibrated_agent_evaluation.review

calibrated_agent_evaluation.review

Friction-free trace-review surface.

The library's flagship "look at the data" surface: a thin FastHTML app whose primary screen is a single-trace inline-label view (not a metrics dashboard). It reads normalized Traces and writes human Labels only through the backend interfaces (TraceStoreBackend / LabelBackend); no vendor SDK type appears in this package. The surface exists to make human review fast enough that error analysis stops being the step teams skip: hotkeys + optimistic auto-advance let a competent labeler read + label 50 traces in under 30 minutes.

Public surface

  • build_app: the FastHTML application factory (app.py).
  • ReviewSession: in-memory work-list / defer / undo / timing (session.py).
  • WorkScope / build_work_list; which traces a session covers: resume per rater, plus trace / frame / cell / limit narrowing (session.py).
  • SessionLog: the append-only JSONL record of verdicts and undos with per-trace timing (session_log.py).
  • build_label + PASS_DEFAULT_CRITIQUE; critique-first label construction (labeling).
  • Prefill / Suggestion / NoPrefill; the opt-in LLM pre-fill seam (prefill.py).
  • RendererRegistry: the plugin-renderer seam keyed on trace metadata; construct with a default renderer, register(predicate, renderer) domain renderers, render(trace) guards a raising renderer back to the default (renderers).
  • inter_rater_agreement / alpha_gate / newest_per_rater_unit; the gold-set agreement gate over double-coded human labels (agreement), and AgreementRecord / agreement_record_of, what calibrated-agent-evaluation label --agreement stores (record).
  • CodebookFile / CodebookRecord / load_codebook_file; the failure-mode list the review form offers, hand-written or ratified by calibrated-agent-evaluation cluster --ratify (record).
  • plan_overlap / MINIMUM_OVERLAP; how many double-coded traces the agreement gate needs before it can say usable (overlap).
  • ReviewError / RatificationError / LabelSubmitError / AgreementDataError; the typed error family.

ALPHA_LOWER_BOUND_GATE module-attribute

ALPHA_LOWER_BOUND_GATE = 0.8

ALPHA_REVISE_FLOOR module-attribute

ALPHA_REVISE_FLOOR = 0.667

MINIMUM_OVERLAP module-attribute

MINIMUM_OVERLAP = 130

PASS_DEFAULT_CRITIQUE module-attribute

PASS_DEFAULT_CRITIQUE = (
    "Reviewed: outcome PASS, no defect observed."
)

AgreementDataError

Bases: ReviewError

The labels handed to the inter-rater agreement path cannot form an agreement set.

Fewer than two distinct raters, a rater with two labels on one unit, a non-human label under a named rater, or no label in the requested codebook version. Each would otherwise mean a silent choice about which labels the number measures.

LabelSubmitError

Bases: ReviewError

A label write through the backend failed and could not be reconciled.

Wraps the underlying SchemaError/BackendError (chained with raise … from) so the review session surfaces the failure and re-queues the trace as unlabeled rather than presenting it as successfully labeled.

RatificationError

Bases: ReviewError

An LLM-suggested label was submitted without an explicit human accept/override.

The human-gold boundary: an LLM Suggestion may pre-fill the form, but a kind="human" label is refused unless a human ratified it. No LLM-origin label reaches the gold path unratified.

ReviewError

Bases: CalibratedAgentEvaluationDataError

Base class for every review-surface error.

NoPrefill

The default: no LLM assist. Every trace is labeled from scratch by the human.

suggest

suggest(trace)

Return None: no suggestion (the offline, zero-dependency default).

Source code in src/calibrated_agent_evaluation/review/prefill.py
def suggest(self, trace: Trace) -> Suggestion | None:  # noqa: ARG002 - protocol shape
    """Return ``None``: no suggestion (the offline, zero-dependency default)."""
    return None

Prefill

Bases: Protocol

Proposes a Suggestion for a trace, or None when it has no proposal.

suggest

suggest(trace)

Return a pre-fill suggestion for trace, or None to leave the form blank.

Source code in src/calibrated_agent_evaluation/review/prefill.py
def suggest(self, trace: Trace) -> Suggestion | None:
    """Return a pre-fill suggestion for ``trace``, or ``None`` to leave the form blank."""
    ...

Suggestion dataclass

Suggestion(outcome, critique)

An LLM proposal for a trace. NEVER a label until a human ratifies it.

outcome instance-attribute

outcome

critique instance-attribute

critique

AgreementRecord pydantic-model

Bases: CalibratedAgentEvaluationModel

The gold-set agreement gate over one scope, one axis and one codebook version.

gate is usable only when alpha.lower reaches alpha_gate (the 0.80 bound) on at least minimum_overlap pairable units; top_up_overlap / revise_codebook / codebook_broken otherwise (see alpha_gate). minimum_overlap is None on a record written before the gate had a minimum. codebook_version None is the unversioned bucket. trace_ids lists every trace with a unit at least one named rater coded, and n_pairable_units how many units two or more coded. n_superseded counts the older labels a rater's relabel replaced (newest label per rater and unit). cell / trace_id / frame record the scope flags the command ran with.

Config:

  • frozen: True
  • extra: forbid

Fields:

agreement_id pydantic-field

agreement_id

axis pydantic-field

axis

codebook_version pydantic-field

codebook_version

raters pydantic-field

raters

trace_ids pydantic-field

trace_ids

n_units pydantic-field

n_units

n_pairable_units pydantic-field

n_pairable_units

n_superseded pydantic-field

n_superseded

alpha pydantic-field

alpha

cohen_kappa pydantic-field

cohen_kappa

per_class pydantic-field

per_class

gate pydantic-field

gate

alpha_gate pydantic-field

alpha_gate = ALPHA_LOWER_BOUND_GATE

alpha_revise_floor pydantic-field

alpha_revise_floor = ALPHA_REVISE_FLOOR

minimum_overlap pydantic-field

minimum_overlap = None

cell pydantic-field

cell = None

trace_id pydantic-field

trace_id = None

frame pydantic-field

frame = None

provenance pydantic-field

provenance

schema_version pydantic-field

schema_version = AGREEMENT_SCHEMA_VERSION

AgreementStat pydantic-model

Bases: CalibratedAgentEvaluationModel

One agreement statistic with its two-sided percentile bootstrap interval over units.

Config:

  • frozen: True
  • extra: forbid

Fields:

point pydantic-field

point

lower pydantic-field

lower

upper pydantic-field

upper

confidence pydantic-field

confidence

n_units pydantic-field

n_units

n_resamples pydantic-field

n_resamples

n_undefined pydantic-field

n_undefined

seed pydantic-field

seed

of classmethod

of(interval)

The stored form of a bootstrap interval.

Source code in src/calibrated_agent_evaluation/review/record.py
@classmethod
def of(cls, interval: AgreementInterval) -> AgreementStat:
    """The stored form of a bootstrap interval."""
    return cls(
        point=interval.point,
        lower=interval.lower,
        upper=interval.upper,
        confidence=interval.confidence,
        n_units=interval.n_units,
        n_resamples=interval.n_resamples,
        n_undefined=interval.n_undefined,
        seed=interval.seed,
    )

CodebookFile pydantic-model

Bases: CalibratedAgentEvaluationModel

A hand-written failure-mode list: {"version": "1", "modes": [{"name": ..., ...}]}.

Config:

  • frozen: True
  • extra: forbid

Fields:

Validators:

  • _check

version pydantic-field

version

modes pydantic-field

modes

CodebookMode pydantic-model

Bases: CalibratedAgentEvaluationModel

One failure mode: the name a label carries in failure_mode, and what it means.

Config:

  • frozen: True
  • extra: forbid

Fields:

Validators:

name pydantic-field

name

description pydantic-field

description = ''

CodebookRecord pydantic-model

Bases: CalibratedAgentEvaluationModel

A ratified failure-mode vocabulary under one codebook version.

ratified_by names the human who ratified it and ratified_at when (UTC ISO-8601). parent_version is the version this one follows, None for the first. n_tagged counts the FAIL labels the ratification stamped with a mode, and source names the map file.

Config:

  • frozen: True
  • extra: forbid

Fields:

Validators:

  • _check

codebook_id pydantic-field

codebook_id

version pydantic-field

version

modes pydantic-field

modes

ratified_by pydantic-field

ratified_by

ratified_at pydantic-field

ratified_at

parent_version pydantic-field

parent_version = None

n_tagged pydantic-field

n_tagged = 0

source pydantic-field

source = None

merged pydantic-field

merged

provenance pydantic-field

provenance

schema_version pydantic-field

schema_version = CODEBOOK_SCHEMA_VERSION

RendererRegistry dataclass

RendererRegistry(default, _rules=list())

Resolve a trace's renderer: the first registered predicate that matches, else the default.

Construct with the default renderer (the app wires the tool-call-tree renderer). Register domain renderers with register. render is the safe entry point; it guards a raising renderer back to the default and logs, so one bad plugin never ends the session.

default instance-attribute

default

register

register(predicate, renderer)

Register renderer for any trace where predicate(trace) is true.

Registration order is match order; the first matching predicate wins.

Source code in src/calibrated_agent_evaluation/review/renderers/base.py
def register(self, predicate: RendererPredicate, renderer: Renderer) -> None:
    """Register ``renderer`` for any trace where ``predicate(trace)`` is true.

    Registration order is match order; the first matching predicate wins.
    """
    self._rules.append((predicate, renderer))

resolve

resolve(trace)

Return the first renderer whose predicate matches trace, else the default.

Note: a predicate that itself raises is treated as non-matching (guarded + logged) so a broken predicate cannot mask the default.

Source code in src/calibrated_agent_evaluation/review/renderers/base.py
def resolve(self, trace: Trace) -> Renderer:
    """Return the first renderer whose predicate matches ``trace``, else the default.

    Note: a predicate that itself raises is treated as non-matching (guarded + logged) so a
    broken predicate cannot mask the default.
    """
    for predicate, renderer in self._rules:
        try:
            matched = predicate(trace)
        except Exception:
            logger.exception(
                "renderer predicate raised for trace %s; treating as no-match", trace.trace_id
            )
            continue
        if matched:
            return renderer
    return self.default

render

render(trace)

Render trace with the resolved renderer; fall back to the default on error.

A registered renderer that raises is caught and logged (logging.exception) and the default renderer is used instead; the labeling session continues rather than crashing. Use render_with_status to also learn that a fallback happened.

Source code in src/calibrated_agent_evaluation/review/renderers/base.py
def render(self, trace: Trace) -> Any:
    """Render ``trace`` with the resolved renderer; fall back to the default on error.

    A registered renderer that raises is caught and logged (``logging.exception``) and the
    default renderer is used instead; the labeling session continues rather than crashing.
    Use :meth:`render_with_status` to also learn that a fallback happened.
    """
    output, _fallback = self.render_with_status(trace)
    return output

render_with_status

render_with_status(trace)

Like render, plus a message naming the failure when the default stood in.

The message is for the labeler's screen. A fallback that only reached the log looked, to the person labeling, exactly like a trace the plugin was never meant to render.

Source code in src/calibrated_agent_evaluation/review/renderers/base.py
def render_with_status(self, trace: Trace) -> tuple[Any, str | None]:
    """Like :meth:`render`, plus a message naming the failure when the default stood in.

    The message is for the labeler's screen. A fallback that only reached the log looked, to
    the person labeling, exactly like a trace the plugin was never meant to render.
    """
    renderer = self.resolve(trace)
    try:
        return renderer(trace), None
    except Exception as exc:
        logger.exception(
            "plugin renderer failed for trace %s; falling back to the default renderer",
            trace.trace_id,
        )
        name = getattr(renderer, "__name__", type(renderer).__name__)
        message = (
            f"The {name!r} renderer for this trace failed ({type(exc).__name__}: {exc}); "
            "showing the default view instead."
        )
        return self.default(trace), message

LabeledEntry dataclass

LabeledEntry(
    trace_id, label, seconds_on_trace, span_labels=()
)

One completed submission: kept for undo and the throughput summary.

span_labels holds the span-level defect labels written with the verdict, so undo can retract everything the submission put in the store, not only the trace-level label.

trace_id instance-attribute

trace_id

label instance-attribute

label

seconds_on_trace instance-attribute

seconds_on_trace

span_labels class-attribute instance-attribute

span_labels = ()

ReviewSession dataclass

ReviewSession(
    pending,
    done=list(),
    clock=time.monotonic,
    _shown_id=None,
    _shown_at=None,
    _unlabeled_seconds=0.0,
)

The work-list for one labeling session.

pending is the ordered queue of trace_ids awaiting a verdict; done is the stack of completed entries (undo pops the last). Construct with the ids returned by LabelBackend.next_item / TraceStoreBackend.query in a deterministic order.

pending instance-attribute

pending

done class-attribute instance-attribute

done = field(default_factory=list)

clock class-attribute instance-attribute

clock = time.monotonic

is_complete property

is_complete

True when no trace remains to label (show the completion state).

from_ids classmethod

from_ids(ids, *, clock=time.monotonic)

Build a session from an ordered iterable of trace_ids (dedup, order-preserving).

Source code in src/calibrated_agent_evaluation/review/session.py
@classmethod
def from_ids(
    cls, ids: Iterable[str], *, clock: Callable[[], float] = time.monotonic
) -> ReviewSession:
    """Build a session from an ordered iterable of ``trace_id``s (dedup, order-preserving)."""
    seen: dict[str, None] = {}
    for tid in ids:
        if tid:
            seen.setdefault(tid, None)
    return cls(pending=list(seen), clock=clock)

current

current()

The trace at the head of the work-list, or None when the queue is empty.

Source code in src/calibrated_agent_evaluation/review/session.py
def current(self) -> str | None:
    """The trace at the head of the work-list, or ``None`` when the queue is empty."""
    return self.pending[0] if self.pending else None

mark_shown

mark_shown(trace_id=None)

Start timing trace_id (default: the queue head) as the trace on screen.

Called whenever a trace is rendered, with the id actually rendered: after navigation that is not the queue head, and timing the head instead charged the interval to the wrong trace. Time on the trace previously shown is banked. Re-rendering the trace already being timed (a refused submit) keeps its clock running.

Source code in src/calibrated_agent_evaluation/review/session.py
def mark_shown(self, trace_id: str | None = None) -> str | None:
    """Start timing ``trace_id`` (default: the queue head) as the trace on screen.

    Called whenever a trace is rendered, with the id actually rendered: after navigation that
    is not the queue head, and timing the head instead charged the interval to the wrong
    trace. Time on the trace previously shown is banked. Re-rendering the trace already being
    timed (a refused submit) keeps its clock running.
    """
    target = trace_id if trace_id is not None else self.current()
    if target is not None and target == self._shown_id and self._shown_at is not None:
        return target
    self._stop_timer()
    if target is not None:
        self._shown_id = target
        self._shown_at = self.clock()
    return target

record

record(label)

Record label against its trace, remove it from the queue, stop its timer.

Raises ReviewError if the label's trace is not in the pending work-list; a label for a trace the session isn't tracking is a caller bug, not something to swallow.

Source code in src/calibrated_agent_evaluation/review/session.py
def record(self, label: Label) -> LabeledEntry:
    """Record ``label`` against its trace, remove it from the queue, stop its timer.

    Raises ``ReviewError`` if the label's trace is not in the pending work-list; a label for
    a trace the session isn't tracking is a caller bug, not something to swallow.
    """
    tid = label.trace_id
    if tid not in self.pending:
        raise ReviewError(f"{tid!r} is not in the pending work-list; cannot record its label.")
    # The interval belongs to this label only if this trace is the one being timed; otherwise
    # the running time is banked as unlabeled and the label carries none.
    seconds = self._elapsed() if self._shown_id == tid else 0.0
    if self._shown_id == tid:
        self._shown_id = None
        self._shown_at = None
    else:
        self._stop_timer()
    entry = LabeledEntry(trace_id=tid, label=label, seconds_on_trace=seconds)
    self.pending.remove(tid)
    self.done.append(entry)
    return entry

last

last()

The most recently recorded entry (what undo would retract), or None.

Source code in src/calibrated_agent_evaluation/review/session.py
def last(self) -> LabeledEntry | None:
    """The most recently recorded entry (what ``undo`` would retract), or ``None``."""
    return self.done[-1] if self.done else None

attach_span_label

attach_span_label(label)

Record a span-level label written alongside the most recent verdict.

Raises ReviewError if nothing has been recorded or the label is for another trace.

Source code in src/calibrated_agent_evaluation/review/session.py
def attach_span_label(self, label: Label) -> LabeledEntry:
    """Record a span-level label written alongside the most recent verdict.

    Raises ``ReviewError`` if nothing has been recorded or the label is for another trace.
    """
    entry = self.last()
    if entry is None or entry.trace_id != label.trace_id:
        raise ReviewError(
            f"no recorded verdict for {label.trace_id!r} to attach a span-level label to."
        )
    amended = replace(entry, span_labels=(*entry.span_labels, label))
    self.done[-1] = amended
    return amended

defer

defer(trace_id=None)

Move trace_id (default: the current trace) to the end of the work-list.

Returns the new current trace. A no-op returning None when the queue is empty.

Source code in src/calibrated_agent_evaluation/review/session.py
def defer(self, trace_id: str | None = None) -> str | None:
    """Move ``trace_id`` (default: the current trace) to the end of the work-list.

    Returns the new current trace. A no-op returning ``None`` when the queue is empty.
    """
    target = trace_id if trace_id is not None else self.current()
    if target is None or target not in self.pending:
        return self.current()
    self.pending.remove(target)
    self.pending.append(target)
    self._stop_timer()
    return self.current()

undo

undo()

Retract the most recently recorded label and re-queue its trace at the front.

Returns the retracted entry, or None when nothing has been labeled yet (empty-stack no-op; undoing an empty history is a benign no-op, not an error).

Source code in src/calibrated_agent_evaluation/review/session.py
def undo(self) -> LabeledEntry | None:
    """Retract the most recently recorded label and re-queue its trace at the front.

    Returns the retracted entry, or ``None`` when nothing has been labeled yet (empty-stack
    no-op; undoing an empty history is a benign no-op, not an error).
    """
    if not self.done:
        return None
    entry = self.done.pop()
    self.pending.insert(0, entry.trace_id)
    self._stop_timer()
    self._unlabeled_seconds += entry.seconds_on_trace  # the time was spent; the verdict is gone
    return entry

summary

summary()

Session throughput.

labeled_seconds is the time on traces whose verdict stands; elapsed_seconds is all on-trace time, including deferred, navigated-past, undone, and the trace on screen now.

Source code in src/calibrated_agent_evaluation/review/session.py
def summary(self) -> dict[str, float | int]:
    """Session throughput.

    ``labeled_seconds`` is the time on traces whose verdict stands; ``elapsed_seconds`` is all
    on-trace time, including deferred, navigated-past, undone, and the trace on screen now.
    """
    labeled = sum(e.seconds_on_trace for e in self.done)
    return {
        "labeled_count": len(self.done),
        "labeled_seconds": labeled,
        "elapsed_seconds": labeled + self._unlabeled_seconds + self._elapsed(),
    }

WorkScope dataclass

WorkScope(
    frame=None,
    cell=None,
    trace_id=None,
    limit=None,
    relabel=False,
)

Which traces a review session covers (label --trace/--frame/--cell/--limit).

frame and cell narrow the backend query; trace_id names one trace; limit caps the work-list after resume filtering, so limit=50 is 50 traces still to label. By default a trace this rater already labeled at trace level is left out (resume); relabel keeps it.

frame class-attribute instance-attribute

frame = None

cell class-attribute instance-attribute

cell = None

trace_id class-attribute instance-attribute

trace_id = None

limit class-attribute instance-attribute

limit = None

relabel class-attribute instance-attribute

relabel = False

SessionLog

SessionLog(path)

Appends session events to a JSONL file. Opens the file per write; holds no handle.

Source code in src/calibrated_agent_evaluation/review/session_log.py
def __init__(self, path: str | Path) -> None:
    self.path = Path(path)
    try:  # fail at startup, not at the first verdict, if the log cannot be written
        self.path.parent.mkdir(parents=True, exist_ok=True)
        with self.path.open("a", encoding="utf-8"):
            pass
    except OSError as exc:
        raise ReviewError(f"cannot write the session log {self.path}: {exc}") from exc

path instance-attribute

path = Path(path)

label

label(entry, *, rater_id)

Record a verdict that was just stored, with the time spent on its trace.

Source code in src/calibrated_agent_evaluation/review/session_log.py
def label(self, entry: LabeledEntry, *, rater_id: str) -> None:
    """Record a verdict that was just stored, with the time spent on its trace."""
    self._append(
        {
            "event": "label",
            "trace_id": entry.trace_id,
            "rater_id": rater_id,
            "outcome": entry.label.outcome,
            "seconds_on_trace": entry.seconds_on_trace,
            "span_labels": [lbl.span_id for lbl in entry.span_labels],
        }
    )

undo

undo(entry, *, rater_id)

Record that the verdict for entry.trace_id was retracted.

Source code in src/calibrated_agent_evaluation/review/session_log.py
def undo(self, entry: LabeledEntry, *, rater_id: str) -> None:
    """Record that the verdict for ``entry.trace_id`` was retracted."""
    self._append({"event": "undo", "trace_id": entry.trace_id, "rater_id": rater_id})

InterRaterReport dataclass

InterRaterReport(
    axis,
    codebook_version,
    raters,
    n_units,
    n_pairable_units,
    alpha,
    cohen_kappa,
    per_class,
    gate,
    n_superseded=0,
    minimum_overlap=MINIMUM_OVERLAP,
)

Agreement among the named raters on one axis, within one codebook version.

n_units counts units at least one named rater coded; n_pairable_units the ones two or more coded, which are the only units alpha uses. cohen_kappa is None unless exactly two raters are named. per_class has an entry for every category observed on a pairable unit. n_superseded counts the older labels a rater's relabel replaced. minimum_overlap is the pairable-unit count below which gate cannot be usable.

axis instance-attribute

axis

codebook_version instance-attribute

codebook_version

raters instance-attribute

raters

n_units instance-attribute

n_units

n_pairable_units instance-attribute

n_pairable_units

alpha instance-attribute

alpha

cohen_kappa instance-attribute

cohen_kappa

per_class instance-attribute

per_class

gate instance-attribute

gate

n_superseded class-attribute instance-attribute

n_superseded = 0

minimum_overlap class-attribute instance-attribute

minimum_overlap = MINIMUM_OVERLAP

OverlapPlan pydantic-model

Bases: CalibratedAgentEvaluationModel

The minimum overlap for raters expected to agree at expected_alpha.

n_units is the smallest overlap from which the plug-in counts clear the gate, with the margin for its resampling, at every size up to scan_limit.

Fields:

n_units pydantic-field

n_units

expected_alpha pydantic-field

expected_alpha

prevalence pydantic-field

prevalence

gate_lower_bound pydantic-field

gate_lower_bound

confidence pydantic-field

confidence

gate_resamples pydantic-field

gate_resamples

scan_limit pydantic-field

scan_limit

build_label

build_label(
    *,
    trace_id,
    outcome,
    critique,
    span_id=None,
    failure_mode=None,
    culprit_span=None,
    manifestation_span=None,
    handoff_implicated=None,
    span_defect=False,
    expected=None,
    codebook_version=None,
    from_suggestion=False,
    ratified=False,
    rater_id=None,
    span_ids=None,
    failure_modes=None,
)

Build a validated kind="human" Label from a review submission.

The arguments mirror the labeling form. from_suggestion/ratified carry the LLM-assist provenance; rater_id names the human in session (the review app always passes one). span_ids, when given, is the trace's span ids; every span the label names must be one. failure_modes, when given and non-empty, is the codebook's failure-mode list: a FAIL must name one of them and a PASS names none. The returned Label still passes through the backend's validate on write.

Raises:

Type Description
ReviewError

outcome is not a valid binary verdict, a FAIL has an empty critique, a named span is not in span_ids, or the failure mode breaks the failure_modes rule.

RatificationError

The values came from a Suggestion the human did not ratify.

Source code in src/calibrated_agent_evaluation/review/labeling.py
def build_label(
    *,
    trace_id: str,
    outcome: str,
    critique: str,
    span_id: str | None = None,
    failure_mode: str | None = None,
    culprit_span: str | None = None,
    manifestation_span: str | None = None,
    handoff_implicated: str | None = None,
    span_defect: bool = False,
    expected: dict[str, Any] | None = None,
    codebook_version: str | None = None,
    from_suggestion: bool = False,
    ratified: bool = False,
    rater_id: str | None = None,
    span_ids: Collection[str] | None = None,
    failure_modes: Collection[str] | None = None,
) -> Label:
    """Build a validated ``kind="human"`` ``Label`` from a review submission.

    The arguments mirror the labeling form. ``from_suggestion``/``ratified`` carry the LLM-assist
    provenance; ``rater_id`` names the human in session (the review app always passes one).
    ``span_ids``, when given, is the trace's span ids; every span the label names must be one.
    ``failure_modes``, when given and non-empty, is the codebook's failure-mode list: a FAIL must
    name one of them and a PASS names none. The returned ``Label`` still passes through the
    backend's ``validate`` on write.

    Raises:
        ReviewError: ``outcome`` is not a valid binary verdict, a FAIL has an empty
            ``critique``, a named span is not in ``span_ids``, or the failure mode breaks the
            ``failure_modes`` rule.
        RatificationError: The values came from a ``Suggestion`` the human did not ratify.
    """
    if not trace_id:
        raise ReviewError("A label requires a non-empty trace_id (trace-central).")

    if outcome not in _VALID_OUTCOMES:
        raise ReviewError(f"outcome must be a binary verdict {_VALID_OUTCOMES}; got {outcome!r}.")

    # Refuse an LLM-suggested label the human never explicitly accepted/overrode,
    # before any critique defaulting; no LLM-origin label reaches the gold path unratified.
    if from_suggestion and not ratified:
        raise RatificationError(
            "This label was pre-filled by an LLM suggestion and has not been ratified by a "
            "human. Accept or override the suggestion before submitting."
        )

    if span_ids is not None:
        # A span id that is not in the trace is refused, never stored:
        # `calibrated-agent-evaluation run --ci` derives the gated agent from `culprit_span`, so a
        # typo minted a failure mode for an agent that does not exist.
        for field_name, value in (
            ("span_id", span_id),
            ("culprit_span", culprit_span),
            ("manifestation_span", manifestation_span),
        ):
            if value is not None and value not in span_ids:
                raise ReviewError(
                    f"{field_name} {value!r} is not a span of trace {trace_id!r}; "
                    f"pick one of: {', '.join(sorted(span_ids)) or '(no spans)'}."
                )

    critique = (critique or "").strip()
    if outcome == "FAIL" and not critique:
        # A hard refusal; the FAIL "why / first failure" is the irreducible asset;
        # a critique-less FAIL is the exact anti-pattern the surface exists to kill.
        raise ReviewError(
            "A FAIL requires a non-empty critique; record the failure reason before submitting "
            "(critique-first)."
        )
    if outcome == "PASS" and not critique:
        critique = PASS_DEFAULT_CRITIQUE  # tiered: a default, not a hard block, on PASS.

    if failure_modes:
        _check_failure_mode(outcome, failure_mode, failure_modes)

    return Label(
        trace_id=trace_id,
        span_id=span_id,
        kind="human",  # the review surface only ever writes human-provenance labels.
        outcome=outcome,  # type: ignore[arg-type]  # validated against _VALID_OUTCOMES above
        critique=critique,
        failure_mode=failure_mode,
        culprit_span=culprit_span,
        manifestation_span=manifestation_span,
        handoff_implicated=handoff_implicated,
        span_defect=span_defect,
        codebook_version=codebook_version,
        expected=expected,
        rater_id=rater_id,
    )

agreement_record_of

agreement_record_of(
    report,
    *,
    trace_ids,
    cell=None,
    trace_id=None,
    frame=None,
    provenance=None,
)

Build the AgreementRecord for report; the id is a hash of its content.

Source code in src/calibrated_agent_evaluation/review/record.py
def agreement_record_of(
    report: InterRaterReport,
    *,
    trace_ids: Collection[str],
    cell: Mapping[str, str] | None = None,
    trace_id: str | None = None,
    frame: str | None = None,
    provenance: ProvenanceStamp | None = None,
) -> AgreementRecord:
    """Build the :class:`AgreementRecord` for ``report``; the id is a hash of its content."""
    stamp = provenance or capture_provenance(
        seed=report.alpha.seed,
        n_resamples=report.alpha.n_resamples,
        thresholds={
            "alpha_gate": ALPHA_LOWER_BOUND_GATE,
            "alpha_revise": ALPHA_REVISE_FLOOR,
            "minimum_overlap": report.minimum_overlap,
        },
    )
    record = AgreementRecord(
        agreement_id="",
        axis=report.axis,
        codebook_version=report.codebook_version,
        raters=tuple(report.raters),
        trace_ids=tuple(sorted(set(trace_ids))),
        n_units=report.n_units,
        n_pairable_units=report.n_pairable_units,
        n_superseded=report.n_superseded,
        alpha=AgreementStat.of(report.alpha),
        cohen_kappa=None if report.cohen_kappa is None else AgreementStat.of(report.cohen_kappa),
        per_class={k: AgreementStat.of(v) for k, v in report.per_class.items()},
        gate=report.gate,
        minimum_overlap=report.minimum_overlap,
        cell=dict(cell) if cell else None,
        trace_id=trace_id,
        frame=frame,
        provenance=stamp,
    )
    return with_content_id(record, "agreement_id", "agreement")

load_codebook_file

load_codebook_file(path)

Read a CodebookFile from JSON.

Raises:

Type Description
ReviewError

The file cannot be read, is not JSON, or is not a valid codebook.

Source code in src/calibrated_agent_evaluation/review/record.py
def load_codebook_file(path: str | Path) -> CodebookFile:
    """Read a :class:`CodebookFile` from JSON.

    Raises:
        ReviewError: The file cannot be read, is not JSON, or is not a valid codebook.
    """
    src = Path(path)
    try:
        doc = json.loads(src.read_text(encoding="utf-8"))
    except OSError as exc:
        raise ReviewError(f"cannot read the codebook file {src}: {exc.strerror or exc}") from exc
    except json.JSONDecodeError as exc:
        raise ReviewError(f"the codebook file {src} is not valid JSON: {exc}") from exc
    if not isinstance(doc, dict):
        raise ReviewError(
            f'the codebook file {src} must be a JSON object: {{"version": "1", "modes": [...]}}'
        )
    return CodebookFile.model_validate(doc)

newest_per_rater_unit

newest_per_rater_unit(labels)

Keep each rater's newest label per (trace_id, span_id) unit, and the drop count.

labels must be oldest first. Labels with no rater_id pass through, since nothing says two of them came from one person. Order is kept.

Source code in src/calibrated_agent_evaluation/review/relabel.py
def newest_per_rater_unit(labels: Sequence[Label]) -> tuple[list[Label], int]:
    """Keep each rater's newest label per ``(trace_id, span_id)`` unit, and the drop count.

    ``labels`` must be oldest first. Labels with no ``rater_id`` pass through, since nothing says
    two of them came from one person. Order is kept.
    """
    newest: dict[tuple[str, str, str], int] = {}
    for index, label in enumerate(labels):
        if label.rater_id is not None:
            newest[(label.rater_id, *unit_of(label))] = index
    keep = set(newest.values())
    kept = [lb for i, lb in enumerate(labels) if lb.rater_id is None or i in keep]
    return kept, len(labels) - len(kept)

build_work_list

build_work_list(backend, scope, *, rater_id)

Resolve scope against the backend into the traces to review, in stable order.

This is the startup reachability probe too: an unreachable backend raises its typed BackendError here, before any session opens. Raises BackendError for a trace_id the backend does not hold, and ReviewError for a bad limit or a narrowed scope (frame, cell or trace) that matches no trace (it would open onto "all done").

Source code in src/calibrated_agent_evaluation/review/session.py
def build_work_list(backend: TraceStoreBackend, scope: WorkScope, *, rater_id: str) -> WorkList:
    """Resolve ``scope`` against the backend into the traces to review, in stable order.

    This is the startup reachability probe too: an unreachable backend raises its typed
    ``BackendError`` here, before any session opens. Raises ``BackendError`` for a
    ``trace_id`` the backend does not hold, and ``ReviewError`` for a bad ``limit`` or a
    narrowed scope (frame, cell or trace) that matches no trace (it would open onto "all done").
    """
    if scope.limit is not None and scope.limit < 1:
        raise ReviewError(f"limit must be a positive number of traces; got {scope.limit}.")
    skipped: tuple[SkippedTrace, ...] = ()
    if scope.trace_id is not None:
        candidates = [backend.get_trace(scope.trace_id)]
    else:
        scan = scan_traces(
            backend, frame=scope.frame, cell=dict(scope.cell) if scope.cell else None
        )
        candidates, skipped = scan.traces, scan.skipped
    candidates = [t for t in candidates if _matches(t, scope)]
    narrowed = scope.frame is not None or bool(scope.cell) or scope.trace_id is not None
    if narrowed and not candidates:
        # A typo in --frame/--cell would otherwise open straight onto the completion screen.
        raise ReviewError(f"no trace in the backend matches the requested scope {scope}.")

    traces: list[Trace] = []
    already: list[str] = []
    for trace in candidates:
        if scope.limit is not None and len(traces) >= scope.limit:
            break  # stop before reading labels for traces the session will never show
        if not scope.relabel and _labeled_by(backend, trace.trace_id, rater_id):
            already.append(trace.trace_id)
            continue
        traces.append(trace)
    return WorkList(traces=traces, already_labeled=already, not_calibrated_agent_evaluation=skipped)

alpha_gate

alpha_gate(
    alpha,
    *,
    n_pairable_units,
    minimum_overlap=MINIMUM_OVERLAP,
)

Return the gold-set verdict for an alpha interval over n_pairable_units shared units.

Below minimum_overlap units coded by two or more raters the interval cannot show the gold is usable (raters who agree on all 20 shared traces get a lower bound of 1.0), but it can show it is not:

  • codebook_broken: alpha's upper bound is below ALPHA_REVISE_FLOOR.
  • revise_codebook: alpha's upper bound is below ALPHA_LOWER_BOUND_GATE.
  • top_up_overlap: otherwise; double-code more of the calibration gold.

From minimum_overlap up:

  • usable: the lower bound is >= ALPHA_LOWER_BOUND_GATE. The bound gates, not the point estimate.
  • revise_codebook: the point estimate is in [ALPHA_REVISE_FLOOR, gate).
  • codebook_broken: the point estimate is below ALPHA_REVISE_FLOOR.
  • top_up_overlap: the point estimate clears the gate but the lower bound does not. The interval is too wide to tell, which is a sample-size problem, not a codebook one.

MINIMUM_OVERLAP comes from plan_overlap.

Source code in src/calibrated_agent_evaluation/review/agreement.py
def alpha_gate(
    alpha: AgreementInterval, *, n_pairable_units: int, minimum_overlap: int = MINIMUM_OVERLAP
) -> GateVerdict:
    """Return the gold-set verdict for an alpha interval over ``n_pairable_units`` shared units.

    Below ``minimum_overlap`` units coded by two or more raters the interval cannot show the
    gold is usable (raters who agree on all 20 shared traces get a lower bound of 1.0), but it
    can show it is not:

    * ``codebook_broken``: alpha's upper bound is below :data:`ALPHA_REVISE_FLOOR`.
    * ``revise_codebook``: alpha's upper bound is below :data:`ALPHA_LOWER_BOUND_GATE`.
    * ``top_up_overlap``: otherwise; double-code more of the calibration gold.

    From ``minimum_overlap`` up:

    * ``usable``: the lower bound is >= :data:`ALPHA_LOWER_BOUND_GATE`. The bound gates, not the
      point estimate.
    * ``revise_codebook``: the point estimate is in [:data:`ALPHA_REVISE_FLOOR`, gate).
    * ``codebook_broken``: the point estimate is below :data:`ALPHA_REVISE_FLOOR`.
    * ``top_up_overlap``: the point estimate clears the gate but the lower bound does not. The
      interval is too wide to tell, which is a sample-size problem, not a codebook one.

    :data:`MINIMUM_OVERLAP` comes from
    :func:`calibrated_agent_evaluation.review.overlap.plan_overlap`.
    """
    if n_pairable_units < minimum_overlap:
        # More shared units narrow the interval but cannot lift a band its upper end is under.
        if alpha.upper < ALPHA_REVISE_FLOOR:
            return "codebook_broken"
        if alpha.upper < ALPHA_LOWER_BOUND_GATE:
            return "revise_codebook"
        return "top_up_overlap"
    if alpha.lower >= ALPHA_LOWER_BOUND_GATE:
        return "usable"
    if alpha.point >= ALPHA_LOWER_BOUND_GATE:
        return "top_up_overlap"
    if alpha.point >= ALPHA_REVISE_FLOOR:
        return "revise_codebook"
    return "codebook_broken"

inter_rater_agreement

inter_rater_agreement(
    labels,
    *,
    raters,
    codebook_version,
    axis="outcome",
    n_resamples=2000,
    confidence=0.95,
    seed=0,
    minimum_overlap=MINIMUM_OVERLAP,
)

Return the named raters' agreement on axis within codebook_version.

Only kind="human" labels whose rater_id is in raters and whose codebook_version matches enter (None selects the unversioned labels); every other label (an adjudicator's, an unattributed one, another version's) is outside this agreement set by construction. A unit is a (trace_id, span_id); a rater who labeled a unit twice counts with the newest label, so labels must be oldest first. Every statistic is bootstrapped over the same seeded unit resamples. Below minimum_overlap pairable units the gate is top_up_overlap unless alpha's upper bound already rules the gold out (see alpha_gate).

Raises AgreementDataError for fewer than two distinct raters, a non-human label under a named rater, or no matching label, and UndefinedAgreementError when the raters share no unit or only one category is observed.

Source code in src/calibrated_agent_evaluation/review/agreement.py
def inter_rater_agreement(
    labels: Sequence[Label],
    *,
    raters: Collection[str],
    codebook_version: str | None,
    axis: AgreementAxis = "outcome",
    n_resamples: int = 2000,
    confidence: float = 0.95,
    seed: int = 0,
    minimum_overlap: int = MINIMUM_OVERLAP,
) -> InterRaterReport:
    """Return the named raters' agreement on ``axis`` within ``codebook_version``.

    Only ``kind="human"`` labels whose ``rater_id`` is in ``raters`` and whose ``codebook_version``
    matches enter (``None`` selects the unversioned labels); every other label (an adjudicator's,
    an unattributed one, another version's) is outside this agreement set by construction. A unit
    is a ``(trace_id, span_id)``; a rater who labeled a unit twice counts with the newest label, so
    ``labels`` must be oldest first. Every statistic is bootstrapped over the same seeded unit
    resamples. Below ``minimum_overlap`` pairable units the gate is ``top_up_overlap`` unless
    alpha's upper bound already rules the gold out (see :func:`alpha_gate`).

    Raises :class:`AgreementDataError` for fewer than two distinct raters, a non-human label under
    a named rater, or no matching label, and
    :class:`~calibrated_agent_evaluation.stats.agreement.UndefinedAgreementError` when the raters
    share no unit or only one category is observed.
    """
    ordered = tuple(dict.fromkeys(raters))
    if len(ordered) < 2:
        raise AgreementDataError(
            f"inter_rater_agreement: needs at least two distinct raters, got {list(ordered)}"
        )
    units, superseded = _reliability_data(labels, ordered, codebook_version, axis)

    def interval(statistic: _Statistic) -> AgreementInterval:
        return bootstrap_agreement(
            units, statistic, n_resamples=n_resamples, confidence=confidence, seed=seed
        )

    alpha = interval(krippendorff_alpha)
    kappa = interval(_two_rater_kappa) if len(ordered) == 2 else None
    per_class = {
        str(category): interval(_class_agreement(category))
        for category in sorted(specific_agreement(units), key=str)
    }
    n_pairable = sum(1 for unit in units if sum(v is not None for v in unit) >= 2)
    return InterRaterReport(
        axis=axis,
        codebook_version=codebook_version,
        raters=ordered,
        n_units=len(units),
        n_pairable_units=n_pairable,
        alpha=alpha,
        cohen_kappa=kappa,
        per_class=per_class,
        gate=alpha_gate(alpha, n_pairable_units=n_pairable, minimum_overlap=minimum_overlap),
        n_superseded=superseded,
        minimum_overlap=minimum_overlap,
    )

plan_overlap

plan_overlap(
    *,
    expected_alpha=DEFAULT_EXPECTED_ALPHA,
    prevalence=DEFAULT_PREVALENCE,
    confidence=0.95,
    gate_resamples=GATE_RESAMPLES,
    scan_limit=SCAN_LIMIT,
)

The smallest overlap from which raters at expected_alpha clear the gate for good.

Every size from 2 to scan_limit is checked with clears_planned on its plug-in counts, and the answer is one more than the largest size that does not clear, so it clears at every checked size above it. With the defaults this is MINIMUM_OVERLAP.

Raises:

Type Description
ReviewError

expected_alpha at or below the gate (no overlap clears it), an invalid joint distribution, or no clearing size up to scan_limit.

Source code in src/calibrated_agent_evaluation/review/overlap.py
def plan_overlap(
    *,
    expected_alpha: float = DEFAULT_EXPECTED_ALPHA,
    prevalence: float = DEFAULT_PREVALENCE,
    confidence: float = 0.95,
    gate_resamples: int = GATE_RESAMPLES,
    scan_limit: int = SCAN_LIMIT,
) -> OverlapPlan:
    """The smallest overlap from which raters at ``expected_alpha`` clear the gate for good.

    Every size from 2 to ``scan_limit`` is checked with :func:`clears_planned` on its plug-in
    counts, and the answer is one more than the largest size that does not clear, so it clears
    at every checked size above it. With the defaults this is :data:`MINIMUM_OVERLAP`.

    Raises:
        ReviewError: ``expected_alpha`` at or below the gate (no overlap clears it), an invalid
            joint distribution, or no clearing size up to ``scan_limit``.
    """
    if expected_alpha <= ALPHA_LOWER_BOUND_GATE:
        raise ReviewError(
            f"plan_overlap: expected alpha {expected_alpha} does not exceed the gate's "
            f"{ALPHA_LOWER_BOUND_GATE}, so no overlap size clears it"
        )
    short = [
        n
        for n in range(2, scan_limit + 1)
        if not clears_planned(
            plug_in_counts(n, expected_alpha, prevalence),
            confidence=confidence,
            gate_resamples=gate_resamples,
        )
    ]
    if short and short[-1] == scan_limit:
        raise ReviewError(
            f"plan_overlap: at expected alpha {expected_alpha} the gate does not clear at "
            f"{scan_limit} shared units"
        )
    return OverlapPlan(
        n_units=(short[-1] + 1) if short else 2,
        expected_alpha=expected_alpha,
        prevalence=prevalence,
        gate_lower_bound=ALPHA_LOWER_BOUND_GATE,
        confidence=confidence,
        gate_resamples=gate_resamples,
        scan_limit=scan_limit,
    )

build_app

build_app(*args, **kwargs)

Lazily import and return the FastHTML app factory (keeps the [ui] extra optional).

Importing calibrated_agent_evaluation.review.app pulls in fasthtml (the [ui] extra); the rest of the package (labeling, session, renderers) has no such dependency, so callers doing pure label construction never need FastHTML installed. See calibrated_agent_evaluation.review.app.build_app.

Source code in src/calibrated_agent_evaluation/review/__init__.py
def build_app(*args: Any, **kwargs: Any) -> Any:
    """Lazily import and return the FastHTML app factory (keeps the ``[ui]`` extra optional).

    Importing ``calibrated_agent_evaluation.review.app`` pulls in ``fasthtml`` (the ``[ui]`` extra);
    the rest of the package (labeling, session, renderers) has no such dependency, so callers doing
    pure label construction never need FastHTML installed. See
    ``calibrated_agent_evaluation.review.app.build_app``.
    """
    from calibrated_agent_evaluation.review.app import build_app as _build_app

    return _build_app(*args, **kwargs)