calibrated_agent_evaluation.review¶
calibrated_agent_evaluation.review ¶
Friction-free trace-review surface.
The library's flagship "look at the data" surface: a thin FastHTML app whose primary screen
is a single-trace inline-label view (not a metrics dashboard). It reads
normalized Traces and writes human Labels only through the backend
interfaces (TraceStoreBackend / LabelBackend); no vendor SDK type appears in this
package. The surface exists to make human review fast enough that
error analysis stops being the step teams skip: hotkeys + optimistic auto-advance let a
competent labeler read + label 50 traces in under 30 minutes.
Public surface
build_app: the FastHTML application factory (app.py).ReviewSession: in-memory work-list / defer / undo / timing (session.py).WorkScope/build_work_list; which traces a session covers: resume per rater, plus trace / frame / cell / limit narrowing (session.py).SessionLog: the append-only JSONL record of verdicts and undos with per-trace timing (session_log.py).build_label+PASS_DEFAULT_CRITIQUE; critique-first label construction (labeling).Prefill/Suggestion/NoPrefill; the opt-in LLM pre-fill seam (prefill.py).RendererRegistry: the plugin-renderer seam keyed on trace metadata; construct with a default renderer,register(predicate, renderer)domain renderers,render(trace)guards a raising renderer back to the default (renderers).inter_rater_agreement/alpha_gate/newest_per_rater_unit; the gold-set agreement gate over double-coded human labels (agreement), andAgreementRecord/agreement_record_of, whatcalibrated-agent-evaluation label --agreementstores (record).CodebookFile/CodebookRecord/load_codebook_file; the failure-mode list the review form offers, hand-written or ratified bycalibrated-agent-evaluation cluster --ratify(record).plan_overlap/MINIMUM_OVERLAP; how many double-coded traces the agreement gate needs before it can sayusable(overlap).ReviewError/RatificationError/LabelSubmitError/AgreementDataError; the typed error family.
PASS_DEFAULT_CRITIQUE
module-attribute
¶
AgreementDataError ¶
Bases: ReviewError
The labels handed to the inter-rater agreement path cannot form an agreement set.
Fewer than two distinct raters, a rater with two labels on one unit, a non-human label under a named rater, or no label in the requested codebook version. Each would otherwise mean a silent choice about which labels the number measures.
LabelSubmitError ¶
Bases: ReviewError
A label write through the backend failed and could not be reconciled.
Wraps the underlying SchemaError/BackendError (chained with raise … from)
so the review session surfaces the failure and re-queues the trace as unlabeled rather than
presenting it as successfully labeled.
RatificationError ¶
Bases: ReviewError
An LLM-suggested label was submitted without an explicit human accept/override.
The human-gold boundary: an LLM Suggestion may pre-fill the form, but a
kind="human" label is refused unless a human ratified it. No LLM-origin label reaches
the gold path unratified.
ReviewError ¶
Bases: CalibratedAgentEvaluationDataError
Base class for every review-surface error.
NoPrefill ¶
The default: no LLM assist. Every trace is labeled from scratch by the human.
suggest ¶
Return None: no suggestion (the offline, zero-dependency default).
Prefill ¶
Suggestion
dataclass
¶
AgreementRecord
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
The gold-set agreement gate over one scope, one axis and one codebook version.
gate is usable only when alpha.lower reaches alpha_gate (the 0.80 bound) on at
least minimum_overlap pairable units; top_up_overlap / revise_codebook /
codebook_broken otherwise (see
alpha_gate). minimum_overlap is
None on a record written before the gate had a minimum. codebook_version None is the
unversioned bucket. trace_ids lists every trace with a unit at least one named rater coded,
and n_pairable_units how many units two or more coded. n_superseded counts the older
labels a rater's relabel replaced (newest label per rater and unit). cell / trace_id /
frame record the scope flags the command ran with.
Config:
frozen:Trueextra:forbid
Fields:
-
agreement_id(str) -
axis(AgreementAxis) -
codebook_version(str | None) -
raters(tuple[str, ...]) -
trace_ids(tuple[str, ...]) -
n_units(int) -
n_pairable_units(int) -
n_superseded(int) -
alpha(AgreementStat) -
cohen_kappa(AgreementStat | None) -
per_class(dict[str, AgreementStat]) -
gate(GateVerdict) -
alpha_gate(float) -
alpha_revise_floor(float) -
minimum_overlap(int | None) -
cell(dict[str, str] | None) -
trace_id(str | None) -
frame(str | None) -
provenance(ProvenanceStamp) -
schema_version(str)
AgreementStat
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
One agreement statistic with its two-sided percentile bootstrap interval over units.
Config:
frozen:Trueextra:forbid
Fields:
-
point(float) -
lower(float) -
upper(float) -
confidence(float) -
n_units(int) -
n_resamples(int) -
n_undefined(int) -
seed(int)
of
classmethod
¶
The stored form of a bootstrap interval.
Source code in src/calibrated_agent_evaluation/review/record.py
CodebookFile
pydantic-model
¶
CodebookMode
pydantic-model
¶
CodebookRecord
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
A ratified failure-mode vocabulary under one codebook version.
ratified_by names the human who ratified it and ratified_at when (UTC ISO-8601).
parent_version is the version this one follows, None for the first. n_tagged counts
the FAIL labels the ratification stamped with a mode, and source names the map file.
Config:
frozen:Trueextra:forbid
Fields:
-
codebook_id(str) -
version(str) -
modes(tuple[CodebookMode, ...]) -
ratified_by(str) -
ratified_at(str) -
parent_version(str | None) -
n_tagged(int) -
source(str | None) -
merged(dict[str, str]) -
provenance(ProvenanceStamp) -
schema_version(str)
Validators:
-
_check
RendererRegistry
dataclass
¶
Resolve a trace's renderer: the first registered predicate that matches, else the default.
Construct with the default renderer (the app wires the tool-call-tree renderer). Register
domain renderers with register. render is the safe entry point; it guards a
raising renderer back to the default and logs, so one bad plugin never ends the session.
register ¶
Register renderer for any trace where predicate(trace) is true.
Registration order is match order; the first matching predicate wins.
Source code in src/calibrated_agent_evaluation/review/renderers/base.py
resolve ¶
Return the first renderer whose predicate matches trace, else the default.
Note: a predicate that itself raises is treated as non-matching (guarded + logged) so a broken predicate cannot mask the default.
Source code in src/calibrated_agent_evaluation/review/renderers/base.py
render ¶
Render trace with the resolved renderer; fall back to the default on error.
A registered renderer that raises is caught and logged (logging.exception) and the
default renderer is used instead; the labeling session continues rather than crashing.
Use render_with_status to also learn that a fallback happened.
Source code in src/calibrated_agent_evaluation/review/renderers/base.py
render_with_status ¶
Like render, plus a message naming the failure when the default stood in.
The message is for the labeler's screen. A fallback that only reached the log looked, to the person labeling, exactly like a trace the plugin was never meant to render.
Source code in src/calibrated_agent_evaluation/review/renderers/base.py
LabeledEntry
dataclass
¶
One completed submission: kept for undo and the throughput summary.
span_labels holds the span-level defect labels written with the verdict, so undo can
retract everything the submission put in the store, not only the trace-level label.
ReviewSession
dataclass
¶
ReviewSession(
pending,
done=list(),
clock=time.monotonic,
_shown_id=None,
_shown_at=None,
_unlabeled_seconds=0.0,
)
The work-list for one labeling session.
pending is the ordered queue of trace_ids awaiting a verdict; done is the stack
of completed entries (undo pops the last). Construct with the ids returned by
LabelBackend.next_item / TraceStoreBackend.query in a deterministic order.
from_ids
classmethod
¶
Build a session from an ordered iterable of trace_ids (dedup, order-preserving).
Source code in src/calibrated_agent_evaluation/review/session.py
current ¶
The trace at the head of the work-list, or None when the queue is empty.
mark_shown ¶
Start timing trace_id (default: the queue head) as the trace on screen.
Called whenever a trace is rendered, with the id actually rendered: after navigation that is not the queue head, and timing the head instead charged the interval to the wrong trace. Time on the trace previously shown is banked. Re-rendering the trace already being timed (a refused submit) keeps its clock running.
Source code in src/calibrated_agent_evaluation/review/session.py
record ¶
Record label against its trace, remove it from the queue, stop its timer.
Raises ReviewError if the label's trace is not in the pending work-list; a label for
a trace the session isn't tracking is a caller bug, not something to swallow.
Source code in src/calibrated_agent_evaluation/review/session.py
last ¶
The most recently recorded entry (what undo would retract), or None.
attach_span_label ¶
Record a span-level label written alongside the most recent verdict.
Raises ReviewError if nothing has been recorded or the label is for another trace.
Source code in src/calibrated_agent_evaluation/review/session.py
defer ¶
Move trace_id (default: the current trace) to the end of the work-list.
Returns the new current trace. A no-op returning None when the queue is empty.
Source code in src/calibrated_agent_evaluation/review/session.py
undo ¶
Retract the most recently recorded label and re-queue its trace at the front.
Returns the retracted entry, or None when nothing has been labeled yet (empty-stack
no-op; undoing an empty history is a benign no-op, not an error).
Source code in src/calibrated_agent_evaluation/review/session.py
summary ¶
Session throughput.
labeled_seconds is the time on traces whose verdict stands; elapsed_seconds is all
on-trace time, including deferred, navigated-past, undone, and the trace on screen now.
Source code in src/calibrated_agent_evaluation/review/session.py
WorkScope
dataclass
¶
Which traces a review session covers (label --trace/--frame/--cell/--limit).
frame and cell narrow the backend query; trace_id names one trace; limit caps
the work-list after resume filtering, so limit=50 is 50 traces still to label. By default
a trace this rater already labeled at trace level is left out (resume); relabel keeps it.
SessionLog ¶
Appends session events to a JSONL file. Opens the file per write; holds no handle.
Source code in src/calibrated_agent_evaluation/review/session_log.py
label ¶
Record a verdict that was just stored, with the time spent on its trace.
Source code in src/calibrated_agent_evaluation/review/session_log.py
undo ¶
Record that the verdict for entry.trace_id was retracted.
InterRaterReport
dataclass
¶
InterRaterReport(
axis,
codebook_version,
raters,
n_units,
n_pairable_units,
alpha,
cohen_kappa,
per_class,
gate,
n_superseded=0,
minimum_overlap=MINIMUM_OVERLAP,
)
Agreement among the named raters on one axis, within one codebook version.
n_units counts units at least one named rater coded; n_pairable_units the ones two or
more coded, which are the only units alpha uses. cohen_kappa is None unless exactly two
raters are named. per_class has an entry for every category observed on a pairable unit.
n_superseded counts the older labels a rater's relabel replaced. minimum_overlap is the
pairable-unit count below which gate cannot be usable.
OverlapPlan
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
The minimum overlap for raters expected to agree at expected_alpha.
n_units is the smallest overlap from which the plug-in counts clear the gate, with the
margin for its resampling, at every size up to scan_limit.
Fields:
build_label ¶
build_label(
*,
trace_id,
outcome,
critique,
span_id=None,
failure_mode=None,
culprit_span=None,
manifestation_span=None,
handoff_implicated=None,
span_defect=False,
expected=None,
codebook_version=None,
from_suggestion=False,
ratified=False,
rater_id=None,
span_ids=None,
failure_modes=None,
)
Build a validated kind="human" Label from a review submission.
The arguments mirror the labeling form. from_suggestion/ratified carry the LLM-assist
provenance; rater_id names the human in session (the review app always passes one).
span_ids, when given, is the trace's span ids; every span the label names must be one.
failure_modes, when given and non-empty, is the codebook's failure-mode list: a FAIL must
name one of them and a PASS names none. The returned Label still passes through the
backend's validate on write.
Raises:
| Type | Description |
|---|---|
ReviewError
|
|
RatificationError
|
The values came from a |
Source code in src/calibrated_agent_evaluation/review/labeling.py
37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 | |
agreement_record_of ¶
Build the AgreementRecord for report; the id is a hash of its content.
Source code in src/calibrated_agent_evaluation/review/record.py
load_codebook_file ¶
Read a CodebookFile from JSON.
Raises:
| Type | Description |
|---|---|
ReviewError
|
The file cannot be read, is not JSON, or is not a valid codebook. |
Source code in src/calibrated_agent_evaluation/review/record.py
newest_per_rater_unit ¶
Keep each rater's newest label per (trace_id, span_id) unit, and the drop count.
labels must be oldest first. Labels with no rater_id pass through, since nothing says
two of them came from one person. Order is kept.
Source code in src/calibrated_agent_evaluation/review/relabel.py
build_work_list ¶
Resolve scope against the backend into the traces to review, in stable order.
This is the startup reachability probe too: an unreachable backend raises its typed
BackendError here, before any session opens. Raises BackendError for a
trace_id the backend does not hold, and ReviewError for a bad limit or a
narrowed scope (frame, cell or trace) that matches no trace (it would open onto "all done").
Source code in src/calibrated_agent_evaluation/review/session.py
alpha_gate ¶
Return the gold-set verdict for an alpha interval over n_pairable_units shared units.
Below minimum_overlap units coded by two or more raters the interval cannot show the
gold is usable (raters who agree on all 20 shared traces get a lower bound of 1.0), but it
can show it is not:
codebook_broken: alpha's upper bound is belowALPHA_REVISE_FLOOR.revise_codebook: alpha's upper bound is belowALPHA_LOWER_BOUND_GATE.top_up_overlap: otherwise; double-code more of the calibration gold.
From minimum_overlap up:
usable: the lower bound is >=ALPHA_LOWER_BOUND_GATE. The bound gates, not the point estimate.revise_codebook: the point estimate is in [ALPHA_REVISE_FLOOR, gate).codebook_broken: the point estimate is belowALPHA_REVISE_FLOOR.top_up_overlap: the point estimate clears the gate but the lower bound does not. The interval is too wide to tell, which is a sample-size problem, not a codebook one.
MINIMUM_OVERLAP comes from
plan_overlap.
Source code in src/calibrated_agent_evaluation/review/agreement.py
inter_rater_agreement ¶
inter_rater_agreement(
labels,
*,
raters,
codebook_version,
axis="outcome",
n_resamples=2000,
confidence=0.95,
seed=0,
minimum_overlap=MINIMUM_OVERLAP,
)
Return the named raters' agreement on axis within codebook_version.
Only kind="human" labels whose rater_id is in raters and whose codebook_version
matches enter (None selects the unversioned labels); every other label (an adjudicator's,
an unattributed one, another version's) is outside this agreement set by construction. A unit
is a (trace_id, span_id); a rater who labeled a unit twice counts with the newest label, so
labels must be oldest first. Every statistic is bootstrapped over the same seeded unit
resamples. Below minimum_overlap pairable units the gate is top_up_overlap unless
alpha's upper bound already rules the gold out (see alpha_gate).
Raises AgreementDataError for fewer than two distinct raters, a non-human label under
a named rater, or no matching label, and
UndefinedAgreementError when the raters
share no unit or only one category is observed.
Source code in src/calibrated_agent_evaluation/review/agreement.py
plan_overlap ¶
plan_overlap(
*,
expected_alpha=DEFAULT_EXPECTED_ALPHA,
prevalence=DEFAULT_PREVALENCE,
confidence=0.95,
gate_resamples=GATE_RESAMPLES,
scan_limit=SCAN_LIMIT,
)
The smallest overlap from which raters at expected_alpha clear the gate for good.
Every size from 2 to scan_limit is checked with clears_planned on its plug-in
counts, and the answer is one more than the largest size that does not clear, so it clears
at every checked size above it. With the defaults this is MINIMUM_OVERLAP.
Raises:
| Type | Description |
|---|---|
ReviewError
|
|
Source code in src/calibrated_agent_evaluation/review/overlap.py
build_app ¶
Lazily import and return the FastHTML app factory (keeps the [ui] extra optional).
Importing calibrated_agent_evaluation.review.app pulls in fasthtml (the [ui] extra);
the rest of the package (labeling, session, renderers) has no such dependency, so callers doing
pure label construction never need FastHTML installed. See
calibrated_agent_evaluation.review.app.build_app.