Skip to content

calibrated_agent_evaluation.run

calibrated_agent_evaluation.run

The run / CI-gate stage -- the run step of probe -> ... -> run -> improve.

The point where the flat per-trace Labels that score_traces and run_judge already produced become a CI pass/fail gate, a per-failure-mode trend report (prevalence-corrected rates with CIs, never a single aggregate accuracy number), a passive staleness warning, and drift detection (PSI/JSD/Wasserstein + judge ΔTPR/ΔTNR). It covers multiplicity control and confidence-treatment everywhere.

Pure methodology plane: this package imports no vendor SDK type (a static test guards it). It reuses the stats primitives (multiplicity/corrected_rate/wilson_lower), enforce_ci_gate, the scorer/judge outputs, assert_production_frame, and the drift block in drift -- it re-implements none of them. The only vendor import lives in the confined backends/dashboard.py adapter. The library gate returns a typed RunGateDecision and a persisted RunRecord; the CLI wires calibrated-agent-evaluation run --ci|--cadence onto it and maps the decision to a process exit code.

The record types, errors, store and per-mode aggregation load eagerly and need nothing beyond the base install. The gate, trend, drift, staleness and multiplicity functions need the stats extra, so they load lazily; backends/dashboard.py imports the report types under the langfuse extra alone.

JUDGE_CHECK_SCHEMA_VERSION module-attribute

JUDGE_CHECK_SCHEMA_VERSION = '1'

RUN_SCHEMA_VERSION module-attribute

RUN_SCHEMA_VERSION = '8'

RunError

Bases: CalibratedAgentEvaluationDataError

Base class for every run-stage refusal.

RunGateError

Bases: RunError

The CI gate refuses to produce a pass/fail decision.

Raised when an exploration-lane scorer is presented as a gate, or when the gate is asked to run over an empty input (no traces / no verdicts) rather than returning a vacuous passed=True.

RunStoreError

Bases: RunError

A run-record / dashboard-sink file read or write failed.

Wraps the underlying OSError / json.JSONDecodeError / csv failure so a filesystem/parse error surfaces inside the typed family rather than as a raw stdlib exception escaping the store's public surface ("wrap external errors at the boundary"). A stored schema_version != the running RUN_SCHEMA_VERSION also refuses here (no coercion).

JudgeCheckRecord pydantic-model

Bases: CalibratedAgentEvaluationModel

One re-check of a certified judge on fresh gold.

against_calib_id is the certifying record the drift is measured against. tpr and tnr are the judge's rates on the fresh gold with their 95% lower bounds; drift is the change from the certifying record, each with its interval. drifted is true when either change's interval lies wholly below zero. needed_fail and needed_pass are the per-class sizes check_size asked for.

Config:

  • frozen: True
  • extra: forbid

Fields:

check_id pydantic-field

check_id

judge_id pydantic-field

judge_id

model pydantic-field

model

prompt_version pydantic-field

prompt_version

judge_fingerprint pydantic-field

judge_fingerprint

against_calib_id pydantic-field

against_calib_id

check_set_id pydantic-field

check_set_id

n_pos pydantic-field

n_pos

n_neg pydantic-field

n_neg

tpr pydantic-field

tpr

tnr pydantic-field

tnr

drift pydantic-field

drift

drifted pydantic-field

drifted

needed_fail pydantic-field

needed_fail

needed_pass pydantic-field

needed_pass

agreement_alpha pydantic-field

agreement_alpha = None

trace_ids pydantic-field

trace_ids

schema_version pydantic-field

schema_version = JUDGE_CHECK_SCHEMA_VERSION

provenance pydantic-field

provenance

DiscoveryEnqueue dataclass

DiscoveryEnqueue(trace_id, frame='discovery')

A CI-failing trace routed into the labeling queue tagged as discovery.

frame is always "discovery": a trace the CI gate caught is non-random augmentation (not a random draw), so it grows the discovery set but must never enter production-frame estimation (the assert_production_frame guard enforces that downstream).

trace_id instance-attribute

trace_id

frame class-attribute instance-attribute

frame = 'discovery'

DriftMetric pydantic-model

Bases: CalibratedAgentEvaluationModel

One input-drift statistic, its permutation p-value, and whether it was flagged.

value is the statistic as method computes it (PSI over baseline-quantile bins, JSD over the same bins, Wasserstein divided by the pooled IQR). p_value is its permutation test against the same-distribution null. For PSI and Wasserstein, exceeded is True only when that p-value survives the BY-FDR correction across those two AND, where threshold is set, value is above it; threshold=None means significance alone decides. JSD is reported only: its threshold is None and exceeded is always False. Records written before JSD left the gate may carry a JSD flag from the older three-metric family.

Config:

  • frozen: True
  • extra: forbid

Fields:

name pydantic-field

name

value pydantic-field

value

threshold pydantic-field

threshold

exceeded pydantic-field

exceeded

p_value pydantic-field

p_value

method pydantic-field

method

InputDriftReport pydantic-model

Bases: CalibratedAgentEvaluationModel

Input-distribution drift: PSI + JSD + Wasserstein over one input dimension.

multiplicity is the BY-FDR family the gated metrics' p-values were corrected in (PSI and Wasserstein, in that order; JSD is reported outside it), and n_permutations / seed say how those p-values were drawn, so the flags can be recomputed from the record. n_baseline / n_fresh are the two sample sizes.

Config:

  • frozen: True
  • extra: forbid

Fields:

metrics pydantic-field

metrics

dimension pydantic-field

dimension

multiplicity pydantic-field

multiplicity = None

n_permutations pydantic-field

n_permutations = None

seed pydantic-field

seed = None

n_baseline pydantic-field

n_baseline = None

n_fresh pydantic-field

n_fresh = None

any_exceeded property

any_exceeded

True when any metric crossed its threshold (the staleness input).

JudgeDriftReport pydantic-model

Bases: CalibratedAgentEvaluationModel

Judge drift vs a frozen baseline: ΔTPR / ΔTNR, each with a CI.

Each delta is an Interval (point = the delta, lo/hi the CI) so the confidence treatment travels with the number. There is no Δκ: TPR and TNR describe a binary judge completely, and κ on a hunted calibration set also moves with the hunt's class mix, which says nothing about the judge.

Config:

  • frozen: True
  • extra: forbid

Fields:

judge_id pydantic-field

judge_id

delta_tpr pydantic-field

delta_tpr

delta_tnr pydantic-field

delta_tnr

ModeCounts pydantic-model

Bases: CalibratedAgentEvaluationModel

Per-mode confusion counts aggregated from the flat verdict/scorer Labels.

n_judged_fail is the judge/scorer FAIL count over the n_total items of this mode; the gate turns it into a pass-rate lower bound and the trend turns it into a corrected rate. calib_id names the CalibrationRecord that corrects this mode (its own judge).

Config:

  • frozen: True
  • extra: forbid

Fields:

mode pydantic-field

mode

n_total pydantic-field

n_total

n_judged_fail pydantic-field

n_judged_fail

calib_id pydantic-field

calib_id = None

ModeGate pydantic-model

Bases: CalibratedAgentEvaluationModel

One gated mode: its verdict counts, its pass-rate 95% lower bound and the bar it must clear.

lower_bound is the one-sided Wilson bound on (n_total - n_fail) / n_total; passed is lower_bound >= threshold. threshold is this mode's own bar (a per-mode override, or the gate's default).

Config:

  • frozen: True
  • extra: forbid

Fields:

mode pydantic-field

mode

n_total pydantic-field

n_total

n_fail pydantic-field

n_fail

lower_bound pydantic-field

lower_bound

threshold pydantic-field

threshold

passed pydantic-field

passed

ModeKey pydantic-model

Bases: CalibratedAgentEvaluationModel

A failure-mode key: failure_type alone, or (agent, failure_type) for a crew.

agent is None for single-agent and workflow-level modes.

Config:

  • frozen: True
  • extra: forbid

Fields:

failure_type pydantic-field

failure_type

agent pydantic-field

agent = None

MultiplicityResult pydantic-model

Bases: CalibratedAgentEvaluationModel

A corrected family's method + comparison count m + reject vector.

m is stated per run. family names the correction site ("drift_flags", "improve_guard", ...); method is Holm (a hard-reject guardrail) or fdr_by (exploratory flags).

Config:

  • frozen: True
  • extra: forbid

Fields:

family pydantic-field

family

method pydantic-field

method

m pydantic-field

m

reject pydantic-field

reject

RunDecision pydantic-model

Bases: CalibratedAgentEvaluationModel

A single tiered run decision -- a staleness warning or an exploratory flag.

inputs is validated non-empty at construction: a recorded decision with no inputs is refused rather than silently accepted (the EnforcementDecision precedent). Staleness decisions are always warning-tier and never flip the gate.

Config:

  • frozen: True
  • extra: forbid

Fields:

Validators:

  • _check_invariants

rule pydantic-field

rule

tier pydantic-field

tier

inputs pydantic-field

inputs

outcome pydantic-field

outcome

message pydantic-field

message

RunGateDecision pydantic-model

Bases: CalibratedAgentEvaluationModel

The hard CI pass/fail verdict.

passed is True iff every gated mode's pass-rate 95% lower bound cleared threshold; failing_modes names the modes that did not and n_modes counts the modes gated. modes holds each gated mode's counts and bound, in the order the gate checked them. Requiring every mode to pass is an intersection-union test, so the false-pass rate is capped at the per-mode alpha without a correction across modes, and none is recorded. decisions carries attached non-gating warnings (e.g. staleness): they are recorded but never flip passed.

Config:

  • frozen: True
  • extra: forbid

Fields:

passed pydantic-field

passed

failing_modes pydantic-field

failing_modes

threshold pydantic-field

threshold

n_modes pydantic-field

n_modes

decisions pydantic-field

decisions = ()

modes pydantic-field

modes = ()

RunRecord pydantic-model

Bases: CalibratedAgentEvaluationModel

The persisted, version-pinned audit artifact.

Records the run mode, the gate outcome, how many modes it gated, which failed and each mode's counts and lower bound, the judge/calibration ids and scorer lanes used, the per-mode corrected rates, the drift, every corrected family + its m, the tiered decisions, and the inputs that produced the decision. inputs is validated non-empty. Round-trips through store; schema_version refuses on mismatch (no coercion).

Config:

  • frozen: True
  • extra: forbid

Fields:

Validators:

  • _check_invariants

run_id pydantic-field

run_id

mode pydantic-field

mode

gate_passed pydantic-field

gate_passed

judge_ids pydantic-field

judge_ids

calib_ids pydantic-field

calib_ids

scorer_lanes pydantic-field

scorer_lanes

trend pydantic-field

trend

input_drift pydantic-field

input_drift

judge_drift pydantic-field

judge_drift

multiplicity pydantic-field

multiplicity

decisions pydantic-field

decisions

inputs pydantic-field

inputs

gate_n_modes pydantic-field

gate_n_modes = None

gate_failing_modes pydantic-field

gate_failing_modes = ()

gate_modes pydantic-field

gate_modes = ()

schema_version pydantic-field

schema_version = RUN_SCHEMA_VERSION

provenance pydantic-field

provenance

RunReport pydantic-model

Bases: CalibratedAgentEvaluationModel

The full cadence-run rollup -- the vendor-neutral artifact the dashboard sink publishes.

Holds the per-mode trend, the input/judge drift, every corrected family's multiplicity result, and the tiered decisions (staleness + flags). It carries no confidence-treatment-free headline number: the only rates it holds are the per-TrendPoint corrected-rate intervals.

Config:

  • frozen: True
  • extra: forbid

Fields:

run_id pydantic-field

run_id

mode pydantic-field

mode

trend pydantic-field

trend

input_drift pydantic-field

input_drift = None

judge_drift pydantic-field

judge_drift = ()

multiplicity pydantic-field

multiplicity = ()

decisions pydantic-field

decisions = ()

gate pydantic-field

gate = None

TrendPoint pydantic-model

Bases: CalibratedAgentEvaluationModel

One per failure mode: the Rogan-Gladen corrected rate + CI, beside the raw rate.

corrected_rate is the shared Interval: a refused correction (J<0.5 or non-identifiable) carries point=None + refusal_reason and the mode is still present with its widening interval -- never dropped, never a fabricated point.

raw_rate is the judge's uncorrected FAIL fraction on the same rate sample with its two-sided 95% Wilson interval: what a reader would conclude without calibration. It is reported so the correction's effect is visible, never as a substitute for corrected_rate; it stays populated when the correction is refused.

Config:

  • frozen: True
  • extra: forbid

Fields:

mode pydantic-field

mode

n_total pydantic-field

n_total

corrected_rate pydantic-field

corrected_rate

calib_id pydantic-field

calib_id = None

raw_rate pydantic-field

raw_rate

DriftThresholds dataclass

DriftThresholds(
    psi=DEFAULT_PSI_THRESHOLD,
    wasserstein=DEFAULT_WASSERSTEIN_THRESHOLD,
)

Severity bars for the gated metrics; None means significance alone flags the metric.

JSD has no bar: it is reported only and never flags.

psi class-attribute instance-attribute

psi = DEFAULT_PSI_THRESHOLD

wasserstein class-attribute instance-attribute

wasserstein = DEFAULT_WASSERSTEIN_THRESHOLD

CiGateRequest dataclass

CiGateRequest(
    labels,
    gold,
    judges=(),
    scorers=(),
    threshold=DEFAULT_PASSRATE_THRESHOLD,
    thresholds=dict(),
    run_id="run",
    decisions=(),
)

The inputs to one CI-gate run.

labels are the per-trace verdict/scorer results; gold maps each (trace_id, span_id) to its failure mode, or is a function giving each label's mode (see aggregate_by_mode; use the function when several modes' judges score one trace). judges + calib_records are the (judge, its CalibrationRecord) pairs whose certification is checked before gating; scorers are the code scorers whose lane is checked. threshold is the default pass-rate lower-bound bar; thresholds overrides it per mode. run_id labels the resulting decision/record.

labels instance-attribute

labels

gold instance-attribute

gold

judges class-attribute instance-attribute

judges = ()

scorers class-attribute instance-attribute

scorers = ()

threshold class-attribute instance-attribute

threshold = DEFAULT_PASSRATE_THRESHOLD

thresholds class-attribute instance-attribute

thresholds = field(default_factory=dict)

run_id class-attribute instance-attribute

run_id = 'run'

decisions class-attribute instance-attribute

decisions = ()

aggregate_by_mode

aggregate_by_mode(labels, gold, *, calib_by_mode=None)

Reduce verdict/scorer labels to per-ModeKey ModeCounts.

gold maps each (trace_id, span_id) to the failure mode it belongs to (the human-ratified assignment), or is a function giving each label's mode. A map holds one mode per target, so when several judges for different modes score one trace, pass the function: a map would put every verdict on the target under whichever mode it holds. A label with no mode (absent from the map, or None from the function) is skipped -- never silently folded into a wrong mode. FAIL is the positive class, so n_judged_fail is the count of outcome == "FAIL" verdicts per mode.

calib_by_mode optionally names the CalibrationRecord id that corrects each mode (its own judge); recorded on the counts so the trend stage can look up the right record. Returns an empty dict for an empty label set (the gate turns that into a refusal).

Source code in src/calibrated_agent_evaluation/run/aggregate.py
def aggregate_by_mode(
    labels: Sequence[Label],
    gold: ModeSource,
    *,
    calib_by_mode: Mapping[ModeKey, str] | None = None,
) -> dict[ModeKey, ModeCounts]:
    """Reduce verdict/scorer ``labels`` to per-:class:`ModeKey` :class:`ModeCounts`.

    ``gold`` maps each ``(trace_id, span_id)`` to the failure mode it belongs to (the human-ratified
    assignment), or is a function giving each label's mode. A map holds one mode per target, so
    when several judges for different modes score one trace, pass the function: a map would put
    every verdict on the target under whichever mode it holds. A label with no mode (absent from
    the map, or ``None`` from the function) is skipped -- never silently folded into a wrong mode.
    ``FAIL`` is the positive class, so ``n_judged_fail`` is the count of ``outcome == "FAIL"``
    verdicts per mode.

    ``calib_by_mode`` optionally names the ``CalibrationRecord`` id that corrects each mode (its own
    judge); recorded on the counts so the trend stage can look up the right record.
    Returns an empty dict for an empty label set (the gate turns that into a refusal).
    """
    calib_by_mode = calib_by_mode or {}
    totals: dict[ModeKey, int] = {}
    fails: dict[ModeKey, int] = {}
    mode_of = gold if callable(gold) else _from_map(gold)
    for label in labels:
        mode = mode_of(label)
        if mode is None:
            continue
        totals[mode] = totals.get(mode, 0) + 1
        if label.outcome == "FAIL":
            fails[mode] = fails.get(mode, 0) + 1

    return {
        mode: ModeCounts(
            mode=mode,
            n_total=n_total,
            n_judged_fail=fails.get(mode, 0),
            calib_id=calib_by_mode.get(mode),
        )
        for mode, n_total in totals.items()
    }

check_size

check_size(
    record, *, threshold=0.8, max_n=MAX_CHECK_PER_CLASS
)

The fresh gold per class (FAIL, PASS) a re-check of record's judge needs.

For each class, the smallest n at which a judge showing threshold (the count rounded up, so the shown rate is never below it) is flagged by the drift interval against the record, at n and every larger size up to max_n. At the planned size a judge that truly fell to the threshold is flagged about half the time, and one that fell further more often.

Raises:

Type Description
InfeasiblePlanError

No size up to max_n detects the drop for some class.

Source code in src/calibrated_agent_evaluation/run/judge_check.py
def check_size(
    record: CalibrationRecord, *, threshold: float = 0.80, max_n: int = MAX_CHECK_PER_CLASS
) -> tuple[int, int]:
    """The fresh gold per class (FAIL, PASS) a re-check of ``record``'s judge needs.

    For each class, the smallest ``n`` at which a judge showing ``threshold`` (the count rounded
    up, so the shown rate is never below it) is flagged by the drift interval against the record,
    at ``n`` and every larger size up to ``max_n``. At the planned size a judge that truly fell
    to the threshold is flagged about half the time, and one that fell further more often.

    Raises:
        InfeasiblePlanError: No size up to ``max_n`` detects the drop for some class.
    """
    tp = round(record.tpr.point * record.n_pos)
    tn = round(record.tnr.point * record.n_neg)
    return (
        _class_size(tp, record.n_pos, threshold, max_n),
        _class_size(tn, record.n_neg, threshold, max_n),
    )

drift_flagged

drift_flagged(drift)

Whether either rate's change has an interval wholly below zero.

Source code in src/calibrated_agent_evaluation/run/judge_check.py
def drift_flagged(drift: JudgeDriftReport) -> bool:
    """Whether either rate's change has an interval wholly below zero."""
    return drift.delta_tpr.hi < 0.0 or drift.delta_tnr.hi < 0.0

dump_judge_checks

dump_judge_checks(records, path, fmt='csv')

Write records to judge_check_record.{csv,json} under path.

Raises:

Type Description
RunStoreError

A bad fmt or any filesystem failure.

Source code in src/calibrated_agent_evaluation/run/judge_check.py
def dump_judge_checks(
    records: Sequence[JudgeCheckRecord], path: str | Path, fmt: str = "csv"
) -> None:
    """Write ``records`` to ``judge_check_record.{csv,json}`` under ``path``.

    Raises:
        RunStoreError: A bad ``fmt`` or any filesystem failure.
    """
    _STORE.dump(records, path, fmt)

load_judge_checks

load_judge_checks(path, fmt='csv')

Read judge check records back from path; a missing file is an empty store.

Raises:

Type Description
RunStoreError

A version mismatch or any read, parse or validation failure.

Source code in src/calibrated_agent_evaluation/run/judge_check.py
def load_judge_checks(path: str | Path, fmt: str = "csv") -> list[JudgeCheckRecord]:
    """Read judge check records back from ``path``; a missing file is an empty store.

    Raises:
        RunStoreError: A version mismatch or any read, parse or validation failure.
    """
    return _STORE.load(path, fmt)

run_record_of

run_record_of(
    report,
    *,
    inputs,
    judge_ids=(),
    scorer_lanes=(),
    seed=None,
    n_resamples=None,
)

Build the persisted RunRecord from a RunReport.

gate_passed comes from the report's gate (None for a cadence run with no gate), and the gate threshold that produced it is stamped on provenance.thresholds["gate"] so the record keeps it. calib_ids are the trend points' calibration ids, in order, without repeats. seed / n_resamples are what the trend's corrected rates were computed with; pass the values given to trend_report. The gate's attached decisions are recorded alongside the report's own.

Source code in src/calibrated_agent_evaluation/run/report.py
def run_record_of(
    report: RunReport,
    *,
    inputs: dict[str, str],
    judge_ids: tuple[str, ...] = (),
    scorer_lanes: tuple[str, ...] = (),
    seed: int | None = None,
    n_resamples: int | None = None,
) -> RunRecord:
    """Build the persisted :class:`RunRecord` from a :class:`RunReport`.

    ``gate_passed`` comes from the report's gate (``None`` for a cadence run with no gate), and the
    gate threshold that produced it is stamped on ``provenance.thresholds["gate"]`` so the record
    keeps it. ``calib_ids`` are the trend points' calibration ids, in order, without repeats.
    ``seed`` / ``n_resamples`` are what the trend's corrected rates were computed with; pass the
    values given to :func:`calibrated_agent_evaluation.run.trend_report`. The gate's attached
    decisions are recorded alongside the report's own.
    """
    gate = report.gate
    thresholds = {"gate": gate.threshold} if gate is not None else {}
    decisions = list(report.decisions)
    if gate is not None:
        decisions.extend(d for d in gate.decisions if d not in decisions)
    calib_ids = tuple(dict.fromkeys(tp.calib_id for tp in report.trend if tp.calib_id))
    return RunRecord(
        run_id=report.run_id,
        mode=report.mode,
        gate_passed=gate.passed if gate is not None else None,
        gate_n_modes=gate.n_modes if gate is not None else None,
        gate_failing_modes=gate.failing_modes if gate is not None else (),
        gate_modes=gate.modes if gate is not None else (),
        judge_ids=judge_ids,
        calib_ids=calib_ids,
        scorer_lanes=scorer_lanes,
        trend=report.trend,
        input_drift=report.input_drift,
        judge_drift=report.judge_drift,
        multiplicity=report.multiplicity,
        decisions=tuple(decisions),
        inputs=inputs,
        provenance=capture_provenance(seed=seed, n_resamples=n_resamples, thresholds=thresholds),
    )

dump_run_records

dump_run_records(records, path, fmt='csv')

Write records to run_record.{csv,json} under path.

Parameters:

Name Type Description Default
records Sequence[RunRecord]

The run records to persist.

required
path str | Path

The directory to write into.

required
fmt str

Which file a later load_run_records reads back (both are always written).

'csv'

Raises:

Type Description
RunStoreError

A bad fmt or any filesystem failure.

Source code in src/calibrated_agent_evaluation/run/store.py
def dump_run_records(records: Sequence[RunRecord], path: str | Path, fmt: str = "csv") -> None:
    """Write ``records`` to ``run_record.{csv,json}`` under ``path``.

    Args:
        records: The run records to persist.
        path: The directory to write into.
        fmt: Which file a later :func:`load_run_records` reads back (both are always written).

    Raises:
        RunStoreError: A bad ``fmt`` or any filesystem failure.
    """
    _STORE.dump(records, path, fmt)

load_run_records

load_run_records(path, fmt='csv')

Read run records back from path; a missing file is an empty store.

Parameters:

Name Type Description Default
path str | Path

The directory dump_run_records wrote.

required
fmt str

"csv" or "json".

'csv'

Returns:

Type Description
list[RunRecord]

The stored records.

Raises:

Type Description
RunStoreError

A schema-version mismatch, or any read, parse or validation failure.

Source code in src/calibrated_agent_evaluation/run/store.py
def load_run_records(path: str | Path, fmt: str = "csv") -> list[RunRecord]:
    """Read run records back from ``path``; a missing file is an empty store.

    Args:
        path: The directory :func:`dump_run_records` wrote.
        fmt: ``"csv"`` or ``"json"``.

    Returns:
        The stored records.

    Raises:
        RunStoreError: A schema-version mismatch, or any read, parse or validation failure.
    """
    return _STORE.load(path, fmt)

input_drift

input_drift(
    baseline,
    fresh,
    *,
    dimension="input",
    thresholds=None,
    jsd_baseline=None,
    jsd_fresh=None,
    bins=DEFAULT_DRIFT_BINS,
    n_permutations=DEFAULT_N_PERMUTATIONS,
    seed=0,
)

Report PSI + JSD + Wasserstein input drift between baseline and fresh.

baseline/fresh are raw-value samples. PSI bins them on the baseline's quantiles; Wasserstein is divided by the pooled IQR. JSD reads jsd_baseline/jsd_fresh when supplied (per-category counts, permuted as category labels); absent those it uses the PSI bins. Each metric gets a permutation p-value (n_permutations relabellings, seed). PSI and Wasserstein are corrected together with BY-FDR at q = 0.10 via flag_slices (family "drift_flags", m = 2), and either is exceeded when it survives that correction and clears its severity bar. JSD is reported with its p-value but sits outside the family and is never exceeded. Empty inputs, or fewer than 5 * bins values per side, raise StatisticsInputError -- never a spurious zero-drift.

Source code in src/calibrated_agent_evaluation/run/drift_report.py
def input_drift(
    baseline: Sequence[float],
    fresh: Sequence[float],
    *,
    dimension: str = "input",
    thresholds: DriftThresholds | None = None,
    jsd_baseline: Sequence[float] | None = None,
    jsd_fresh: Sequence[float] | None = None,
    bins: int = DEFAULT_DRIFT_BINS,
    n_permutations: int = DEFAULT_N_PERMUTATIONS,
    seed: int = 0,
) -> InputDriftReport:
    """Report PSI + JSD + Wasserstein input drift between ``baseline`` and ``fresh``.

    ``baseline``/``fresh`` are raw-value samples. PSI bins them on the baseline's quantiles;
    Wasserstein is divided by the pooled IQR. JSD reads ``jsd_baseline``/``jsd_fresh`` when supplied
    (per-category counts, permuted as category labels); absent those it uses the PSI bins. Each
    metric gets a permutation p-value (``n_permutations`` relabellings, ``seed``). PSI and
    Wasserstein are corrected together with BY-FDR at q = 0.10 via
    :func:`calibrated_agent_evaluation.run.flag_slices` (family ``"drift_flags"``, ``m = 2``), and
    either is ``exceeded`` when it survives that correction and clears its severity bar. JSD is
    reported with its p-value but sits outside the family and is never ``exceeded``. Empty inputs,
    or fewer than ``5 * bins`` values per side, raise :class:`StatisticsInputError` -- never a
    spurious zero-drift.
    """
    thresholds = thresholds or DriftThresholds()
    psi_value = psi(baseline, fresh, bins=bins)
    psi_p = psi_permutation_p(baseline, fresh, bins=bins, n_perm=n_permutations, seed=seed)

    if jsd_baseline is not None and jsd_fresh is not None:
        jsd_value = jensen_shannon(jsd_baseline, jsd_fresh)
        k = max(len(jsd_baseline), len(jsd_fresh))

        def _jsd_counts(b: np.ndarray, f: np.ndarray) -> float:
            return jensen_shannon(
                np.bincount(b.astype(int), minlength=k).tolist(),
                np.bincount(f.astype(int), minlength=k).tolist(),
            )

        jsd_p = permutation_p(
            _jsd_counts,
            _expand_counts(jsd_baseline, name="jsd_baseline"),
            _expand_counts(jsd_fresh, name="jsd_fresh"),
            n_perm=n_permutations,
            seed=seed,
        )
        jsd_method = METHOD_JSD_COUNTS
    else:
        jsd_value = jensen_shannon(*quantile_histogram(baseline, fresh, bins=bins))
        jsd_p = permutation_p(
            lambda b, f: jsd_of_samples(b, f, bins=bins),
            baseline,
            fresh,
            n_perm=n_permutations,
            seed=seed,
        )
        jsd_method = METHOD_JSD_BINNED

    wass_value = wasserstein_normalized(baseline, fresh)
    wass_p = permutation_p(
        wasserstein_normalized_arrays, baseline, fresh, n_perm=n_permutations, seed=seed
    )

    family = flag_slices([psi_p, wass_p], family="drift_flags")
    rej_psi, rej_wass = family.reject
    metrics = (
        DriftMetric(
            name="psi",
            value=psi_value,
            threshold=thresholds.psi,
            exceeded=_flag(psi_value, thresholds.psi, rej_psi),
            p_value=psi_p,
            method=METHOD_PSI,
        ),
        DriftMetric(
            name="jsd",
            value=jsd_value,
            threshold=None,
            exceeded=False,
            p_value=jsd_p,
            method=jsd_method,
        ),
        DriftMetric(
            name="wasserstein",
            value=wass_value,
            threshold=thresholds.wasserstein,
            exceeded=_flag(wass_value, thresholds.wasserstein, rej_wass),
            p_value=wass_p,
            method=METHOD_WASSERSTEIN,
        ),
    )
    return InputDriftReport(
        metrics=metrics,
        dimension=dimension,
        multiplicity=family,
        n_permutations=n_permutations,
        seed=seed,
        n_baseline=len(baseline),
        n_fresh=len(fresh),
    )

judge_drift

judge_drift(before, after)

Report ΔTPR / ΔTNR between two calibration records on independent samples, each with a CI.

Reconstructs the TPR/TNR counts from each record (round(tpr.point * n_pos) etc.) and takes the difference-of-proportions CI for two independent samples (stats.drift.delta_proportion_ci). A judge whose TPR/TNR is unchanged yields deltas straddling 0; a degraded judge yields a delta whose CI sits below 0 -- the signal that the judge needs re-calibration. before/after are the two frozen calibration records the caller supplies (the trickle that produces them in production is a later active-cadence feature).

The independent interval is wrong when the two records share items, because the same item judged twice makes the comparison paired. Sharing is refused with SampleMisuseError when the records name the same hunted set or any member trace id in common; use judge_drift_paired with both runs' verdicts instead. A record without member trace ids (built outside certify) is taken to be on its own sample.

Source code in src/calibrated_agent_evaluation/run/drift_report.py
def judge_drift(before: CalibrationRecord, after: CalibrationRecord) -> JudgeDriftReport:
    """Report ΔTPR / ΔTNR between two calibration records on independent samples, each with a CI.

    Reconstructs the TPR/TNR counts from each record (``round(tpr.point * n_pos)`` etc.) and takes
    the difference-of-proportions CI for two independent samples
    (``stats.drift.delta_proportion_ci``). A judge whose TPR/TNR is unchanged yields deltas
    straddling 0; a degraded judge yields a delta whose CI sits below 0 -- the signal that the judge
    needs re-calibration. ``before``/``after`` are the two frozen calibration records the caller
    supplies (the trickle that produces them in production is a later active-cadence feature).

    The independent interval is wrong when the two records share items, because the same item
    judged twice makes the comparison paired. Sharing is refused with
    :class:`SampleMisuseError` when the records name the same hunted set or any member trace id in
    common; use :func:`judge_drift_paired` with both runs' verdicts instead. A record without
    member trace ids (built outside :func:`~calibrated_agent_evaluation.calibrate.certify`) is taken
    to be on its own sample.
    """
    shared = set(before.trace_ids) & set(after.trace_ids)
    if before.hunted_set_id == after.hunted_set_id or shared:
        what = (
            f"the same hunted set {after.hunted_set_id!r}"
            if before.hunted_set_id == after.hunted_set_id
            else f"{len(shared)} shared trace id(s)"
        )
        raise SampleMisuseError(
            f"judge_drift: the before and after records share items ({what}), so the comparison "
            "is paired and an independent-samples interval is wrong; call judge_drift_paired with "
            "both runs' verdicts on the shared items"
        )
    tp_before = round(before.tpr.point * before.n_pos)
    tp_after = round(after.tpr.point * after.n_pos)
    tn_before = round(before.tnr.point * before.n_neg)
    tn_after = round(after.tnr.point * after.n_neg)
    return JudgeDriftReport(
        judge_id=after.judge_id,
        delta_tpr=_delta(tp_after, after.n_pos, tp_before, before.n_pos),
        delta_tnr=_delta(tn_after, after.n_neg, tn_before, before.n_neg),
    )

judge_drift_paired

judge_drift_paired(before, after, *, judge_id)

Report ΔTPR / ΔTNR between two runs of a judge over the same items, each with a CI.

before and after are (human gold, judge verdict) pairs for the same items, matched on (trace_id, span_id). Within each gold class, an item judged correctly by one run and not the other is a discordant pair; the change in TPR (over gold FAIL items) and TNR (over gold PASS items) gets Newcombe's (1998) paired hybrid-score interval, which uses that pairing instead of treating the two runs as independent samples.

Raises SampleMisuseError when the two runs do not cover exactly the same items or an item's gold label differs between them, and StatisticsInputError when a gold class is empty.

Source code in src/calibrated_agent_evaluation/run/drift_report.py
def judge_drift_paired(
    before: Sequence[LabelPair], after: Sequence[LabelPair], *, judge_id: str
) -> JudgeDriftReport:
    """Report ΔTPR / ΔTNR between two runs of a judge over the same items, each with a CI.

    ``before`` and ``after`` are (human gold, judge verdict) pairs for the same items, matched on
    ``(trace_id, span_id)``. Within each gold class, an item judged correctly by one run and not the
    other is a discordant pair; the change in TPR (over gold FAIL items) and TNR (over gold PASS
    items) gets Newcombe's (1998) paired hybrid-score interval, which uses that pairing instead of
    treating the two runs as independent samples.

    Raises :class:`SampleMisuseError` when the two runs do not cover exactly the same items or an
    item's gold label differs between them, and :class:`StatisticsInputError` when a gold class is
    empty.
    """

    def _index(pairs: Sequence[LabelPair], side: str) -> dict[tuple[str, str | None], LabelPair]:
        out: dict[tuple[str, str | None], LabelPair] = {}
        for pair in pairs:
            key = (pair.gold.trace_id, pair.gold.span_id)
            if key in out:
                raise SampleMisuseError(f"judge_drift_paired: {side} has item {key} twice")
            out[key] = pair
        return out

    b_items, a_items = _index(before, "before"), _index(after, "after")
    if b_items.keys() != a_items.keys():
        missing = len(b_items.keys() ^ a_items.keys())
        raise SampleMisuseError(
            f"judge_drift_paired: the runs must cover the same items; {missing} item(s) are in "
            "only one of them"
        )
    by_class: dict[str, tuple[list[bool], list[bool]]] = {"FAIL": ([], []), "PASS": ([], [])}
    for key, b_pair in b_items.items():
        a_pair = a_items[key]
        gold = b_pair.gold.outcome
        if a_pair.gold.outcome != gold:
            raise SampleMisuseError(
                f"judge_drift_paired: item {key} has gold {gold!r} before and "
                f"{a_pair.gold.outcome!r} after"
            )
        if gold not in by_class:
            raise StatisticsInputError(
                f"judge_drift_paired: gold outcome must be FAIL or PASS, got {gold!r}"
            )
        before_ok, after_ok = by_class[gold]
        before_ok.append(b_pair.verdict.outcome == gold)
        after_ok.append(a_pair.verdict.outcome == gold)
    for gold, (ok, _) in by_class.items():
        if not ok:
            raise StatisticsInputError(f"judge_drift_paired: no gold {gold} items to compare")
    return JudgeDriftReport(
        judge_id=judge_id,
        delta_tpr=_paired_delta(*by_class["FAIL"]),
        delta_tnr=_paired_delta(*by_class["PASS"]),
    )

flag_slices

flag_slices(pvals, *, family='drift_flags')

Flag exploratory cross-slice signals via Benjamini-Yekutieli FDR at q = 0.10.

pvals are the per-slice p-values (one per watched slice); family names the flag family ("drift_flags", "regression_flags", ...). Returns a MultiplicityResult whose reject[i] is True for a slice that survives the BY correction (a genuine flag) and whose m == the number of slices watched (stated per run). Delegates to the multiplicity primitive via apply_multiplicity; fdr_bh is not offered here -- correlated slices forbid it.

Source code in src/calibrated_agent_evaluation/run/flags.py
def flag_slices(pvals: Sequence[float], *, family: str = "drift_flags") -> MultiplicityResult:
    """Flag exploratory cross-slice signals via Benjamini-Yekutieli FDR at q = 0.10.

    ``pvals`` are the per-slice p-values (one per watched slice); ``family`` names the flag family
    ("drift_flags", "regression_flags", ...). Returns a
    :class:`~calibrated_agent_evaluation.run.report.MultiplicityResult` whose ``reject[i]`` is True
    for a slice that survives the BY correction (a genuine flag) and whose ``m`` == the number of
    slices watched (stated per run). Delegates to the multiplicity primitive via
    :func:`apply_multiplicity`; ``fdr_bh`` is not offered here -- correlated slices forbid it.
    """
    return apply_multiplicity(pvals, family=family, method="fdr_by", alpha=FLAG_FDR_Q)

discovery_traces

discovery_traces(request)

Return the CI-failing traces as discovery-tagged labeling-queue entries.

A trace whose verdict is FAIL is CI-augmentation, not a random draw: it is enqueued tagged frame="discovery" (the caller does the enqueue; the gate supplies the frame-tagged set). The frame tag is what keeps it out of production-frame rate estimation (enforced by the assert_production_frame guard on the trend side). A passing trace is not enqueued.

Source code in src/calibrated_agent_evaluation/run/gate.py
def discovery_traces(request: CiGateRequest) -> tuple[DiscoveryEnqueue, ...]:
    """Return the CI-failing traces as discovery-tagged labeling-queue entries.

    A trace whose verdict is ``FAIL`` is CI-augmentation, not a random draw: it is enqueued tagged
    ``frame="discovery"`` (the caller does the enqueue; the gate supplies the frame-tagged set). The
    frame tag is what keeps it out of production-frame rate estimation (enforced by the
    ``assert_production_frame`` guard on the trend side). A passing trace is not enqueued.
    """
    failing_ids = dict.fromkeys(
        label.trace_id for label in request.labels if label.outcome == "FAIL"
    )
    return tuple(DiscoveryEnqueue(trace_id=tid) for tid in failing_ids)

gate_mode_counts

gate_mode_counts(
    counts,
    *,
    threshold=DEFAULT_PASSRATE_THRESHOLD,
    thresholds=None,
    decisions=(),
)

Gate per-mode counts on each mode's pass-rate 95% lower bound.

The gating step of run_ci_gate, for a caller that has already reduced its verdicts to one observation per trace and mode (for example human labels, where one mode-free PASS is a pass for every mode). Admits nothing: judge and scorer admission is run_ci_gate's.

Parameters:

Name Type Description Default
counts Mapping[ModeKey, ModeCounts]

Each mode's verdict total and FAIL count.

required
threshold float

The default bar every mode's lower bound must clear.

DEFAULT_PASSRATE_THRESHOLD
thresholds Mapping[ModeKey, float] | None

Per-mode overrides of threshold.

None
decisions Sequence[RunDecision]

Non-gating decisions to attach (recorded, never flip passed).

()

Returns:

Type Description
RunGateDecision

Whether every mode passed, which failed, and each mode's counts and bound.

Raises:

Type Description
RunGateError

counts is empty, or a mode has no verdicts.

Source code in src/calibrated_agent_evaluation/run/gate.py
def gate_mode_counts(
    counts: Mapping[ModeKey, ModeCounts],
    *,
    threshold: float = DEFAULT_PASSRATE_THRESHOLD,
    thresholds: Mapping[ModeKey, float] | None = None,
    decisions: Sequence[RunDecision] = (),
) -> RunGateDecision:
    """Gate per-mode counts on each mode's pass-rate 95% lower bound.

    The gating step of :func:`run_ci_gate`, for a caller that has already reduced its verdicts to
    one observation per trace and mode (for example human labels, where one mode-free PASS is a
    pass for every mode). Admits nothing: judge and scorer admission is :func:`run_ci_gate`'s.

    Args:
        counts: Each mode's verdict total and FAIL count.
        threshold: The default bar every mode's lower bound must clear.
        thresholds: Per-mode overrides of ``threshold``.
        decisions: Non-gating decisions to attach (recorded, never flip ``passed``).

    Returns:
        Whether every mode passed, which failed, and each mode's counts and bound.

    Raises:
        RunGateError: ``counts`` is empty, or a mode has no verdicts.
    """
    if not counts:
        raise RunGateError(
            "no verdict mapped to a measured failure mode; refusing a vacuous pass (a gate over "
            "zero verdicts is not a green build)"
        )
    overrides = thresholds or {}
    ordered = sorted(counts.values(), key=lambda c: (c.mode.failure_type, c.mode.agent or ""))
    gated: list[ModeGate] = []
    for c in ordered:
        if c.n_total <= 0:
            raise RunGateError(f"mode {c.mode!r} has no verdicts to gate on")
        bar = overrides.get(c.mode, threshold)
        # The bound, not the point. Every mode must clear its own bar; requiring all of them is
        # what caps P(false pass) at the per-mode alpha, so the per-mode bound is used as is.
        lb95 = wilson_lower(c.n_total - c.n_judged_fail, c.n_total)
        gated.append(
            ModeGate(
                mode=c.mode,
                n_total=c.n_total,
                n_fail=c.n_judged_fail,
                lower_bound=lb95,
                threshold=bar,
                passed=lb95 >= bar,
            )
        )
    failing = tuple(g.mode for g in gated if not g.passed)
    return RunGateDecision(
        passed=not failing,
        failing_modes=failing,
        threshold=threshold,
        n_modes=len(gated),
        decisions=tuple(decisions),
        modes=tuple(gated),
    )

run_ci_gate

run_ci_gate(request)

Run the CI gate and return a RunGateDecision.

Refuses an empty input rather than passing vacuously. Admits inputs, aggregates, and gates each mode on its pass-rate 95% lower bound. passed is True iff every gated mode clears its threshold (an intersection-union test, so no correction across modes); failing_modes names those that do not and n_modes counts the modes gated. Attached decisions (e.g. a staleness warning) are recorded but never flip passed.

Parameters:

Name Type Description Default
request CiGateRequest

The verdicts, thresholds and options for the gate.

required

Returns:

Type Description
RunGateDecision

Whether every mode passed, which failed, and any attached decisions.

Raises:

Type Description
RunGateError

The input is empty, or a scorer is not allowed to gate.

UncalibratedScoreError

A judge verdict comes from a judge with no certifying record.

Source code in src/calibrated_agent_evaluation/run/gate.py
def run_ci_gate(request: CiGateRequest) -> RunGateDecision:
    """Run the CI gate and return a :class:`RunGateDecision`.

    Refuses an empty input rather than passing vacuously. Admits inputs, aggregates, and gates each
    mode on its pass-rate 95% lower bound. ``passed`` is True iff every gated mode clears its
    threshold (an intersection-union test, so no correction across modes); ``failing_modes`` names
    those that do not and ``n_modes`` counts the modes gated. Attached ``decisions`` (e.g. a
    staleness warning) are recorded but never flip ``passed``.

    Args:
        request: The verdicts, thresholds and options for the gate.

    Returns:
        Whether every mode passed, which failed, and any attached decisions.

    Raises:
        RunGateError: The input is empty, or a scorer is not allowed to gate.
        UncalibratedScoreError: A judge verdict comes from a judge with no certifying record.
    """
    if not request.labels:
        raise RunGateError(
            "run_ci_gate: no verdicts to gate on -- refusing a vacuous pass over an empty run "
            "(a gate over zero traces is not a green build)"
        )

    _admit_inputs(request)

    counts = aggregate_by_mode(request.labels, request.gold)
    if not counts:
        raise RunGateError(
            "run_ci_gate: no verdict mapped to a measured failure mode (check the gold map); "
            "refusing a vacuous pass"
        )

    return gate_mode_counts(
        counts,
        threshold=request.threshold,
        thresholds=request.thresholds,
        decisions=request.decisions,
    )

apply_multiplicity

apply_multiplicity(pvals, *, family, method, alpha=0.05)

Correct pvals for multiplicity and package the method + m + reject vector.

family names the correction site ("drift_flags", "improve_guard", ...). method is "holm" (a guardrail family where one rejection blocks the change) or "fdr_by" (exploratory dependence-aware FDR); fdr_bh and any other method are refused by the underlying primitive (StatisticsInputError), because slices are built from overlapping traces and BH assumes an independence they do not have. m == len(pvals) is recorded so the run is auditable.

Source code in src/calibrated_agent_evaluation/run/multi.py
def apply_multiplicity(
    pvals: Sequence[float],
    *,
    family: str,
    method: MultMethod,
    alpha: float = 0.05,
) -> MultiplicityResult:
    """Correct ``pvals`` for multiplicity and package the method + ``m`` + reject vector.

    ``family`` names the correction site ("drift_flags", "improve_guard", ...). ``method`` is
    ``"holm"`` (a guardrail family where one rejection blocks the change) or ``"fdr_by"``
    (exploratory dependence-aware FDR); ``fdr_bh`` and any other method are refused by the
    underlying primitive (:class:`StatisticsInputError`), because slices are built from overlapping
    traces and BH assumes an independence they do not have. ``m`` == ``len(pvals)`` is recorded so
    the run is auditable.
    """
    reject, m = multiplicity(pvals, method, alpha=alpha)
    return MultiplicityResult(family=family, method=method, m=m, reject=tuple(reject))

check_staleness

check_staleness(judge, record, input_drift=None)

Return a warning-tier staleness RunDecision, or None.

Warns when EITHER trigger fires:

  • version drift -- the record no longer certifies judge (its (model, prompt_version) changed, or it is not CERTIFIED), read via certifies; or
  • input drift -- input_drift is supplied and any of its metrics crossed its threshold.

Returns None when the version matches AND drift is sub-threshold (or absent). The decision is always warning tier -- staleness never hard-refuses (it is passive; a caller must not let flip a gate). Never raises.

Source code in src/calibrated_agent_evaluation/run/staleness.py
def check_staleness(
    judge: Judge,
    record: CalibrationRecord,
    input_drift: InputDriftReport | None = None,
) -> RunDecision | None:
    """Return a ``warning``-tier staleness :class:`RunDecision`, or ``None``.

    Warns when EITHER trigger fires:

      * **version drift** -- the record no longer certifies ``judge`` (its ``(model,
        prompt_version)`` changed, or it is not CERTIFIED), read via ``certifies``; or
      * **input drift** -- ``input_drift`` is supplied and any of its metrics crossed its threshold.

    Returns ``None`` when the version matches AND drift is sub-threshold (or absent). The decision
    is always ``warning`` tier -- staleness never hard-refuses (it is passive; a caller must not let
    flip a gate). Never raises.
    """
    version_drifted = not certifies(record, judge)
    drift_exceeded = input_drift is not None and input_drift.any_exceeded
    if not (version_drifted or drift_exceeded):
        return None

    reasons: list[str] = []
    if version_drifted:
        reasons.append(
            f"judge {judge.judge_id!r} ({judge.model}@{judge.prompt_version}) no longer matches "
            f"its calibration record ({record.model}@{record.prompt_version}, "
            f"result={record.result}): "
            + (pin_mismatch(record, judge) or "the record is not CERTIFIED")
        )
    if drift_exceeded:
        assert input_drift is not None  # narrowed by drift_exceeded  # noqa: S101
        crossed = [m.name for m in input_drift.metrics if m.exceeded]
        reasons.append(
            f"input-distribution drift on {input_drift.dimension!r} crossed threshold "
            f"({', '.join(crossed)})"
        )

    return RunDecision(
        rule="RUN-4",
        tier="warning",
        inputs={
            "judge_id": judge.judge_id,
            "version_drifted": str(version_drifted),
            "drift_exceeded": str(drift_exceeded),
        },
        outcome="warned",
        message=(
            f"suite for judge {judge.judge_id!r} may be stale and need re-calibration: "
            + "; ".join(reasons)
            + ". This warning never changes the gate; "
            "`calibrated-agent-evaluation run --fail-on-drift` gives it its own exit code."
        ),
    )

assert_production_rate_frame

assert_production_rate_frame(rows)

Refuse a discovery-frame row from entering a headline (production) rate.

Reuses assert_production_frame: a CI-augmented (discovery) trace is non-random augmentation, so it grows the discovery set but must never be counted toward a production-representative headline rate (raises FrameContaminationError). The run stage does not re-implement the guard -- it wires its rate path through the existing one.

Source code in src/calibrated_agent_evaluation/run/trend.py
def assert_production_rate_frame(rows: Sequence[FrameRow]) -> None:
    """Refuse a discovery-frame row from entering a headline (production) rate.

    Reuses :func:`calibrated_agent_evaluation.sampling.assert_production_frame`: a CI-augmented
    (discovery) trace is non-random augmentation, so it grows the discovery set but must never be
    counted toward a production-representative headline rate (raises ``FrameContaminationError``).
    The run stage does not re-implement the guard -- it wires its rate path through the existing
    one.
    """
    assert_production_frame(rows)

trend_report

trend_report(
    counts,
    calib_by_mode,
    rate_samples,
    *,
    seed=0,
    n_resamples=DEFAULT_N_RESAMPLES,
    rate_trace_ids,
    rate_cluster_ids=None,
)

One TrendPoint per mode: the corrected rate + CI, keyed per failure mode.

counts is the per-mode aggregation (keyed by ModeKey); calib_by_mode gives each mode its own CalibrationRecord; rate_samples gives each mode the judge verdicts (1=FAIL, 0=PASS) over its SEPARATE production-representative sample that corrected_rate corrects. A mode absent from calib_by_mode or rate_samples is skipped (it cannot be corrected without its own judge + rate sample) rather than mis-reported.

rate_trace_ids gives, per mode, the trace id of each rate-sample verdict, which corrected_rate checks against the calibration gold; a mode without them is refused there. rate_cluster_ids gives, per mode, the trace id of each rate-sample verdict when a trace contributes several, so that mode's interval accounts for the clustering. A mode absent from it is taken as independent verdicts.

A refused correction (J < 0.5, non-identifiable, a rate sample below the minimum, or a record whose gold is below the coverage floor) surfaces as an Interval with point=None + refusal_reason; the mode is still emitted, so the gate and the other modes still report. The report contains no single blended aggregate accuracy scalar: the only rates are the per-mode corrected-rate intervals.

Source code in src/calibrated_agent_evaluation/run/trend.py
def trend_report(
    counts: Mapping[ModeKey, ModeCounts],
    calib_by_mode: Mapping[ModeKey, CalibrationRecord],
    rate_samples: Mapping[ModeKey, Sequence[int]],
    *,
    seed: int = 0,
    n_resamples: int = DEFAULT_N_RESAMPLES,
    rate_trace_ids: Mapping[ModeKey, Sequence[str]],
    rate_cluster_ids: Mapping[ModeKey, Sequence[str]] | None = None,
) -> list[TrendPoint]:
    """One :class:`TrendPoint` per mode: the corrected rate + CI, keyed per failure mode.

    ``counts`` is the per-mode aggregation (keyed by
    :class:`~calibrated_agent_evaluation.run.report.ModeKey`); ``calib_by_mode`` gives each mode its
    own CalibrationRecord; ``rate_samples`` gives each mode the judge verdicts (1=FAIL, 0=PASS) over
    its SEPARATE production-representative sample
    that ``corrected_rate`` corrects. A mode absent from ``calib_by_mode`` or ``rate_samples`` is
    skipped (it cannot be corrected without its own judge + rate sample) rather than mis-reported.

    ``rate_trace_ids`` gives, per mode, the trace id of each rate-sample verdict, which
    ``corrected_rate`` checks against the calibration gold; a mode without them is refused there.
    ``rate_cluster_ids`` gives, per mode, the trace id of each rate-sample verdict when a trace
    contributes several, so that mode's interval accounts for the clustering.
    A mode absent from it is taken as independent verdicts.

    A refused correction (J < 0.5, non-identifiable, a rate sample below the minimum, or a record
    whose gold is below the coverage floor) surfaces as an ``Interval`` with ``point=None`` +
    ``refusal_reason``; the mode is still emitted, so the gate and the other modes still report.
    The report contains no single blended aggregate accuracy scalar:
    the only rates are the per-mode corrected-rate intervals.
    """
    points: list[TrendPoint] = []
    for key, mode_counts in counts.items():
        record = calib_by_mode.get(key)
        verdicts = rate_samples.get(key)
        if record is None or verdicts is None:
            continue
        # Reuse corrected_rate; it returns a copy of the record carrying the
        # Interval (point + CI, or point=None + refusal_reason when refused). We surface that
        # Interval verbatim -- the widening-bound-on-refusal behavior is the correction's contract.
        rate_set_id = f"rate-{mode_counts.mode.failure_type}-{mode_counts.mode.agent or 'workflow'}"
        try:
            scored = corrected_rate(
                record,
                verdicts,
                rate_set_id=rate_set_id,
                rate_trace_ids=rate_trace_ids.get(key, ()),
                seed=seed,
                n_resamples=n_resamples,
                rate_cluster_ids=None if rate_cluster_ids is None else rate_cluster_ids.get(key),
            )
        except (RateSampleTooSmallError, RateUnsafeError) as exc:
            # The mode stays in the report with the reason; [0, 1] claims nothing.
            why = (
                "rate sample too small"
                if isinstance(exc, RateSampleTooSmallError)
                else ("gold below the coverage floor")
            )
            interval: Interval | None = Interval(
                point=None,
                lo=0.0,
                hi=1.0,
                refusal_reason=f"{why}: {exc}",
                method=METHOD_RG_JEFFREYS,
                level=RG_LEVEL,
                sided="two-sided",
                n=len(verdicts),
            )
        else:
            interval = scored.corrected_rate
        if interval is None:  # defensive: corrected_rate always attaches one; never leak a None
            interval = Interval(
                point=None,
                lo=0.0,
                hi=1.0,
                refusal_reason="no interval produced",
                method=METHOD_RG_JEFFREYS,
                level=RG_LEVEL,
                sided="two-sided",
                n=len(verdicts),
            )
        points.append(
            TrendPoint(
                mode=mode_counts.mode,
                n_total=mode_counts.n_total,
                corrected_rate=interval,
                calib_id=record.calib_id,
                raw_rate=raw_rate_interval(verdicts),
            )
        )
    return points