Skip to content

calibrated_agent_evaluation.calibrate

calibrated_agent_evaluation.calibrate

Check a judge against human labels, and correct the failure rate it reports.

Works on pairs of a human gold label and the judge's verdict on the same trace, and answers three questions: is this judge good enough to trust (both its true-positive and true-negative rates must clear the gate at the low end of their 95% range), what is the real failure rate (the Rogan-Gladen correction, with a range that carries the uncertainty of both the gold set and the scored sample), and when must the judge be checked again (version and fingerprint pinning, and single-use test sets). Running a judge is the judge runtime's job, not this package's.

Reads normalized Label objects only; the numeric work lives in stats. The engine and the planner load lazily because they need the stats extra, while the record types (Interval, Bound) and the errors need nothing beyond the base install.

CALIB_SCHEMA_VERSION module-attribute

CALIB_SCHEMA_VERSION = '9'

MIN_RATE_SAMPLE module-attribute

MIN_RATE_SAMPLE = 150

CalibrationError

Bases: CalibratedAgentEvaluationDataError

Base class for every calibration-engine refusal.

CalibrationStoreError

Bases: CalibrationError

A calibration-record file read or write failed.

Wraps the underlying OSError so a filesystem failure surfaces inside the typed family rather than as a raw OS exception escaping the store's public surface.

CalibrationVersionError

Bases: CalibrationError

A stored calibration record's schema_version != the running CALIB_SCHEMA_VERSION.

Refuse-loudly rather than coercing the data (the no-silent-coercion precedent).

DuplicateGoldError

DuplicateGoldError(message, *, duplicates)

Bases: UnpairedObservationError

One (trace_id, span_id) target carries more than one gold label.

Two raters' labels on one target are one piece of evidence, not two: counting both doubles the sample size behind the bounds and can certify a judge the single-label set refuses. duplicates names the repeated targets.

Source code in src/calibrated_agent_evaluation/calibrate/errors.py
def __init__(self, message: str, *, duplicates: tuple[tuple[str, str | None], ...]) -> None:
    super().__init__(message)
    self.duplicates = duplicates

duplicates instance-attribute

duplicates = duplicates

ErrorRatesDoNotTransferError

Bases: NonIdentifiableRateError

The rate sample contradicts the judge's calibrated error rates.

Almost every draw puts the measured rate below the judge's false-positive floor (or above its true-positive ceiling), so the corrected rate clips to a single point. A zero-width interval there would claim the true rate is exactly 0 (or 1); what the data show is that the error rates measured on the calibration gold do not hold on this sample.

FewShotEvaluationError

Bases: CalibrationError

Certification/TPR/TNR metrics were requested over the few-shot pool.

The few-shot pool is never an evaluation set; evaluating on it inflates the numbers.

InfeasiblePlanError

Bases: CalibrationError

The calibration planner was asked for a target no finite sample size can reach.

tau >= p_hat: the expected judge accuracy never clears the threshold, so the lower bound cannot either -- refused rather than searching unboundedly.

LockedTestReuseError

Bases: CalibrationError

A locked test set that was already consumed once was re-evaluated (single-use).

NonIdentifiableRateError

Bases: CalibrationError

The measured rate lies outside the Rogan-Gladen identifiable region.

measured < 1 - TNR or the denominator TPR - (1 - TNR)) <= 0; the corrected point is refused and the bound is reported rather than silently clamping to 0/1.

RateSampleTooSmallError

RateSampleTooSmallError(message, *, n, minimum)

Bases: CalibrationError

A corrected rate was requested from a rate sample smaller than the measured minimum.

The interval was measured to under-cover with too few evaluation traces at low prevalence, so a rate sample below MIN_RATE_SAMPLE is refused. n is the sample's size (distinct traces when verdicts are clustered) and minimum the size needed.

Source code in src/calibrated_agent_evaluation/calibrate/errors.py
def __init__(self, message: str, *, n: int, minimum: int) -> None:
    super().__init__(message)
    self.n = n
    self.minimum = minimum

n instance-attribute

n = n

minimum instance-attribute

minimum = minimum

RateUnsafeError

Bases: CalibrationError

A corrected rate was requested from a record certified on gold below the coverage floor.

The certification stands (the Wilson bound is valid at any n), but below the coverage floor the corrected rate's interval was measured to cover the true rate less often than 95%, so an interval built from smaller gold is refused. At or above the floor coverage is close to 95% but not everywhere: a class with about 2 to 3 errors still under-covers slightly, a known limit of the current interval.

SampleMisuseError

Bases: CalibrationError

The prevalence-corrected rate was requested from the hunted fail-heavy set.

The true rate is measured on a separate production-representative sample -- two samples, two jobs. Reading it off the hunted set would report a wildly inflated rate. Raised too when the rate sample shares traces with the calibration gold.

StatisticsInputError

Bases: CalibrationError

A pure statistics primitive received an out-of-domain numeric input.

n <= 0, k outside [0, n], or a probability outside [0, 1] -- guarded so a bad count surfaces as a typed error rather than a NaN leaking out of scipy/statsmodels.

UncalibratedJudgeError

Bases: CalibrationError

A corrected rate or promotion was requested for a judge with no CERTIFIED calibration.

UnpairedObservationError

Bases: CalibrationError

An observation could not be paired to exactly one human gold + one judge verdict.

Raised rather than silently dropping the observation (which would bias TPR/TNR).

UnratifiedGoldError

Bases: CalibrationError

A calibration gold set contains a label not produced/ratified by a human.

Certifying a judge against another judge turns every confidence bound into theater -- refused.

WeakJudgeError

Bases: CalibrationError

Youden's J < 0.5 -- the Rogan-Gladen point correction is refused.

The widening interval is still returned as the honest signal; only the point rate accessor raises, so a near-random judge cannot be "rescued" by a correction that divides by ~0.

InMemoryLockedTestLedger

InMemoryLockedTestLedger(
    consumed=(), *, consumed_trace_ids=()
)

A LockedTestLedger held in memory, optionally seeded with consumed sets.

Refuses a set whose id, content digest, or any trace was already consumed, except a strict superset of sets opened with allow_top_up for the same judge fingerprint. Not safe for concurrent consume calls on one instance.

Parameters:

Name Type Description Default
consumed Iterable[str]

Locked-set ids already used, for example from stored calibration records.

()
consumed_trace_ids Iterable[str]

Trace ids already used in a locked test, for example the trace_ids of stored calibration records for the same failure mode.

()
Source code in src/calibrated_agent_evaluation/calibrate/ledger.py
def __init__(
    self, consumed: Iterable[str] = (), *, consumed_trace_ids: Iterable[str] = ()
) -> None:
    self._ids: set[str] = set(consumed)
    self._digests: dict[str, str] = {}
    self._traces: dict[str, str] = dict.fromkeys(consumed_trace_ids, "an earlier locked test")
    self._members: dict[str, frozenset[str]] = {}
    self._looks: dict[str, int] = {}
    self._open: dict[str, str] = {}

from_records classmethod

from_records(records)

A ledger holding every set records consumed, with refused-short sets left open.

Pass the records for one failure mode. A refused set already topped up by a later record (one holding all its traces) is closed, so it cannot be grown twice.

Source code in src/calibrated_agent_evaluation/calibrate/ledger.py
@classmethod
def from_records(cls, records: Sequence[CalibrationRecord]) -> InMemoryLockedTestLedger:
    """A ledger holding every set ``records`` consumed, with refused-short sets left open.

    Pass the records for one failure mode. A refused set already topped up by a later
    record (one holding all its traces) is closed, so it cannot be grown twice.
    """
    ledger = cls()
    for record in records:
        members = frozenset(record.trace_ids)
        ledger._ids.add(record.hunted_set_id)
        ledger._members[record.hunted_set_id] = members
        ledger._looks[record.hunted_set_id] = record.looks
        ledger._traces.update(dict.fromkeys(record.trace_ids, record.hunted_set_id))
        if open_for_top_up(record) and record.judge_fingerprint is not None:
            ledger._open[record.hunted_set_id] = record.judge_fingerprint
    for record in records:
        members = ledger._members[record.hunted_set_id]
        for other in list(ledger._open):
            if other != record.hunted_set_id and ledger._members[other] < members:
                del ledger._open[other]
    return ledger

is_consumed

is_consumed(set_id)

Whether a set with id set_id was already consumed.

Source code in src/calibrated_agent_evaluation/calibrate/ledger.py
def is_consumed(self, set_id: str) -> bool:
    """Whether a set with id ``set_id`` was already consumed."""
    return set_id in self._ids

allow_top_up

allow_top_up(set_id, *, judge_fingerprint)

Open the consumed set set_id to a top-up by the judge with judge_fingerprint.

Raises:

Type Description
LockedTestReuseError

set_id was never consumed here.

Source code in src/calibrated_agent_evaluation/calibrate/ledger.py
def allow_top_up(self, set_id: str, *, judge_fingerprint: str) -> None:
    """Open the consumed set ``set_id`` to a top-up by the judge with ``judge_fingerprint``.

    Raises:
        LockedTestReuseError: ``set_id`` was never consumed here.
    """
    if set_id not in self._members:
        raise LockedTestReuseError(
            f"locked test {set_id!r} was not consumed by this ledger, so there is nothing to "
            "top up"
        )
    self._open[set_id] = judge_fingerprint

consume

consume(locked, *, judge_fingerprint=None)

Mark locked consumed, refusing a reused id, reused content or a shared trace.

A set holding every trace of open sets for the same judge_fingerprint, plus at least one fresh trace, is their top-up: it is accepted and the looks are counted.

Returns:

Type Description
int

How many looks the judge has had at this gold, 1 for a fresh set.

Raises:

Type Description
LockedTestReuseError

The set, its content or one of its traces was already used.

Source code in src/calibrated_agent_evaluation/calibrate/ledger.py
def consume(self, locked: CalibrationSet, *, judge_fingerprint: str | None = None) -> int:
    """Mark ``locked`` consumed, refusing a reused id, reused content or a shared trace.

    A set holding every trace of open sets for the same ``judge_fingerprint``, plus at least
    one fresh trace, is their top-up: it is accepted and the looks are counted.

    Returns:
        How many looks the judge has had at this gold, 1 for a fresh set.

    Raises:
        LockedTestReuseError: The set, its content or one of its traces was already used.
    """
    if locked.set_id in self._ids:
        raise LockedTestReuseError(
            f"locked test {locked.set_id!r} was already consumed once; it is single-use, so "
            "re-iterating after touching the locked test is refused"
        )
    digest = locked_set_digest(locked)
    if digest in self._digests:
        first = self._digests[digest]
        raise LockedTestReuseError(
            f"locked test {locked.set_id!r} holds exactly the gold of {first!r}, "
            "which was already consumed; renaming a locked test does not make it fresh. Label "
            "new traces for a new locked test"
        )
    traces = list(dict.fromkeys(p.gold.trace_id for p in locked.pairs))
    members = frozenset(traces)
    topped = self._topped_up(members, judge_fingerprint)
    covered = {t for set_id in topped for t in self._members[set_id]}
    shared = [t for t in traces if t in self._traces and t not in covered]
    if shared:
        shown = ", ".join(shared[:_SHOWN_TRACES])
        more = len(shared) - _SHOWN_TRACES
        raise LockedTestReuseError(
            f"{len(shared)} of locked test {locked.set_id!r}'s {len(traces)} trace(s) were "
            f"already used in {self._traces[shared[0]]!r}: {shown}"
            + (f" and {more} more" if more > 0 else "")
            + ". A judge re-scored on traces it was tested on is a retest, so the set is "
            "refused. Leave those traces out, or label fresh ones"
        )
    looks = 1 + max((self._looks.get(s, 1) for s in topped), default=0)
    for set_id in topped:
        del self._open[set_id]
    self._ids.add(locked.set_id)
    self._digests[digest] = locked.set_id
    self._members[locked.set_id] = members
    self._looks[locked.set_id] = looks
    self._traces.update(dict.fromkeys(traces, locked.set_id))
    return looks

LockedTestLedger

Bases: Protocol

Records which locked test sets have been evaluated.

consume

consume(locked, *, judge_fingerprint=None)

Mark locked consumed and return how many looks the judge has had at its gold.

1 for a fresh set; one more for each refused set it tops up. Raises LockedTestReuseError if the set, its content or a trace was already used.

Source code in src/calibrated_agent_evaluation/calibrate/ledger.py
def consume(self, locked: CalibrationSet, *, judge_fingerprint: str | None = None) -> int:
    """Mark ``locked`` consumed and return how many looks the judge has had at its gold.

    1 for a fresh set; one more for each refused set it tops up. Raises
    :class:`LockedTestReuseError` if the set, its content or a trace was already used.
    """
    ...

allow_top_up

allow_top_up(set_id, *, judge_fingerprint)

Let a superset of the consumed set set_id be consumed by the same judge.

Source code in src/calibrated_agent_evaluation/calibrate/ledger.py
def allow_top_up(self, set_id: str, *, judge_fingerprint: str) -> None:
    """Let a superset of the consumed set ``set_id`` be consumed by the same judge."""
    ...

ConfusionCounts dataclass

ConfusionCounts(tp, fn, tn, fp)

2x2 confusion counts with FAIL as the positive class.

tp -- gold FAIL, verdict FAIL (caught failure). fn -- gold FAIL, verdict PASS (missed). tn -- gold PASS, verdict PASS (correct clear). fp -- gold PASS, verdict FAIL (false alarm).

tp instance-attribute

tp

fn instance-attribute

fn

tn instance-attribute

tn

fp instance-attribute

fp

n_pos property

n_pos

Number of real positives (gold FAIL) -- the denominator of TPR.

n_neg property

n_neg

Number of real negatives (gold PASS) -- the denominator of TNR.

Bound pydantic-model

Bases: CalibratedAgentEvaluationModel

A gated rate with its 95% one-sided lower confidence bound.

The gate reads lb95, never point -- the point estimate at the threshold does not certify. Carrying both in one object is how the "no bare number" rule is made structural.

lb95 is a one-sided 95% Wilson lower bound (sided="lower"), not the lower edge of a two-sided 95% interval; n is the class size it was computed over. For gold whose items cluster (several spans of one trace) the bound is taken at the effective size and method says so.

Config:

  • frozen: True
  • extra: forbid

Fields:

Validators:

point pydantic-field

point

lb95 pydantic-field

lb95

n pydantic-field

n

method pydantic-field

method = METHOD_WILSON

level pydantic-field

level = 0.95

sided pydantic-field

sided = 'lower'

CalibrationRecord pydantic-model

Bases: CalibratedAgentEvaluationModel

The reproducible gate evidence.

Pins the judge's exact (model, prompt_version) -- a change to either invalidates the record. Carries the gated bounds, Youden's J, the corrected-rate interval, and the CERTIFIED/REFUSED result. Certification is an intersection-union test per judge (both lower bounds must clear), and certifying several judges is not a joint claim, so the record carries no multiplicity family. References its sample members by trace_id even though the record lives outside the trace store.

Config:

  • frozen: True
  • extra: forbid

Fields:

calib_id pydantic-field

calib_id

judge_id pydantic-field

judge_id

model pydantic-field

model

prompt_version pydantic-field

prompt_version

hunted_set_id pydantic-field

hunted_set_id

n_pos pydantic-field

n_pos

n_neg pydantic-field

n_neg

tpr pydantic-field

tpr

tnr pydantic-field

tnr

tpr_design_effect pydantic-field

tpr_design_effect = 1.0

tnr_design_effect pydantic-field

tnr_design_effect = 1.0

youden_j pydantic-field

youden_j

result pydantic-field

result

rate_set_id pydantic-field

rate_set_id = None

corrected_rate pydantic-field

corrected_rate = None

locked_test_consumed pydantic-field

locked_test_consumed = False

bias_battery_ref pydantic-field

bias_battery_ref = None

schema_version pydantic-field

schema_version = CALIB_SCHEMA_VERSION

trace_ids pydantic-field

trace_ids

planned_per_class pydantic-field

planned_per_class = None

planned_coverage_floor pydantic-field

planned_coverage_floor = None

observed_coverage_floor pydantic-field

observed_coverage_floor = None

observed_floor_unmeasured pydantic-field

observed_floor_unmeasured = False

judge_fingerprint pydantic-field

judge_fingerprint = None

looks pydantic-field

looks = 1

agreement_alpha pydantic-field

agreement_alpha = None

provenance pydantic-field

provenance

coverage_floor property

coverage_floor

The floor that applies: the larger of the planned and the observed floor, if any.

effective_n_pos property

effective_n_pos

The FAIL-class gold size the bounds were taken at: n_pos / tpr_design_effect.

effective_n_neg property

effective_n_neg

The PASS-class gold size the bounds were taken at: n_neg / tnr_design_effect.

below_coverage_floor property

below_coverage_floor

Whether either class of the gold is smaller than the coverage floor that applies.

Such a record certifies (the Wilson bound is valid at any n), but the corrected rate's interval is not known to cover nominally at that size, so corrected_rate refuses it. A record whose observed rates have no measured floor is below it at any size. Clustered gold is compared at its effective size.

CalibrationSet dataclass

CalibrationSet(pairs, split, set_id)

A split-tagged set of paired observations.

split tags the provenance: "few_shot" (never evaluated), "dev" (iterate freely, never certifies), "locked_test" (the only source of a CERTIFIED verdict, consumed exactly once). set_id names the set; the ledger also keys on its content, so a renamed copy is still a used set.

pairs instance-attribute

pairs

split instance-attribute

split

set_id instance-attribute

set_id

Interval pydantic-model

Bases: CalibratedAgentEvaluationModel

A two-sided confidence interval for the Rogan-Gladen corrected rate.

point is None when the correction was refused (J<0.5, or non-identifiable); refusal_reason records which. The (lo, hi) bound is always populated and is the honest signal even when the point is refused.

method, level, sided and n say what kind of interval this is, so a 95% two-sided interval and a 95% one-sided bound can never be confused downstream. n is the sample the interval was computed over (for a two-sample difference, both samples together).

Config:

  • frozen: True
  • extra: forbid

Fields:

Validators:

point pydantic-field

point

lo pydantic-field

lo

hi pydantic-field

hi

refusal_reason pydantic-field

refusal_reason = None

method pydantic-field

method

level pydantic-field

level

sided pydantic-field

sided

n pydantic-field

n

require_point

require_point()

Return the corrected point rate, or raise the typed refusal that produced None.

The point-rate accessor: a caller wanting the scalar must go through this, so a refused correction surfaces as a typed WeakJudgeError / NonIdentifiableRateError rather than a silent None leaking into a report. The (lo, hi) bound stays readable as the signal.

Source code in src/calibrated_agent_evaluation/calibrate/record.py
def require_point(self) -> float:
    """Return the corrected point rate, or raise the typed refusal that produced ``None``.

    The point-rate accessor: a caller wanting the scalar must go through this,
    so a refused correction surfaces as a typed :class:`WeakJudgeError` /
    :class:`NonIdentifiableRateError` rather than a silent ``None`` leaking into a report. The
    ``(lo, hi)`` bound stays readable as the signal.
    """
    if self.point is not None:
        return self.point
    reason = self.refusal_reason or "the correction was refused"
    if self.refusal_reason is not None and TRANSFER_REFUSAL in self.refusal_reason:
        raise ErrorRatesDoNotTransferError(f"corrected point rate is unavailable: {reason}")
    if self.refusal_reason is not None and self.refusal_reason.startswith("non-identifiable"):
        raise NonIdentifiableRateError(
            f"corrected point rate is unavailable: {reason}; the interval "
            f"[{self.lo:.4f}, {self.hi:.4f}] is the honest signal"
        )
    raise WeakJudgeError(
        f"corrected point rate is unavailable: {reason}; the widening interval "
        f"[{self.lo:.4f}, {self.hi:.4f}] is the honest signal"
    )

LabelPair dataclass

LabelPair(gold, verdict)

One paired observation: a human gold label and the judge's verdict for the same item.

Both reference the same (trace_id, span_id); the engine derives the 2x2 confusion counts from a set of these, treating outcome == "FAIL" as the positive class.

gold instance-attribute

gold

verdict instance-attribute

verdict

CalibrationPlan dataclass

CalibrationPlan(
    per_class,
    certification,
    coverage_floor,
    double_coded=None,
)

The per-class sizes plan_calibration derives, and the one to hunt.

certification certifies the judge; coverage_floor is the smallest size at which the corrected rate's interval covers nominally at the planned accuracy; per_class is the larger, the size to hunt, or the rate-safe size for a plan made with rate_safe=True. A judge certified on fewer than coverage_floor per class is certified, but its corrected rate is refused. double_coded is double_coded_per_class for gold two or more raters will code, and per_class is then at least that; None for one rater.

per_class instance-attribute

per_class

certification instance-attribute

certification

coverage_floor instance-attribute

coverage_floor

double_coded class-attribute instance-attribute

double_coded = None

locked_set_digest

locked_set_digest(locked)

A hash of the set's targets and their gold outcomes, independent of order and set_id.

Source code in src/calibrated_agent_evaluation/calibrate/ledger.py
def locked_set_digest(locked: CalibrationSet) -> str:
    """A hash of the set's targets and their gold outcomes, independent of order and ``set_id``."""
    members = sorted(
        f"{p.gold.trace_id}\x1f{p.gold.span_id or ''}\x1f{p.gold.outcome}" for p in locked.pairs
    )
    return hashlib.sha256("\x1e".join(members).encode()).hexdigest()

open_for_top_up

open_for_top_up(record)

Whether record's set may be grown: REFUSED short of its plan, by a fingerprinted judge.

A judge refused at its planned size, or certified, was measured on as much gold as was asked for, so more gold on top is a second test, not the first one completed.

Source code in src/calibrated_agent_evaluation/calibrate/ledger.py
def open_for_top_up(record: CalibrationRecord) -> bool:
    """Whether ``record``'s set may be grown: REFUSED short of its plan, by a fingerprinted judge.

    A judge refused at its planned size, or certified, was measured on as much gold as was asked
    for, so more gold on top is a second test, not the first one completed.
    """
    return (
        record.result == "REFUSED"
        and record.judge_fingerprint is not None
        and record.planned_per_class is not None
        and min(record.n_pos, record.n_neg) < record.planned_per_class
    )

pair_observations

pair_observations(pairs)

Reduce paired observations to 2x2 confusion counts.

Raises UnratifiedGoldError if any gold label is not kind == "human", UnpairedObservationError if a pair's gold and verdict do not reference the same (trace_id, span_id), and DuplicateGoldError if a target appears in more than one pair. All are refusals, never silent drops.

Parameters:

Name Type Description Default
pairs Sequence[LabelPair]

The (judge verdict, gold label) pairs.

required

Returns:

Type Description
ConfusionCounts

The 2x2 confusion counts.

Raises:

Type Description
UnratifiedGoldError

A gold label is not human.

UnpairedObservationError

A verdict and its gold label name different targets.

DuplicateGoldError

A target appears in more than one pair.

Source code in src/calibrated_agent_evaluation/calibrate/pairing.py
def pair_observations(pairs: Sequence[LabelPair]) -> ConfusionCounts:
    """Reduce paired observations to 2x2 confusion counts.

    Raises :class:`UnratifiedGoldError` if any gold label is not ``kind == "human"``,
    :class:`UnpairedObservationError` if a pair's gold and verdict do not reference the same
    ``(trace_id, span_id)``, and :class:`DuplicateGoldError` if a target appears in more than one
    pair. All are refusals, never silent drops.

    Args:
        pairs: The (judge verdict, gold label) pairs.

    Returns:
        The 2x2 confusion counts.

    Raises:
        UnratifiedGoldError: A gold label is not human.
        UnpairedObservationError: A verdict and its gold label name different targets.
        DuplicateGoldError: A target appears in more than one pair.
    """
    refuse_duplicate_targets((p.gold.trace_id, p.gold.span_id) for p in pairs)
    tp = fn = tn = fp = 0
    for pair in pairs:
        gold, verdict = pair.gold, pair.verdict
        if gold.kind != "human":
            raise UnratifiedGoldError(
                f"gold label for {gold.trace_id} has kind={gold.kind!r}, not 'human'; "
                "calibrating from a non-human gold set certifies a judge against a judge "
                ""
            )
        if gold.trace_id != verdict.trace_id or gold.span_id != verdict.span_id:
            raise UnpairedObservationError(
                f"observation mispaired: gold {(gold.trace_id, gold.span_id)} != "
                f"verdict {(verdict.trace_id, verdict.span_id)}"
            )
        gold_fail = gold.outcome == "FAIL"
        verdict_fail = verdict.outcome == "FAIL"
        if gold_fail and verdict_fail:
            tp += 1
        elif gold_fail and not verdict_fail:
            fn += 1
        elif not gold_fail and not verdict_fail:
            tn += 1
        else:
            fp += 1
    return ConfusionCounts(tp=tp, fn=fn, tn=tn, fp=fp)

refuse_duplicate_targets

refuse_duplicate_targets(targets)

Refuse a gold set in which any (trace_id, span_id) target appears more than once.

Raises:

Type Description
DuplicateGoldError

A target appears more than once; the error names the repeats.

Source code in src/calibrated_agent_evaluation/calibrate/pairing.py
def refuse_duplicate_targets(targets: Iterable[tuple[str, str | None]]) -> None:
    """Refuse a gold set in which any ``(trace_id, span_id)`` target appears more than once.

    Raises:
        DuplicateGoldError: A target appears more than once; the error names the repeats.
    """
    counts = Counter(targets)
    repeated = tuple(
        sorted((k for k, n in counts.items() if n > 1), key=lambda k: (k[0], k[1] or ""))
    )
    if not repeated:
        return
    shown = ", ".join(
        f"{trace_id}"
        + (f"/{span_id}" if span_id is not None else "")
        + f" (x{counts[(trace_id, span_id)]})"
        for trace_id, span_id in repeated[:_SHOWN_DUPLICATES]
    )
    more = len(repeated) - _SHOWN_DUPLICATES
    raise DuplicateGoldError(
        f"{len(repeated)} target(s) carry more than one gold label: {shown}"
        + (f" and {more} more" if more > 0 else "")
        + ". Each extra label would count as an independent observation and inflate the sample "
        "behind the bounds. Keep one adjudicated label per target (for double-coded gold, resolve "
        "the raters' disagreements first) and calibrate again.",
        duplicates=repeated,
    )

dump_records

dump_records(records, path, fmt='csv')

Write records to calibration_record.{csv,json} under path.

Parameters:

Name Type Description Default
records Sequence[CalibrationRecord]

The records to persist.

required
path str | Path

The directory to write into.

required
fmt str

Which file a later load_records reads back (both are always written).

'csv'

Raises:

Type Description
CalibrationStoreError

A bad fmt or any filesystem failure.

Source code in src/calibrated_agent_evaluation/calibrate/store.py
def dump_records(records: Sequence[CalibrationRecord], path: str | Path, fmt: str = "csv") -> None:
    """Write ``records`` to ``calibration_record.{csv,json}`` under ``path``.

    Args:
        records: The records to persist.
        path: The directory to write into.
        fmt: Which file a later :func:`load_records` reads back (both are always written).

    Raises:
        CalibrationStoreError: A bad ``fmt`` or any filesystem failure.
    """
    _STORE.dump(records, path, fmt)

load_records

load_records(path, fmt='csv')

Read calibration records back from path; a missing file is an empty store.

Parameters:

Name Type Description Default
path str | Path

The directory dump_records wrote.

required
fmt str

"csv" or "json".

'csv'

Returns:

Type Description
list[CalibrationRecord]

The stored records.

Raises:

Type Description
CalibrationVersionError

A stored schema_version differs from the running one.

CalibrationStoreError

Any read, parse or validation failure.

Source code in src/calibrated_agent_evaluation/calibrate/store.py
def load_records(path: str | Path, fmt: str = "csv") -> list[CalibrationRecord]:
    """Read calibration records back from ``path``; a missing file is an empty store.

    Args:
        path: The directory :func:`dump_records` wrote.
        fmt: ``"csv"`` or ``"json"``.

    Returns:
        The stored records.

    Raises:
        CalibrationVersionError: A stored ``schema_version`` differs from the running one.
        CalibrationStoreError: Any read, parse or validation failure.
    """
    return _STORE.load(path, fmt)

certifies

certifies(record, judge)

Return whether record still certifies judge as it is now.

False when the record is not CERTIFIED or is not pinned to the judge (pin_mismatch): a model, prompt version or prompt content change requires re-calibration.

Source code in src/calibrated_agent_evaluation/calibrate/engine.py
def certifies(record: CalibrationRecord, judge: Judge) -> bool:
    """Return whether ``record`` still certifies ``judge`` as it is now.

    False when the record is not CERTIFIED or is not pinned to the judge (:func:`pin_mismatch`):
    a model, prompt version or prompt content change requires re-calibration.
    """
    return record.result == "CERTIFIED" and pin_mismatch(record, judge) is None

certify

certify(
    judge,
    locked,
    *,
    ledger,
    calib_id=None,
    plan=None,
    sampling=None,
    agreement_alpha=None,
)

Run the locked-test gate and return a CERTIFIED/REFUSED CalibrationRecord.

Refuses (never certifies) unless locked.split == "locked_test" -- a dev set never certifies and the few-shot pool is never evaluated. Consumes locked from ledger and refuses a set whose id, content or any trace was already consumed (single-use). Gold must be human, one label per target (enforced in pairing). CERTIFIED iff both lower bounds clear GATE_THRESHOLD.

plan -- the CalibrationPlan the gold was hunted against. Its per-class size and coverage floor are recorded, and gold below the floor is marked (CalibrationRecord.below_coverage_floor) without changing the result: the judge still certifies, and corrected_rate refuses an interval from the record. A certified judge's floor is also planned at the TPR and TNR it showed, with or without a plan, and the larger of the floors applies; a judge whose accuracy differs from the plan can need more gold.

Each judge's certification stands alone. Certifying several judges in one run is not a joint claim that all of them are good, so no correction across judges is applied and none is recorded.

Promotion is an intersection-union test: a judge deficient in TPR or in TNR is certified with probability at most about 5% (slightly more at some n, since Wilson coverage oscillates), with no per-judge correction needed for using two bounds. The two bounds are not a joint 95% region: at TPR = TNR = 0.80 and 80 per class their simultaneous coverage is 0.890.

Parameters:

Name Type Description Default
judge Judge

The judge identity (id, model, prompt version) the record pins to.

required
locked CalibrationSet

The locked test set of (judge verdict, human gold) pairs.

required
ledger LockedTestLedger

The record of locked sets already used; locked is consumed from it.

required
calib_id str | None

The record id; generated when omitted.

None
plan CalibrationPlan | None

The calibration plan the gold was collected against.

None
sampling Mapping[str, float] | None

The sampling parameters the judge's verdicts were drawn with, recorded on the provenance stamp (empty: none were sent).

None
agreement_alpha Interval | None

The gold raters' agreement, recorded on the record when measured.

None

A REFUSED set short of plan's per-class size is opened in ledger for a top-up by the same judge: a later set holding all its traces plus fresh ones may be tested, and the record counts the looks (looks) so the retest is disclosed.

Returns:

Type Description
CalibrationRecord

A CERTIFIED or REFUSED calibration record.

Raises:

Type Description
FewShotEvaluationError

locked is the few-shot pool or the dev set.

LockedTestReuseError

locked, its content or one of its traces was already consumed.

UnratifiedGoldError

A gold label is not human.

UnpairedObservationError

A verdict and its gold label name different targets.

DuplicateGoldError

A target carries more than one gold label.

Source code in src/calibrated_agent_evaluation/calibrate/engine.py
def certify(
    judge: Judge,
    locked: CalibrationSet,
    *,
    ledger: LockedTestLedger,
    calib_id: str | None = None,
    plan: CalibrationPlan | None = None,
    sampling: Mapping[str, float] | None = None,
    agreement_alpha: Interval | None = None,
) -> CalibrationRecord:
    """Run the locked-test gate and return a CERTIFIED/REFUSED :class:`CalibrationRecord`.

    Refuses (never certifies) unless ``locked.split == "locked_test"`` -- a dev set never certifies
    and the few-shot pool is never evaluated. Consumes ``locked`` from ``ledger`` and refuses a set
    whose id, content or any trace was already consumed (single-use). Gold must be human, one
    label per target (enforced in pairing). CERTIFIED iff both lower bounds clear
    :data:`GATE_THRESHOLD`.

    ``plan`` -- the :class:`~calibrated_agent_evaluation.calibrate.plan.CalibrationPlan` the gold
    was hunted against. Its per-class size and coverage floor are recorded, and gold below the floor
    is marked (``CalibrationRecord.below_coverage_floor``) without changing the result: the judge
    still certifies, and :func:`corrected_rate` refuses an interval from the record.
    A certified judge's floor is also planned at the TPR and TNR it showed, with or without a
    ``plan``, and the larger of the floors applies; a judge whose accuracy differs from the plan
    can need more gold.

    Each judge's certification stands alone. Certifying several judges in one run is not a joint
    claim that all of them are good, so no correction across judges is applied and none is
    recorded.

    Promotion is an intersection-union test: a judge deficient in TPR or in TNR is certified with
    probability at most about 5% (slightly more at some ``n``, since Wilson coverage oscillates),
    with no per-judge correction needed for using two bounds. The two bounds are not a joint 95%
    region: at TPR = TNR = 0.80 and 80 per class their simultaneous coverage is 0.890.

    Args:
        judge: The judge identity (id, model, prompt version) the record pins to.
        locked: The locked test set of (judge verdict, human gold) pairs.
        ledger: The record of locked sets already used; ``locked`` is consumed from it.
        calib_id: The record id; generated when omitted.
        plan: The calibration plan the gold was collected against.
        sampling: The sampling parameters the judge's verdicts were drawn with, recorded on the
            provenance stamp (empty: none were sent).
        agreement_alpha: The gold raters' agreement, recorded on the record when measured.

    A REFUSED set short of ``plan``'s per-class size is opened in ``ledger`` for a top-up by the
    same judge: a later set holding all its traces plus fresh ones may be tested, and the record
    counts the looks (``looks``) so the retest is disclosed.

    Returns:
        A CERTIFIED or REFUSED calibration record.

    Raises:
        FewShotEvaluationError: ``locked`` is the few-shot pool or the dev set.
        LockedTestReuseError: ``locked``, its content or one of its traces was already consumed.
        UnratifiedGoldError: A gold label is not human.
        UnpairedObservationError: A verdict and its gold label name different targets.
        DuplicateGoldError: A target carries more than one gold label.
    """
    if locked.split == "few_shot":
        raise FewShotEvaluationError(
            f"refusing to evaluate judge {judge.judge_id!r} on the few-shot pool "
            f"(set_id={locked.set_id!r}); the few-shot pool is never an evaluation set"
        )
    if locked.split == "dev":
        raise FewShotEvaluationError(
            f"refusing to certify judge {judge.judge_id!r} from the dev set "
            f"(set_id={locked.set_id!r}); certification derives only from the locked test"
        )
    counts = pair_observations(locked.pairs)  # human-gold, exact pairing, one label per target
    fingerprint = fingerprint_of(judge)
    looks = ledger.consume(locked, judge_fingerprint=fingerprint)

    deff_pos = _class_design_effect(locked.pairs, "FAIL")
    deff_neg = _class_design_effect(locked.pairs, "PASS")
    tpr = _bound(counts.tp, counts.n_pos, deff_pos)
    tnr = _bound(counts.tn, counts.n_neg, deff_neg)
    j = youden_j(tpr.point, tnr.point)
    certified = tpr.lb95 >= GATE_THRESHOLD and tnr.lb95 >= GATE_THRESHOLD
    trace_ids = tuple(dict.fromkeys(p.gold.trace_id for p in locked.pairs))
    # The observed floor applies to every certified record, planned or not: a library caller who
    # passes no plan must not get a corrected rate from gold below the floor.
    observed_floor, unmeasured = (
        _observed_floor(tpr.point, tnr.point) if certified else (None, False)
    )
    if (
        not certified
        and fingerprint is not None
        and plan is not None
        and min(counts.n_pos, counts.n_neg) < plan.per_class
    ):
        ledger.allow_top_up(locked.set_id, judge_fingerprint=fingerprint)
    return CalibrationRecord(
        calib_id=calib_id or f"calib-{judge.judge_id}-{locked.set_id}",
        judge_id=judge.judge_id,
        judge_fingerprint=fingerprint,
        model=judge.model,
        prompt_version=judge.prompt_version,
        hunted_set_id=locked.set_id,
        n_pos=counts.n_pos,
        n_neg=counts.n_neg,
        tpr=tpr,
        tnr=tnr,
        tpr_design_effect=deff_pos,
        tnr_design_effect=deff_neg,
        youden_j=j,
        result="CERTIFIED" if certified else "REFUSED",
        locked_test_consumed=True,
        trace_ids=trace_ids,
        planned_per_class=None if plan is None else plan.per_class,
        planned_coverage_floor=None if plan is None else plan.coverage_floor,
        observed_coverage_floor=observed_floor,
        observed_floor_unmeasured=unmeasured,
        looks=looks,
        agreement_alpha=agreement_alpha,
        provenance=capture_provenance(thresholds=_gate_thresholds(), sampling=sampling),
    )

corrected_rate

corrected_rate(
    record,
    rate_sample_verdicts,
    rate_set_id,
    *,
    rate_trace_ids,
    seed=0,
    n_resamples=DEFAULT_N_RESAMPLES,
    rate_cluster_ids=None,
)

Return a copy of record with the Rogan-Gladen corrected rate + CI attached.

seed and n_resamples (the Monte Carlo draw count) drive the interval and are stamped on the returned record's provenance, so the interval can be recomputed from the stored record alone.

Refuses for an uncertified judge, and with RateUnsafeError for a record whose gold is below its coverage floor, the larger of the planned and the observed one, or whose observed rates have no measured floor, and with RateSampleTooSmallError for a rate sample of fewer than MIN_RATE_SAMPLE traces (distinct traces when rate_cluster_ids is given). When J<0.5 the corrected point is refused (the interval is still returned as the signal); when the rate is non-identifiable the point is refused and the bound reported. When more than one tail's share of the interval's draws have no corrected rate, the interval is refused too: (lo, hi) = (0, 1) with a refusal_reason starting "non-identifiable interval:". The interval always stays finite and in [0,1].

Parameters:

Name Type Description Default
record CalibrationRecord

A CERTIFIED calibration record for the judge.

required
rate_sample_verdicts Sequence[int]

The judge's verdicts (1 = FAIL, 0 = PASS) on a separate production-representative sample; their mean is the raw rate.

required
rate_set_id str

The id of that sample; it must differ from record.hunted_set_id.

required
rate_trace_ids Sequence[str]

The trace id of each verdict, in the same order. Required, because the rate sample must be checked against the calibration gold: hunted traces re-entering it break the independence the interval assumes.

required
seed int

Fixes the Monte Carlo draws behind the interval.

0
n_resamples int

The number of Monte Carlo draws.

DEFAULT_N_RESAMPLES
rate_cluster_ids Sequence[str] | None

The trace id of each verdict, when one trace contributes several and the interval should account for the clustering.

None

Returns:

Type Description
CalibrationRecord

A copy of record with the corrected rate and its interval attached.

Raises:

Type Description
UncalibratedJudgeError

record is not CERTIFIED.

RateUnsafeError

The record's gold is below its coverage floor.

SampleMisuseError

The rate sample is empty, is the hunted calibration set, shares traces with the calibration gold, or its trace ids do not match its verdicts.

RateSampleTooSmallError

The rate sample has fewer than MIN_RATE_SAMPLE traces.

Source code in src/calibrated_agent_evaluation/calibrate/engine.py
def corrected_rate(
    record: CalibrationRecord,
    rate_sample_verdicts: Sequence[int],
    rate_set_id: str,
    *,
    rate_trace_ids: Sequence[str],
    seed: int = 0,
    n_resamples: int = DEFAULT_N_RESAMPLES,
    rate_cluster_ids: Sequence[str] | None = None,
) -> CalibrationRecord:
    """Return a copy of ``record`` with the Rogan-Gladen corrected rate + CI attached.

    ``seed`` and ``n_resamples`` (the Monte Carlo draw count) drive the interval and are stamped
    on the returned record's ``provenance``, so the interval can be recomputed from the stored
    record alone.

    Refuses for an uncertified judge, and with :class:`RateUnsafeError` for a record whose gold is
    below its coverage floor, the larger of the planned and the observed one,
    or whose observed rates have no measured floor, and with :class:`RateSampleTooSmallError` for a
    rate sample of fewer than :data:`~calibrated_agent_evaluation.calibrate.plan.MIN_RATE_SAMPLE`
    traces (distinct traces when ``rate_cluster_ids`` is given). When J<0.5 the corrected *point* is
    refused (the interval is still returned as the signal); when the rate is non-identifiable the
    point is refused and the bound reported. When more than one tail's share of the interval's draws
    have no corrected rate, the interval is refused too: ``(lo, hi) = (0, 1)`` with a
    ``refusal_reason`` starting ``"non-identifiable interval:"``. The interval always stays finite
    and in [0,1].

    Args:
        record: A CERTIFIED calibration record for the judge.
        rate_sample_verdicts: The judge's verdicts (1 = FAIL, 0 = PASS) on a separate
            production-representative sample; their mean is the raw rate.
        rate_set_id: The id of that sample; it must differ from ``record.hunted_set_id``.
        rate_trace_ids: The trace id of each verdict, in the same order. Required, because the
            rate sample must be checked against the calibration gold: hunted traces re-entering
            it break the independence the interval assumes.
        seed: Fixes the Monte Carlo draws behind the interval.
        n_resamples: The number of Monte Carlo draws.
        rate_cluster_ids: The trace id of each verdict, when one trace contributes several and
            the interval should account for the clustering.

    Returns:
        A copy of ``record`` with the corrected rate and its interval attached.

    Raises:
        UncalibratedJudgeError: ``record`` is not CERTIFIED.
        RateUnsafeError: The record's gold is below its coverage floor.
        SampleMisuseError: The rate sample is empty, is the hunted calibration set, shares
            traces with the calibration gold, or its trace ids do not match its verdicts.
        RateSampleTooSmallError: The rate sample has fewer than ``MIN_RATE_SAMPLE`` traces.
    """
    if record.result != "CERTIFIED":
        raise UncalibratedJudgeError(
            f"judge {record.judge_id!r} has no CERTIFIED calibration (result={record.result!r}); "
            "refusing to compute or display a corrected rate"
        )
    if record.below_coverage_floor:
        raise RateUnsafeError(_rate_unsafe_message(record))
    if rate_set_id == record.hunted_set_id:
        raise SampleMisuseError(
            f"rate_set_id {rate_set_id!r} equals the hunted_set_id; the true rate must come from a "
            "SEPARATE production-representative sample, never the hunted fail-heavy set"
        )
    if not rate_sample_verdicts:
        raise SampleMisuseError("corrected_rate: the rate sample must be non-empty")
    if len(rate_trace_ids) != len(rate_sample_verdicts) or not all(
        isinstance(t, str) and t for t in rate_trace_ids
    ):
        raise SampleMisuseError(
            f"corrected_rate: rate_trace_ids must name the trace of each of the "
            f"{len(rate_sample_verdicts)} verdict(s), in order, got {len(rate_trace_ids)} id(s); "
            "without them the rate sample cannot be checked against the calibration gold"
        )
    _refuse_calibration_overlap(record, rate_trace_ids, rate_cluster_ids)
    n_traces = len(rate_sample_verdicts) if rate_cluster_ids is None else len(set(rate_cluster_ids))
    if n_traces < MIN_RATE_SAMPLE:
        raise RateSampleTooSmallError(
            f"the rate sample has {n_traces} trace(s); a corrected rate needs at least "
            f"{MIN_RATE_SAMPLE}, so score {MIN_RATE_SAMPLE - n_traces} more "
            "production-representative trace(s) and rerun. Below that the interval was measured "
            "to cover the true rate less often than it claims.",
            n=n_traces,
            minimum=MIN_RATE_SAMPLE,
        )

    measured = sum(1 for v in rate_sample_verdicts if v) / len(rate_sample_verdicts)
    tpr_point, tnr_point = record.tpr.point, record.tnr.point

    # Reconstruct the hunted per-class verdict arrays from the confusion counts so the double
    # bootstrap can resample them alongside the eval sample. The arrays are 1/0 draws whose
    # means are exactly TPR (positive class) and 1-TNR (negative class) -- verdict==FAIL indicators.
    n_pos, n_neg = record.n_pos, record.n_neg
    tp = round(tpr_point * n_pos)
    fp = round((1.0 - tnr_point) * n_neg)
    cal_pos = [1] * tp + [0] * (n_pos - tp)  # judge-FAIL on the true-FAIL class -> mean = TPR
    cal_neg = [1] * fp + [0] * (n_neg - fp)  # judge-FAIL on the true-PASS class -> mean = 1-TNR
    try:
        drawn = rg_interval(
            cal_pos,
            cal_neg,
            list(rate_sample_verdicts),
            conf=RG_LEVEL,
            n_resamples=n_resamples,
            seed=seed,
            eval_cluster_ids=rate_cluster_ids,
            cal_pos_design_effect=record.tpr_design_effect,
            cal_neg_design_effect=record.tnr_design_effect,
        )
    except NonIdentifiableRateError as exc:
        # Too many draws without a corrected rate to place a tail:
        # refuse the interval as well as the point. [0, 1] claims nothing and keeps the mode
        # visible with the reason, rather than an interval over a mixture of estimands.
        lo, hi, interval_refusal = 0.0, 1.0, f"non-identifiable interval: {exc}"
    else:
        lo, hi, interval_refusal = drawn.lo, drawn.hi, None

    j = youden_j(tpr_point, tnr_point)
    n_rate = len(rate_sample_verdicts)

    def _interval(point: float | None, reason: str | None) -> Interval:
        return Interval(
            point=point,
            lo=lo,
            hi=hi,
            refusal_reason=reason,
            method=METHOD_RG_JEFFREYS if rate_cluster_ids is None else METHOD_RG_JEFFREYS_CLUSTERED,
            level=RG_LEVEL,
            sided="two-sided",
            n=n_rate,
        )

    if interval_refusal is not None:
        interval = _interval(None, interval_refusal)
    elif j < YOUDEN_FLOOR:
        interval = _interval(None, f"youden_j={j:.3f}<0.5")
    else:
        try:
            point = rogan_gladen(measured, tpr_point, tnr_point)
        except NonIdentifiableRateError:
            # Outside the identifiable region -- refuse the point, report the bound.
            interval = _interval(None, "non-identifiable")
        else:
            interval = _interval(point, None)

    stamp = capture_provenance(
        seed=seed,
        n_resamples=n_resamples,
        thresholds=_gate_thresholds(),
        sampling=record.provenance.sampling,
    )
    return replace(record, corrected_rate=interval, rate_set_id=rate_set_id, provenance=stamp)

fingerprint_of

fingerprint_of(judge)

The judge's content fingerprint, or None for a judge that carries none.

Source code in src/calibrated_agent_evaluation/calibrate/engine.py
def fingerprint_of(judge: object) -> str | None:
    """The judge's content fingerprint, or ``None`` for a judge that carries none."""
    fingerprint = getattr(judge, "fingerprint", None)
    return fingerprint if isinstance(fingerprint, str) else None

pin_mismatch

pin_mismatch(record, judge)

Why record is not pinned to judge as it is now, or None when it is.

record is any record pinning a judge (calibration, bias battery): it carries judge_id, model, prompt_version and judge_fingerprint. A record made for another judge id never pins this one. A judge with a fingerprint matches only a record stamped with the same one, so an in-place prompt edit under an unchanged prompt_version is caught, and a record stamped before fingerprints existed proves nothing about the current prompt. A record stamped with a fingerprint never pins a judge that carries none, since nothing shows that judge's content is the one measured. A judge without a fingerprint matches an unstamped record on (judge_id, model, prompt_version) alone.

Source code in src/calibrated_agent_evaluation/calibrate/engine.py
def pin_mismatch(record: object, judge: Judge) -> str | None:
    """Why ``record`` is not pinned to ``judge`` as it is now, or ``None`` when it is.

    ``record`` is any record pinning a judge (calibration, bias battery): it carries ``judge_id``,
    ``model``, ``prompt_version`` and ``judge_fingerprint``. A record made for another judge id
    never pins this one. A judge with a fingerprint matches only a record stamped with the same
    one, so an in-place prompt edit under an unchanged ``prompt_version`` is caught, and a record
    stamped before fingerprints existed proves nothing about the current prompt. A record stamped
    with a fingerprint never pins a judge that carries none, since nothing shows that judge's
    content is the one measured. A judge without a fingerprint matches an unstamped record on
    ``(judge_id, model, prompt_version)`` alone.
    """
    recorded_id = getattr(record, "judge_id", None)
    if recorded_id != judge.judge_id:
        return (
            f"this record was made for judge {recorded_id!r}, not {judge.judge_id!r}; a judge is "
            "certified on its own locked test"
        )
    if getattr(record, "model", None) != judge.model:
        return (
            f"the judge's model changed since this record was made "
            f"({getattr(record, 'model', None)} to {judge.model}); re-certify on a fresh locked set"
        )
    if getattr(record, "prompt_version", None) != judge.prompt_version:
        return (
            f"the judge's prompt version changed since this record was made "
            f"({getattr(record, 'prompt_version', None)} to {judge.prompt_version}); re-certify "
            "on a fresh locked set"
        )
    current = fingerprint_of(judge)
    stamped = getattr(record, "judge_fingerprint", None)
    if current is None:
        if stamped is None:
            return None
        return (
            "this record was made for a judge whose content was fingerprinted, and this judge "
            "carries no fingerprint, so nothing shows it is the judge that was measured; certify "
            "it on its own locked test"
        )
    if stamped is None:
        return (
            "this record was made before judge prompts were fingerprinted, so it cannot show the "
            "prompt is unchanged; re-certify on a fresh locked set"
        )
    if stamped != current:
        return (
            "the judge's prompt changed since it was certified (same prompt version, different "
            "content); re-certify on a fresh locked set"
        )
    return None

coverage_floor

coverage_floor(p_hat)

Return the smallest per-class n at which the corrected-rate interval covers nominally.

Read from COVERAGE_FLOOR, which was measured at a set of p_hat values. Between two measured values the larger of their floors applies; below the lowest measured value (0.85) its floor applies, an extrapolation that rests on coverage improving as the per-class error count grows. Depends on p_hat only: coverage was measured across prevalence 0.02 to 0.50 and taken at its worst, and prevalence is what the rate sample estimates, so it cannot be an input.

Raises InfeasiblePlanError when p_hat is outside (0, 1] or no measured size covers nominally there.

Source code in src/calibrated_agent_evaluation/calibrate/plan.py
def coverage_floor(p_hat: float) -> int:
    """Return the smallest per-class ``n`` at which the corrected-rate interval covers nominally.

    Read from :data:`COVERAGE_FLOOR`, which was measured at a set of ``p_hat`` values. Between two
    measured values the larger of their floors applies; below the lowest measured value (0.85) its
    floor applies, an extrapolation that rests on coverage improving as the per-class error count
    grows. Depends on ``p_hat`` only: coverage was measured across prevalence 0.02 to 0.50 and
    taken at its worst, and prevalence is what the rate sample estimates, so it cannot be an input.

    Raises :class:`InfeasiblePlanError` when ``p_hat`` is outside (0, 1] or no measured size
    covers nominally there.
    """
    if not 0.0 < p_hat <= 1.0:
        raise InfeasiblePlanError(f"coverage_floor: p_hat must be in (0, 1], got {p_hat}")
    grid = COVERAGE_FLOOR
    if p_hat <= grid[0][0] + 1e-12:
        candidates = [grid[0]]
    else:
        upper = next(i for i, (p, _) in enumerate(grid) if p_hat <= p + 1e-12)
        on_point = abs(grid[upper][0] - p_hat) <= 1e-12
        candidates = [grid[upper]] if on_point else [grid[upper - 1], grid[upper]]
    if any(n is None for _, n in candidates):
        measured = ", ".join(f"{p:.2f}" for p, _ in candidates)
        raise InfeasiblePlanError(
            "coverage_floor: the corrected rate's interval was measured at p_hat="
            f"{measured} and did not cover nominally at any scanned per-class size up to 260; "
            "plan at an accuracy with a measured floor"
        )
    return max(n for _, n in candidates if n is not None)

double_coded_per_class

double_coded_per_class(minimum_overlap=MINIMUM_OVERLAP)

The per-class gold that holds minimum_overlap double-coded traces, 65 at the default.

The calibration gold is balanced per class, so the agreement gate's overlap splits evenly across FAIL and PASS (rounded up).

Source code in src/calibrated_agent_evaluation/calibrate/plan.py
def double_coded_per_class(minimum_overlap: int = MINIMUM_OVERLAP) -> int:
    """The per-class gold that holds ``minimum_overlap`` double-coded traces, 65 at the default.

    The calibration gold is balanced per class, so the agreement gate's overlap splits evenly
    across FAIL and PASS (rounded up).
    """
    if minimum_overlap < 1:
        raise InfeasiblePlanError(
            f"double_coded_per_class: minimum_overlap must be at least 1, got {minimum_overlap}"
        )
    return math.ceil(minimum_overlap / 2)

gate_fails_allowed

gate_fails_allowed(n, threshold, alpha=0.1)

The most FAIL verdicts among n that still clear the run gate; -1 when none do.

Source code in src/calibrated_agent_evaluation/calibrate/plan.py
def gate_fails_allowed(n: int, threshold: float, alpha: float = 0.10) -> int:
    """The most FAIL verdicts among ``n`` that still clear the run gate; -1 when none do."""
    allowed = -1
    for fails in range(n + 1):
        if wilson_lower(n - fails, n, alpha=alpha) < threshold:
            break
        allowed = fails
    return allowed

gate_set_size

gate_set_size(expected_pass, threshold)

The traces per failure mode the run gate needs to pass a suite that passes as expected.

calibrated-agent-evaluation run passes a mode when the 95% one-sided Wilson lower bound of its pass rate reaches threshold, the same bound the calibration gate reads. So the size is wilson_calibration_size at expected_pass: the smallest n from which a suite passing expected_pass of the time (count rounded down) clears the gate, at n and at every larger size, so adding traces never turns a suite as good as expected red.

Raises:

Type Description
InfeasiblePlanError

threshold is at or above expected_pass, or the size exceeds 100,000.

Source code in src/calibrated_agent_evaluation/calibrate/plan.py
def gate_set_size(expected_pass: float, threshold: float) -> int:
    """The traces per failure mode the run gate needs to pass a suite that passes as expected.

    ``calibrated-agent-evaluation run`` passes a mode when the 95% one-sided Wilson lower bound of
    its pass rate reaches ``threshold``, the same bound the calibration gate reads. So the size is
    :func:`wilson_calibration_size` at ``expected_pass``: the smallest ``n`` from which a suite
    passing ``expected_pass`` of the time (count rounded down) clears the gate, at ``n`` and at
    every larger size, so adding traces never turns a suite as good as expected red.

    Raises:
        InfeasiblePlanError: ``threshold`` is at or above ``expected_pass``, or the size exceeds
            100,000.
    """
    return wilson_calibration_size(expected_pass, threshold)

plan_calibration

plan_calibration(
    p_hat,
    tau,
    alpha=0.1,
    *,
    tnr_hat=None,
    rate_safe=False,
    raters=1,
)

The sizes behind plan_calibration_size, each kept; see that function.

rate_safe -- for a caller who wants a corrected rate: per_class is then rate_safe_calibration_size, and certification and coverage_floor are the same as without it.

raters -- how many raters will code the gold. From two up the agreement gate measures them on the gold itself, so per_class is raised to double_coded_per_class when it is smaller.

Parameters:

Name Type Description Default
p_hat float

The accuracy you expect the judge to have on each class (or on TPR, with tnr_hat set).

required
tau float

The lower bound each class must clear, normally GATE_THRESHOLD (0.80).

required
alpha float

The two-sided level of the Wilson bound; 0.10 gives a one-sided 95% bound.

0.1
tnr_hat float | None

The expected TNR when it differs from p_hat.

None
rate_safe bool

Size the plan for a corrected rate as well as certification.

False
raters int

How many raters will code the gold (1 or more).

1

Returns:

Type Description
CalibrationPlan

The per-class label count, the certification size, the coverage floor and, for

CalibrationPlan

double-coded gold, the per-class size the agreement overlap needs.

Raises:

Type Description
InfeasiblePlanError

No label count can clear tau at the expected accuracy, or raters is below 1.

Source code in src/calibrated_agent_evaluation/calibrate/plan.py
def plan_calibration(
    p_hat: float,
    tau: float,
    alpha: float = 0.10,
    *,
    tnr_hat: float | None = None,
    rate_safe: bool = False,
    raters: int = 1,
) -> CalibrationPlan:
    """The sizes behind :func:`plan_calibration_size`, each kept; see that function.

    ``rate_safe`` -- for a caller who wants a corrected rate: ``per_class`` is then
    :func:`rate_safe_calibration_size`, and ``certification`` and ``coverage_floor`` are the same
    as without it.

    ``raters`` -- how many raters will code the gold. From two up the agreement gate measures
    them on the gold itself, so ``per_class`` is raised to :func:`double_coded_per_class` when it
    is smaller.

    Args:
        p_hat: The accuracy you expect the judge to have on each class (or on TPR, with
            ``tnr_hat`` set).
        tau: The lower bound each class must clear, normally ``GATE_THRESHOLD`` (0.80).
        alpha: The two-sided level of the Wilson bound; 0.10 gives a one-sided 95% bound.
        tnr_hat: The expected TNR when it differs from ``p_hat``.
        rate_safe: Size the plan for a corrected rate as well as certification.
        raters: How many raters will code the gold (1 or more).

    Returns:
        The per-class label count, the certification size, the coverage floor and, for
        double-coded gold, the per-class size the agreement overlap needs.

    Raises:
        InfeasiblePlanError: No label count can clear ``tau`` at the expected accuracy, or
            ``raters`` is below 1.
    """
    if tnr_hat is not None and not 0.0 < tnr_hat <= 1.0:
        raise InfeasiblePlanError(
            f"plan_calibration_size: tnr_hat must be in (0, 1], got {tnr_hat}"
        )
    if raters < 1:
        raise InfeasiblePlanError(f"plan_calibration: raters must be at least 1, got {raters}")
    tpr, tnr = p_hat, p_hat if tnr_hat is None else tnr_hat
    certification = wilson_calibration_size(min(tpr, tnr), tau, alpha)
    floor = coverage_floor(max(tpr, tnr))
    per_class = max(certification, floor)
    if rate_safe:
        per_class = rate_safe_calibration_size(p_hat, tau, alpha, tnr_hat=tnr_hat)
    double_coded = double_coded_per_class() if raters >= 2 else None
    if double_coded is not None:
        per_class = max(per_class, double_coded)
    return CalibrationPlan(
        per_class=per_class,
        certification=certification,
        coverage_floor=floor,
        double_coded=double_coded,
    )

plan_calibration_size

plan_calibration_size(
    p_hat, tau, alpha=0.1, *, tnr_hat=None, raters=1
)

Return the per-class n to hunt: enough to certify the judge and to estimate the rate.

The larger of wilson_calibration_size (the gate's own bound, which depends on tau and alpha) and coverage_floor (the corrected-rate interval's measured minimum, which depends on the expected accuracy only). The certification size is computed first, so a tau the expected accuracy cannot reach is refused as before; the floor can only raise a feasible plan or refuse it, never make an infeasible one feasible.

tnr_hat -- for a judge expected to be more accurate on one class than the other, its expected TNR; p_hat is then its expected TPR. Both classes are hunted at the returned size. The gate must certify both rates, so the certification size is taken at the lower of the two; the floor is taken at the higher, the rule the coverage simulations checked (see the module docstring). Omitted, the judge is expected to be equally accurate on both classes and the plan is the one for p_hat alone.

raters -- from two up, the gold is also the raters' agreement overlap, and the size is at least double_coded_per_class (65 per class at the defaults).

Raises InfeasiblePlanError when either job has no answer.

Source code in src/calibrated_agent_evaluation/calibrate/plan.py
def plan_calibration_size(
    p_hat: float,
    tau: float,
    alpha: float = 0.10,
    *,
    tnr_hat: float | None = None,
    raters: int = 1,
) -> int:
    """Return the per-class ``n`` to hunt: enough to certify the judge and to estimate the rate.

    The larger of :func:`wilson_calibration_size` (the gate's own bound, which depends on ``tau``
    and ``alpha``) and :func:`coverage_floor` (the corrected-rate interval's measured minimum,
    which depends on the expected accuracy only). The certification size is computed first, so a
    ``tau`` the expected accuracy cannot reach is refused as before; the floor can only raise a
    feasible plan or refuse it, never make an infeasible one feasible.

    ``tnr_hat`` -- for a judge expected to be more accurate on one class than the other, its
    expected TNR; ``p_hat`` is then its expected TPR. Both classes are hunted at the returned
    size. The gate must certify both rates, so the certification size is taken at the lower of the
    two; the floor is taken at the higher, the rule the coverage simulations checked (see the
    module docstring). Omitted, the judge is
    expected to be equally accurate on both classes and the plan is the one for ``p_hat`` alone.

    ``raters`` -- from two up, the gold is also the raters' agreement overlap, and the size is at
    least :func:`double_coded_per_class` (65 per class at the defaults).

    Raises :class:`InfeasiblePlanError` when either job has no answer.
    """
    return plan_calibration(p_hat, tau, alpha, tnr_hat=tnr_hat, raters=raters).per_class

rate_safe_at

rate_safe_at(n, tau, alpha=0.1)

Whether n per class is rate-safe at every accuracy that would certify at n.

Certification marks a judge rate-unsafe when its gold is below the coverage floor at the TPR and TNR it showed. At n per class the gate certifies a class on k correct when the Wilson lower bound of k of n reaches tau, so a judge can show any such k/n. n is rate-safe when the floor at every one of them is at most n. Accuracies above the highest one with a measured floor are skipped: no size is known to make the interval cover there, so no plan can help, and certify marks those records rate-unsafe at any size. With the committed table every accuracy has a measured floor, so nothing is skipped.

Source code in src/calibrated_agent_evaluation/calibrate/plan.py
def rate_safe_at(n: int, tau: float, alpha: float = 0.10) -> bool:
    """Whether ``n`` per class is rate-safe at every accuracy that would certify at ``n``.

    Certification marks a judge rate-unsafe when its gold is below the coverage floor at
    the TPR and TNR it showed. At ``n`` per class the gate certifies a class on ``k`` correct when
    the Wilson lower bound of ``k`` of ``n`` reaches ``tau``, so a judge can show any such ``k/n``.
    ``n`` is rate-safe when the floor at every one of them is at most ``n``. Accuracies above the
    highest one with a measured floor are skipped: no size is known to make the interval cover
    there, so no plan can help, and certify marks those records rate-unsafe at any size. With the
    committed table every accuracy has a measured floor, so nothing is skipped.
    """
    highest, _ = _highest_measured()
    for k in range(n, -1, -1):
        if wilson_lower(k, n, alpha=alpha) < tau:
            break  # the bound rises with k, so no smaller count certifies either
        if k / n > highest + 1e-12:
            continue
        if coverage_floor(k / n) > n:
            return False
    return True

rate_safe_calibration_size

rate_safe_calibration_size(
    p_hat, tau, alpha=0.1, *, tnr_hat=None
)

The smallest per-class n to hunt when the caller wants a corrected rate.

At least the certification-only plan (plan_calibration_size), and rate-safe (rate_safe_at) there and at every larger size, so a judge more accurate than planned does not make its own record rate-unsafe. Every size from the largest measured floor up is rate-safe, since no certifying accuracy with a measured floor needs more, so the search runs down from there. With the committed table this is 162 per class at every planned accuracy from 0.88 to 0.99 at tau=0.80, because below 162 some count of errors always gives an accuracy between 0.98 and 0.99, whose floor is 162; at 0.85 certifying alone needs 208.

Raises InfeasiblePlanError where the certification-only plan does: tau the expected accuracy cannot reach, or a planned rate with no measured floor.

Source code in src/calibrated_agent_evaluation/calibrate/plan.py
def rate_safe_calibration_size(
    p_hat: float, tau: float, alpha: float = 0.10, *, tnr_hat: float | None = None
) -> int:
    """The smallest per-class ``n`` to hunt when the caller wants a corrected rate.

    At least the certification-only plan (:func:`plan_calibration_size`), and rate-safe
    (:func:`rate_safe_at`) there and at every larger size, so a judge more accurate than planned
    does not make its own record rate-unsafe. Every size from the largest measured floor up is
    rate-safe, since no certifying accuracy with a measured floor needs more, so the search runs
    down from there. With the committed table this is 162 per class at every planned accuracy from
    0.88 to 0.99 at ``tau=0.80``, because below 162 some count of errors always gives an accuracy
    between 0.98 and 0.99, whose floor is 162; at 0.85 certifying alone needs 208.

    Raises :class:`InfeasiblePlanError` where the certification-only plan does: ``tau`` the
    expected accuracy cannot reach, or a planned rate with no measured floor.
    """
    start = plan_calibration(p_hat, tau, alpha, tnr_hat=tnr_hat).per_class
    n = max(start, _highest_measured()[1])
    while n > start and rate_safe_at(n - 1, tau, alpha):
        n -= 1
    return n

wilson_calibration_size

wilson_calibration_size(p_hat, tau, alpha=0.1)

Return the smallest per-class n from which the gate's bound at p_hat clears tau.

The certification size alone. plan_calibration_size is what to hunt, because it also covers the corrected rate's interval.

p_hat -- the judge accuracy you expect to observe (e.g. 0.90). tau -- the gate threshold the lower bound must clear (e.g. 0.80). alpha -- the two-sided level whose lower edge is the one-sided bound (default 0.10 -> 95% LB).

The returned n holds from there on: for every n' >= n, a judge that gets floor(p_hat * n') of n' right clears the bound. The first n that clears is not enough, because the bound is a sawtooth in n: at p_hat=0.90 it first clears at 48, then 51 to 54 fall short again (51 labels at 90% is 45 correct, whose bound is 0.788). Collecting more labels than planned must never un-certify a judge as accurate as planned.

The search is finite. With z the one-sided normal quantile and k = floor(p_hat * n), so k / n >= p_hat - 1 / n, the Wilson lower bound is at least (p_hat - 1 / n - z / (2 sqrt(n))) / (1 + z^2 / n), using k (n - k) / n^2 <= 1 / 4. That clears tau once p_hat - tau >= z / (2 sqrt(n)) + (1 + tau z^2) / n, whose right side falls as n grows, so every size from the first n meeting it clears the bound. The search checks the sizes below that one.

Raises InfeasiblePlanError when tau >= p_hat (the point estimate never clears the threshold, so its lower bound cannot either) or when the guarantee needs more than 100,000 per class.

Source code in src/calibrated_agent_evaluation/calibrate/plan.py
def wilson_calibration_size(p_hat: float, tau: float, alpha: float = 0.10) -> int:
    """Return the smallest per-class ``n`` from which the gate's bound at ``p_hat`` clears ``tau``.

    The certification size alone. :func:`plan_calibration_size` is what to hunt, because it also
    covers the corrected rate's interval.

    ``p_hat`` -- the judge accuracy you expect to observe (e.g. 0.90).
    ``tau``   -- the gate threshold the lower bound must clear (e.g. 0.80).
    ``alpha`` -- the two-sided level whose lower edge is the one-sided bound (default 0.10 -> 95%
    LB).

    The returned ``n`` holds from there on: for every ``n' >= n``, a judge that gets
    ``floor(p_hat * n')`` of ``n'`` right clears the bound. The first ``n`` that clears is not
    enough, because the bound is a sawtooth in ``n``: at ``p_hat=0.90`` it first clears at 48, then
    51 to 54 fall short again (51 labels at 90% is 45 correct, whose bound is 0.788). Collecting
    more labels than planned must never un-certify a judge as accurate as planned.

    The search is finite. With ``z`` the one-sided normal quantile and ``k = floor(p_hat * n)``, so
    ``k / n >= p_hat - 1 / n``, the Wilson lower bound is at least
    ``(p_hat - 1 / n - z / (2 sqrt(n))) / (1 + z^2 / n)``, using ``k (n - k) / n^2 <= 1 / 4``. That
    clears ``tau`` once ``p_hat - tau >= z / (2 sqrt(n)) + (1 + tau z^2) / n``, whose right side
    falls as ``n`` grows, so every size from the first ``n`` meeting it clears the bound. The search
    checks the sizes below that one.

    Raises :class:`InfeasiblePlanError` when ``tau >= p_hat`` (the point estimate never clears the
    threshold, so its lower bound cannot either) or when the guarantee needs more than
    100,000 per class.
    """
    if not 0.0 < p_hat <= 1.0:
        raise InfeasiblePlanError(f"wilson_calibration_size: p_hat must be in (0, 1], got {p_hat}")
    if not 0.0 < tau < 1.0:
        raise InfeasiblePlanError(f"wilson_calibration_size: tau must be in (0, 1), got {tau}")
    if tau >= p_hat:
        raise InfeasiblePlanError(
            f"wilson_calibration_size: tau={tau} >= p_hat={p_hat}; no finite sample clears the "
            "bound (the point estimate itself never reaches the threshold, so its lower bound "
            "cannot)"
        )
    z = NormalDist().inv_cdf(1.0 - alpha / 2.0)
    margin = p_hat - tau
    beyond = next(
        (
            n
            for n in range(2, _MAX_N_PER_CLASS + 1)
            if margin >= z / (2.0 * math.sqrt(n)) + (1.0 + tau * z * z) / n
        ),
        None,
    )
    if beyond is None:
        raise InfeasiblePlanError(
            f"wilson_calibration_size: no size up to {_MAX_N_PER_CLASS} is known to keep the bound "
            f"above tau={tau} at p_hat={p_hat} (the target is too close to p_hat for a practical "
            "hunt)"
        )
    planned = beyond
    for n in range(beyond - 1, 1, -1):
        if wilson_lower(_correct_at(p_hat, n), n, alpha=alpha) < tau:
            break
        planned = n
    return planned