Skip to content

calibrated_agent_evaluation.stats

calibrated_agent_evaluation.stats

Statistics used by the calibration gate and the later stages.

Wilson lower bounds and Holm/BY multiplicity come from statsmodels. The Rogan-Gladen interval (Jeffreys posteriors), Youden's J and Cohen's kappa are implemented here, kappa to avoid a large scikit-learn dependency. Saturation (Good-Turing, Kish), McNemar and drift live in their own modules in this package. Pure functions, no vendor trace type, no I/O.

McNemarStat dataclass

McNemarStat(statistic, p_value, method, n_discordant)

The pure McNemar computation (the improve-stage McNemarResult wraps it).

statistic is min(b, c) for the exact/mid-p path (the binomial statistic statsmodels reports) or the uncorrected chi-square (b - c)^2 / (b + c) for the asymptotic path. n_discordant is b + c.

statistic instance-attribute

statistic

p_value instance-attribute

p_value

method instance-attribute

method

n_discordant instance-attribute

n_discordant

cohen_kappa

cohen_kappa(gold, verdict)

Return Cohen's κ between two equal-length label sequences.

κ = (p_o - p_e) / (1 - p_e), where p_o is observed agreement and p_e is the agreement expected by chance from the marginals. Returns 1.0 for perfect agreement; 0.0 when observed equals chance. When p_e == 1 (both raters gave a single constant label) κ is undefined -- returns 1.0 if they fully agree, else 0.0 (the standard degenerate-case convention).

Raises StatisticsInputError on empty or unequal-length inputs.

Source code in src/calibrated_agent_evaluation/stats/agreement.py
def cohen_kappa(gold: Sequence[object], verdict: Sequence[object]) -> float:
    """Return Cohen's κ between two equal-length label sequences.

    κ = (p_o - p_e) / (1 - p_e), where ``p_o`` is observed agreement and ``p_e`` is the agreement
    expected by chance from the marginals. Returns 1.0 for perfect agreement; 0.0 when observed
    equals chance. When ``p_e == 1`` (both raters gave a single constant label) κ is undefined --
    returns 1.0 if they fully agree, else 0.0 (the standard degenerate-case convention).

    Raises :class:`StatisticsInputError` on empty or unequal-length inputs.
    """
    if len(gold) != len(verdict):
        raise StatisticsInputError(
            f"cohen_kappa: gold and verdict must be equal length, "
            f"got {len(gold)} and {len(verdict)}"
        )
    n = len(gold)
    if n == 0:
        raise StatisticsInputError("cohen_kappa: inputs must be non-empty")

    labels = set(gold) | set(verdict)
    observed = sum(1 for g, v in zip(gold, verdict, strict=True) if g == v) / n
    expected = sum(
        (sum(1 for g in gold if g == lab) / n) * (sum(1 for v in verdict if v == lab) / n)
        for lab in labels
    )
    if expected >= 1.0:
        # Degenerate: both sequences are a single constant label. κ is undefined; by convention
        # 1.0 if they agree everywhere, else 0.0.
        return 1.0 if observed >= 1.0 else 0.0
    return (observed - expected) / (1.0 - expected)

minority_fraction

minority_fraction(labels)

Return the fraction of the least common label value, or 0.0 for fewer than two values.

Source code in src/calibrated_agent_evaluation/stats/agreement.py
def minority_fraction(labels: Sequence[object]) -> float:
    """Return the fraction of the least common label value, or 0.0 for fewer than two values."""
    if not labels:
        return 0.0
    counts: dict[object, int] = {}
    for lab in labels:
        counts[lab] = counts.get(lab, 0) + 1
    if len(counts) < 2:
        return 0.0
    return min(counts.values()) / len(labels)

should_hide_raw_agreement

should_hide_raw_agreement(labels)

Return whether raw percent agreement should be hidden for this label set.

True when the minority class is below IMBALANCE_THRESHOLD -- under imbalance raw agreement flatters a do-nothing judge, so the report surfaces κ instead. A set with a single label value is the extreme case (the other class has 0% of the labels) and is hidden: a judge that answers FAIL to everything scores 100% raw agreement on an all-FAIL set. An empty set has nothing to hide and returns False; the callers refuse an empty pairing themselves.

Source code in src/calibrated_agent_evaluation/stats/agreement.py
def should_hide_raw_agreement(labels: Sequence[object]) -> bool:
    """Return whether raw percent agreement should be hidden for this label set.

    True when the minority class is below :data:`IMBALANCE_THRESHOLD` -- under imbalance raw
    agreement flatters a do-nothing judge, so the report surfaces κ instead. A set with a single
    label value is the extreme case (the other class has 0% of the labels) and is hidden: a judge
    that answers FAIL to everything scores 100% raw agreement on an all-FAIL set. An empty set has
    nothing to hide and returns False; the callers refuse an empty pairing themselves.
    """
    return len(labels) > 0 and minority_fraction(labels) < IMBALANCE_THRESHOLD

jensen_shannon

jensen_shannon(p, q)

Jensen-Shannon divergence (0..1, base 2) of two frequency/probability vectors.

scipy.spatial.distance.jensenshannon returns the JS distance (the square root of the divergence); this squares it back so the drift report reads a divergence bounded in [0, 1]. p and q need not be normalized -- jensenshannon normalizes each to a probability vector -- but they must be the same length and not both all-zero (which yields NaN).

Raises StatisticsInputError on an empty / mismatched / non-numeric input or a non-finite result (an all-zero vector) -- never a raw exception, never a NaN.

Source code in src/calibrated_agent_evaluation/stats/drift.py
def jensen_shannon(p: Sequence[float], q: Sequence[float]) -> float:
    """Jensen-Shannon **divergence** (0..1, base 2) of two frequency/probability vectors.

    ``scipy.spatial.distance.jensenshannon`` returns the JS *distance* (the square root of the
    divergence); this squares it back so the drift report reads a divergence bounded in ``[0, 1]``.
    ``p`` and ``q`` need not be normalized -- ``jensenshannon`` normalizes each to a probability
    vector -- but they must be the same length and not both all-zero (which yields NaN).

    Raises :class:`StatisticsInputError` on an empty / mismatched / non-numeric input or a
    non-finite result (an all-zero vector) -- never a raw exception, never a NaN.
    """
    pa = _as_float_array(p, name="p")
    qa = _as_float_array(q, name="q")
    if pa.shape != qa.shape:
        raise StatisticsInputError(
            f"drift: jensen_shannon needs equal-length vectors, got {pa.shape[0]} and {qa.shape[0]}"
        )
    if pa.sum() <= 0 or qa.sum() <= 0:
        raise StatisticsInputError(
            "drift: jensen_shannon needs a positive-mass vector on each side (an all-zero "
            "distribution has no probability mass)"
        )
    return _finite(_jsd_arrays(pa, qa), metric="jensen_shannon")

permutation_p

permutation_p(
    statistic,
    baseline,
    fresh,
    *,
    n_perm=DEFAULT_N_PERMUTATIONS,
    seed=0,
)

One-sided permutation p-value that statistic(baseline, fresh) is no larger than chance.

The pooled sample is relabelled n_perm times with numpy.random.default_rng(seed) (group sizes kept), and p = (1 + #{null >= observed}) / (1 + n_perm), which is never 0 and is a valid p-value at any n. statistic receives float arrays and returns a float.

Raises StatisticsInputError on empty / non-numeric input or n_perm < 1.

Source code in src/calibrated_agent_evaluation/stats/drift.py
def permutation_p(
    statistic: ArrayStatistic,
    baseline: Sequence[float],
    fresh: Sequence[float],
    *,
    n_perm: int = DEFAULT_N_PERMUTATIONS,
    seed: int = 0,
) -> float:
    """One-sided permutation p-value that ``statistic(baseline, fresh)`` is no larger than chance.

    The pooled sample is relabelled ``n_perm`` times with ``numpy.random.default_rng(seed)``
    (group sizes kept), and ``p = (1 + #{null >= observed}) / (1 + n_perm)``, which is never 0 and
    is a valid p-value at any ``n``. ``statistic`` receives float arrays and returns a float.

    Raises :class:`StatisticsInputError` on empty / non-numeric input or ``n_perm < 1``.
    """
    if n_perm < 1:
        raise StatisticsInputError(f"drift: permutation_p needs n_perm >= 1, got {n_perm}")
    base = _as_float_array(baseline, name="baseline")
    fut = _as_float_array(fresh, name="fresh")
    observed = _finite(statistic(base, fut), metric="permutation statistic")
    pooled = np.concatenate([base, fut])
    nb = len(base)
    rng = np.random.default_rng(seed)
    # A relative tolerance so a relabelling that reproduces the observed value counts as >=.
    bar = observed - 1e-12 * max(1.0, abs(observed))
    hits = 0
    for _ in range(n_perm):
        perm = rng.permutation(pooled)
        if statistic(perm[:nb], perm[nb:]) >= bar:
            hits += 1
    return (1 + hits) / (1 + n_perm)

psi

psi(baseline, fresh, *, bins=10)

Population Stability Index between two raw-value samples over baseline-quantile bins.

PSI = sum_i (f_i - b_i) * ln(f_i / b_i) over the bin proportions b (baseline) and f (fresh), each computed from counts plus 0.5. Bins are the baseline's quantiles (deciles for bins=10), merged until each holds 5 baseline values. 0 for identical samples.

Raises StatisticsInputError on an empty / non-numeric / non-finite input, on bins < 2, when either side has fewer than 5 * bins values, or on a non-finite result.

Source code in src/calibrated_agent_evaluation/stats/drift.py
def psi(baseline: Sequence[float], fresh: Sequence[float], *, bins: int = 10) -> float:
    """Population Stability Index between two raw-value samples over baseline-quantile bins.

    ``PSI = sum_i (f_i - b_i) * ln(f_i / b_i)`` over the bin proportions ``b`` (baseline) and
    ``f`` (fresh), each computed from counts plus 0.5. Bins are the baseline's quantiles (deciles
    for ``bins=10``), merged until each holds 5 baseline values. ``0`` for identical samples.

    Raises :class:`StatisticsInputError` on an empty / non-numeric / non-finite input, on
    ``bins < 2``, when either side has fewer than ``5 * bins`` values, or on a non-finite result.
    """
    base = _as_float_array(baseline, name="baseline")
    fut = _as_float_array(fresh, name="fresh")
    _check_bins(bins, base, fut, metric="psi")
    return _finite(_psi_arrays(base, fut, bins), metric="psi")

psi_permutation_p

psi_permutation_p(
    baseline,
    fresh,
    *,
    bins=10,
    n_perm=DEFAULT_N_PERMUTATIONS,
    seed=0,
)

Permutation p-value for psi, recutting the bins on each relabelled baseline.

Refuses the inputs psi refuses.

Source code in src/calibrated_agent_evaluation/stats/drift.py
def psi_permutation_p(
    baseline: Sequence[float],
    fresh: Sequence[float],
    *,
    bins: int = 10,
    n_perm: int = DEFAULT_N_PERMUTATIONS,
    seed: int = 0,
) -> float:
    """Permutation p-value for :func:`psi`, recutting the bins on each relabelled baseline.

    Refuses the inputs :func:`psi` refuses.
    """
    base = _as_float_array(baseline, name="baseline")
    fut = _as_float_array(fresh, name="fresh")
    _check_bins(bins, base, fut, metric="psi")
    return permutation_p(
        lambda b, f: _psi_arrays(b, f, bins), baseline, fresh, n_perm=n_perm, seed=seed
    )

wasserstein

wasserstein(baseline, fresh)

1-D Wasserstein (earth-mover) distance between two raw-value samples, in the data's units.

Wraps scipy.stats.wasserstein_distance (verified surface). 0 for identical samples, growing with the shift in location/shape. Use wasserstein_normalized for anything thresholded. Raises StatisticsInputError on an empty / non-numeric / non-finite input or a non-finite result -- never a raw scipy exception.

Source code in src/calibrated_agent_evaluation/stats/drift.py
def wasserstein(baseline: Sequence[float], fresh: Sequence[float]) -> float:
    """1-D Wasserstein (earth-mover) distance between two raw-value samples, in the data's units.

    Wraps ``scipy.stats.wasserstein_distance`` (verified surface). ``0`` for identical samples,
    growing with the shift in location/shape. Use :func:`wasserstein_normalized` for anything
    thresholded. Raises :class:`StatisticsInputError` on an empty / non-numeric / non-finite input
    or a non-finite result -- never a raw scipy exception.
    """
    base = _as_float_array(baseline, name="baseline")
    fut = _as_float_array(fresh, name="fresh")
    try:
        value = float(wasserstein_distance(base, fut))
    except (ValueError, TypeError) as exc:
        raise StatisticsInputError(
            f"drift: wasserstein failed on the given samples ({exc})"
        ) from exc
    return _finite(value, metric="wasserstein")

wasserstein_normalized

wasserstein_normalized(baseline, fresh)

Wasserstein distance divided by the pooled sample's interquartile range.

Scale-free: multiplying both samples by a constant leaves it unchanged, so one threshold means the same thing for a [0, 1] score and for a token count. Falls back to the pooled standard deviation when the IQR is 0, and returns 0.0 when every value is identical.

Source code in src/calibrated_agent_evaluation/stats/drift.py
def wasserstein_normalized(baseline: Sequence[float], fresh: Sequence[float]) -> float:
    """Wasserstein distance divided by the pooled sample's interquartile range.

    Scale-free: multiplying both samples by a constant leaves it unchanged, so one threshold means
    the same thing for a [0, 1] score and for a token count. Falls back to the pooled standard
    deviation when the IQR is 0, and returns 0.0 when every value is identical.
    """
    base = _as_float_array(baseline, name="baseline")
    fut = _as_float_array(fresh, name="fresh")
    try:
        value = wasserstein_normalized_arrays(base, fut)
    except (ValueError, TypeError) as exc:
        raise StatisticsInputError(
            f"drift: wasserstein_normalized failed on the given samples ({exc})"
        ) from exc
    return _finite(value, metric="wasserstein_normalized")

mcnemar_paired

mcnemar_paired(table, *, exact=None, midp=False)

Run McNemar's paired test over a 2x2 contingency table.

table is [[a, b], [c, d]]; the discordant cells b (before-fail/after-pass) and c (before-pass/after-fail) are what the test counts. exact selects the variant: None (default) auto-selects the exact path when b + c < 25 and the asymptotic path otherwise; True forces exact; False forces asymptotic. midp=True applies the mid-p adjustment to the exact path (ignored on the asymptotic path).

Raises StatisticsInputError on a non-2x2 / negative table, or on a zero-discordant table (b + c == 0) -- McNemar is undefined with no discordant pairs, so this refuses rather than fabricate a p-value (the caller maps it to an explicit undefined result).

Source code in src/calibrated_agent_evaluation/stats/mcnemar.py
def mcnemar_paired(
    table: Sequence[Sequence[int]],
    *,
    exact: bool | None = None,
    midp: bool = False,
) -> McNemarStat:
    """Run McNemar's paired test over a 2x2 contingency ``table``.

    ``table`` is ``[[a, b], [c, d]]``; the discordant cells ``b`` (before-fail/after-pass) and ``c``
    (before-pass/after-fail) are what the test counts. ``exact`` selects the variant: ``None``
    (default) auto-selects the exact path when ``b + c < 25`` and the asymptotic path otherwise;
    ``True`` forces exact; ``False`` forces asymptotic. ``midp=True`` applies the mid-p adjustment
    to the exact path (ignored on the asymptotic path).

    Raises :class:`StatisticsInputError` on a non-2x2 / negative table, or on a **zero-discordant**
    table (``b + c == 0``) -- McNemar is undefined with no discordant pairs, so this refuses rather
    than fabricate a p-value (the caller maps it to an explicit ``undefined`` result).
    """
    a, b, c, d = _validate_table(table)
    n_discordant = b + c
    if n_discordant == 0:
        raise StatisticsInputError(
            "mcnemar_paired: the table has zero discordant pairs (b + c == 0); McNemar's test is "
            "undefined with no discordant pairs -- refusing to fabricate a p-value"
        )

    use_exact = (n_discordant < _SMALL_DISCORDANT) if exact is None else exact

    if use_exact and midp:
        return McNemarStat(
            statistic=float(min(b, c)),
            p_value=_midp(b, c),
            method="midp",
            n_discordant=n_discordant,
        )

    try:
        # statsmodels returns a private ``_Bunch`` with dynamic ``pvalue`` / ``statistic`` attrs
        # (no static type surface); read them via getattr so the vendor's untyped bunch does not
        # leak a pyright attribute error into this pure block.
        # correction=False: the uncorrected asymptotic test (Fagerland et al. 2013); the corrected
        # one made p non-monotone in n across the 25-discordant switch from mid-p.
        result = _sm_mcnemar(np.array([[a, b], [c, d]]), exact=use_exact, correction=False)
        p_value = float(getattr(result, "pvalue"))  # noqa: B009 - dynamic vendor _Bunch attr
        statistic = float(getattr(result, "statistic"))  # noqa: B009 - dynamic vendor _Bunch attr
    except (ValueError, TypeError, ZeroDivisionError, AttributeError) as exc:
        raise StatisticsInputError(
            f"mcnemar_paired: statsmodels failed on table {[[a, b], [c, d]]} ({exc})"
        ) from exc
    if not math.isfinite(p_value):
        raise StatisticsInputError(
            f"mcnemar_paired: produced a non-finite p-value ({p_value}) on table {[[a, b], [c, d]]}"
        )
    return McNemarStat(
        statistic=statistic,
        p_value=p_value,
        method="exact" if use_exact else "asymptotic",
        n_discordant=n_discordant,
    )

newcombe_paired_diff

newcombe_paired_diff(a, b, c, d, *, alpha=0.05)

Newcombe's (1998) hybrid-score interval for a paired difference of proportions (method 10).

For the paired 2x2 table [[a, b], [c, d]] with p1 = (a + b)/n and p2 = (a + c)/n, returns (p1 - p2, lower, upper) at two-sided level 1 - alpha. The limits combine the two marginal Wilson score intervals (l1, u1), (l2, u2) with a correlation correction::

lower = (p1 - p2) - sqrt((p1 - l1)^2 - 2 phi (p1 - l1)(u2 - p2) + (u2 - p2)^2)
upper = (p1 - p2) + sqrt((u1 - p1)^2 - 2 phi (u1 - p1)(p2 - l2) + (p2 - l2)^2)

where phi = max(ad - bc - n/2, 0) / sqrt((a+b)(c+d)(a+c)(b+d)) when ad > bc, else (ad - bc) / sqrt(...), and 0 when the denominator is 0 (Newcombe 1998, Statistics in Medicine 17:2635-2650, p. 2639; as tabulated in the NCSS "Two Correlated Proportions" procedure). Unlike the matched-pairs Wald interval it does not collapse to zero width when a discordant cell is empty, and it stays inside [-1, 1].

Raises StatisticsInputError on a negative cell or an empty table.

Parameters:

Name Type Description Default
a int

Pairs where both observations are successes.

required
b int

Pairs where only the first is a success.

required
c int

Pairs where only the second is a success.

required
d int

Pairs where neither is a success.

required
alpha float

The two-sided level of the interval.

0.05

Returns:

Type Description
tuple[float, float, float]

(difference, lower, upper) for p1 - p2.

Raises:

Type Description
StatisticsInputError

A cell is negative or the table is empty.

Source code in src/calibrated_agent_evaluation/stats/proportions.py
def newcombe_paired_diff(
    a: int, b: int, c: int, d: int, *, alpha: float = 0.05
) -> tuple[float, float, float]:
    """Newcombe's (1998) hybrid-score interval for a paired difference of proportions (method 10).

    For the paired 2x2 table ``[[a, b], [c, d]]`` with ``p1 = (a + b)/n`` and ``p2 = (a + c)/n``,
    returns ``(p1 - p2, lower, upper)`` at two-sided level ``1 - alpha``. The limits combine the
    two marginal Wilson score intervals ``(l1, u1)``, ``(l2, u2)`` with a correlation correction::

        lower = (p1 - p2) - sqrt((p1 - l1)^2 - 2 phi (p1 - l1)(u2 - p2) + (u2 - p2)^2)
        upper = (p1 - p2) + sqrt((u1 - p1)^2 - 2 phi (u1 - p1)(p2 - l2) + (p2 - l2)^2)

    where ``phi = max(ad - bc - n/2, 0) / sqrt((a+b)(c+d)(a+c)(b+d))`` when ``ad > bc``, else
    ``(ad - bc) / sqrt(...)``, and 0 when the denominator is 0 (Newcombe 1998, Statistics in
    Medicine 17:2635-2650, p. 2639; as tabulated in the NCSS "Two Correlated Proportions"
    procedure). Unlike the matched-pairs Wald interval it does not collapse to zero width when a
    discordant cell is empty, and it stays inside ``[-1, 1]``.

    Raises :class:`StatisticsInputError` on a negative cell or an empty table.

    Args:
        a: Pairs where both observations are successes.
        b: Pairs where only the first is a success.
        c: Pairs where only the second is a success.
        d: Pairs where neither is a success.
        alpha: The two-sided level of the interval.

    Returns:
        ``(difference, lower, upper)`` for ``p1 - p2``.

    Raises:
        StatisticsInputError: A cell is negative or the table is empty.
    """
    if min(a, b, c, d) < 0:
        raise StatisticsInputError(f"newcombe_paired_diff: cells must be >= 0, got {(a, b, c, d)}")
    n = a + b + c + d
    if n == 0:
        raise StatisticsInputError("newcombe_paired_diff: the table is empty")
    x1, x2 = a + b, a + c
    p1, p2 = x1 / n, x2 / n
    l1, u1 = wilson_lower(x1, n, alpha=alpha), wilson_upper(x1, n, alpha=alpha)
    l2, u2 = wilson_lower(x2, n, alpha=alpha), wilson_upper(x2, n, alpha=alpha)
    denom = (a + b) * (c + d) * (a + c) * (b + d)
    if denom == 0:
        phi = 0.0
    else:
        num = a * d - b * c
        if num > 0:
            num = max(num - n / 2.0, 0.0)
        phi = num / denom**0.5
    delta = ((p1 - l1) ** 2 - 2.0 * phi * (p1 - l1) * (u2 - p2) + (u2 - p2) ** 2) ** 0.5
    eps = ((u1 - p1) ** 2 - 2.0 * phi * (u1 - p1) * (p2 - l2) + (p2 - l2) ** 2) ** 0.5
    diff = p1 - p2
    return diff, max(-1.0, diff - delta), min(1.0, diff + eps)

wilson_lower

wilson_lower(k, n, alpha=0.1)

Return the one-sided 95%-by-default Wilson score lower bound for k of n successes.

alpha is the two-sided level passed to proportion_confint; the returned value is that interval's lower edge, which equals the one-sided lower bound at confidence 1 - alpha/2 (so the default alpha=0.10 gives the 95% one-sided lower bound the gate uses).

Raises StatisticsInputError on n <= 0 or k outside [0, n] -- a proportion from a non-count is a caller bug, surfaced as a typed error rather than a NaN from statsmodels. k and n may be fractional effective counts (k / deff of n / deff) for clustered items.

Parameters:

Name Type Description Default
k float

The number of successes.

required
n float

The number of trials.

required
alpha float

The two-sided level; 0.10 gives the one-sided 95% bound.

0.1

Returns:

Type Description
float

The lower bound, in [0, 1].

Raises:

Type Description
StatisticsInputError

n <= 0 or k is outside [0, n].

Source code in src/calibrated_agent_evaluation/stats/proportions.py
def wilson_lower(k: float, n: float, alpha: float = 0.10) -> float:
    """Return the one-sided 95%-by-default Wilson score lower bound for ``k`` of ``n`` successes.

    ``alpha`` is the two-sided level passed to ``proportion_confint``; the returned value is that
    interval's lower edge, which equals the one-sided lower bound at confidence ``1 - alpha/2``
    (so the default ``alpha=0.10`` gives the 95% one-sided lower bound the gate uses).

    Raises :class:`StatisticsInputError` on ``n <= 0`` or ``k`` outside ``[0, n]`` -- a proportion
    from a non-count is a caller bug, surfaced as a typed error rather than a NaN from statsmodels.
    ``k`` and ``n`` may be fractional effective counts (``k / deff`` of ``n / deff``) for clustered
    items.

    Args:
        k: The number of successes.
        n: The number of trials.
        alpha: The two-sided level; 0.10 gives the one-sided 95% bound.

    Returns:
        The lower bound, in [0, 1].

    Raises:
        StatisticsInputError: ``n <= 0`` or ``k`` is outside ``[0, n]``.
    """
    if n <= 0:
        raise StatisticsInputError(f"wilson_lower: n must be positive, got {n}")
    if not 0 <= k <= n:
        raise StatisticsInputError(f"wilson_lower: k={k} must lie in [0, n={n}]")
    if not 0.0 < alpha < 1.0:
        raise StatisticsInputError(f"wilson_lower: alpha must be in (0, 1), got {alpha}")
    low = proportion_confint(k, n, alpha=alpha, method="wilson")[0]
    # Clamp the theoretical [0,1] range: statsmodels can return a tiny negative near k=0.
    return max(0.0, float(low))

wilson_upper

wilson_upper(k, n, alpha=0.1)

Return the one-sided 95%-by-default Wilson score upper bound for k of n successes.

The mirror of wilson_lower, reading the same Wilson score interval's upper edge at the same alpha convention.

Which edge is conservative depends on which direction is bad, so the two are not interchangeable. A rate that must clear a floor (TPR, TNR: large is good) is gated on the lower bound. A rate that must stay under a ceiling (bias magnitude: small is good) is gated on the upper bound; gating a ceiling on the lower bound reads as "promote unless we are confident the judge IS biased", which inverts the burden of proof a safety gate exists to hold.

Raises StatisticsInputError on n <= 0 or k outside [0, n].

Source code in src/calibrated_agent_evaluation/stats/proportions.py
def wilson_upper(k: int, n: int, alpha: float = 0.10) -> float:
    """Return the one-sided 95%-by-default Wilson score upper bound for ``k`` of ``n`` successes.

    The mirror of :func:`wilson_lower`, reading the same Wilson score interval's upper edge at the
    same ``alpha`` convention.

    Which edge is conservative depends on which direction is *bad*, so the two are not
    interchangeable. A rate that must clear a **floor** (TPR, TNR: large is good) is gated on the
    lower bound. A rate that must stay under a **ceiling** (bias magnitude: small is good) is gated
    on the upper bound; gating a ceiling on the lower bound reads as "promote unless we are
    confident the judge IS biased", which inverts the burden of proof a safety gate exists to hold.

    Raises :class:`StatisticsInputError` on ``n <= 0`` or ``k`` outside ``[0, n]``.
    """
    if n <= 0:
        raise StatisticsInputError(f"wilson_upper: n must be positive, got {n}")
    if not 0 <= k <= n:
        raise StatisticsInputError(f"wilson_upper: k={k} must lie in [0, n={n}]")
    if not 0.0 < alpha < 1.0:
        raise StatisticsInputError(f"wilson_upper: alpha must be in (0, 1), got {alpha}")
    high = proportion_confint(k, n, alpha=alpha, method="wilson")[1]
    # Clamp the theoretical [0,1] range: statsmodels can overshoot 1.0 near k=n.
    return min(1.0, float(high))

rg_interval

rg_interval(
    cal_pos,
    cal_neg,
    eval_preds,
    *,
    conf=0.95,
    n_resamples=9999,
    seed=0,
    eval_cluster_ids=None,
    cal_pos_design_effect=1.0,
    cal_neg_design_effect=1.0,
)

Two-sided interval for the Rogan-Gladen corrected rate from three independent samples.

TPR, 1 - TNR and measured each get an independent Jeffreys posterior Beta(x + 1/2, n - x + 1/2). n_resamples Monte Carlo draws of the three are pushed through the Rogan-Gladen inversion, clipped into [0, 1], and the interval is the central conf percentile range of the result. seed fixes the draws.

eval_cluster_ids -- when several eval verdicts come from one trace (a span- or process-level judge), the trace id of each verdict. The verdicts are then not independent, and the measured rate's posterior is taken at the effective sample size n / deff, with deff from cluster_design_effect (the effective-sample-size approach of Korn and Graubard 1998), so the interval widens by the clustering the sample shows. The design effect used is returned. cal_pos_design_effect and cal_neg_design_effect do the same for calibration gold whose items cluster by trace (the record's tpr_design_effect / tnr_design_effect); at 1 the calibration items are independent.

When the interval collapses to the single point 0 (or 1), almost every draw has the measured rate beyond the judge's false-positive floor (or true-positive ceiling). ErrorRatesDoNotTransferError is raised then, since the sample contradicts the error rates measured on the calibration gold.

A draw whose denominator TPR - (1 - TNR) is not positive has no corrected rate. Such draws are left out and counted. Because a left-out draw could have belonged in either tail, each tail's percentile moves outward by the left-out share f: the bounds are the identifiable draws' quantiles at (a - f) / (1 - f) and (1 - a) / (1 - f), with a = (1 - conf) / 2. When f > a a tail is not determined and NonIdentifiableRateError is raised instead of an interval.

A boundary count (no false positive, every positive caught, or a measured count of 0 or of every verdict) takes the limit it governs with that rate at its boundary, the modified Jeffreys rule described in the module docstring.

Parameters:

Name Type Description Default
cal_pos Sequence[int] | NDArray[integer]

The judge's verdicts (1 = FAIL) on the hunted true-FAIL class; their mean is TPR.

required
cal_neg Sequence[int] | NDArray[integer]

The judge's verdicts (1 = FAIL) on the hunted true-PASS class; 1 - mean is TNR.

required
eval_preds Sequence[int] | NDArray[integer]

The judge's verdicts on the separate rate sample; their mean is the measured rate.

required
conf float

The interval's two-sided coverage target.

0.95
n_resamples int

The number of Monte Carlo draws.

9999
seed int

Fixes the draws.

0
eval_cluster_ids Sequence[str] | None

The trace id of each eval verdict, when one trace contributes several.

None
cal_pos_design_effect float

The design effect of the clustered true-FAIL calibration class.

1.0
cal_neg_design_effect float

The design effect of the clustered true-PASS calibration class.

1.0

Returns:

Type Description
RGInterval

The interval, the share of draws with no corrected rate, and the design effect used.

Raises:

Type Description
StatisticsInputError

A sample is empty or holds a value other than 0 or 1, or conf or n_resamples is out of range.

NonIdentifiableRateError

Too many draws have no corrected rate for a tail to be determined.

ErrorRatesDoNotTransferError

The interval collapses to 0 or to 1.

Source code in src/calibrated_agent_evaluation/stats/rogan_gladen.py
def rg_interval(
    cal_pos: Sequence[int] | NDArray[np.integer],
    cal_neg: Sequence[int] | NDArray[np.integer],
    eval_preds: Sequence[int] | NDArray[np.integer],
    *,
    conf: float = 0.95,
    n_resamples: int = 9999,
    seed: int = 0,
    eval_cluster_ids: Sequence[str] | None = None,
    cal_pos_design_effect: float = 1.0,
    cal_neg_design_effect: float = 1.0,
) -> RGInterval:
    """Two-sided interval for the Rogan-Gladen corrected rate from three independent samples.

    TPR, 1 - TNR and ``measured`` each get an independent Jeffreys posterior
    ``Beta(x + 1/2, n - x + 1/2)``. ``n_resamples`` Monte Carlo draws of the three are pushed
    through the Rogan-Gladen inversion, clipped into [0, 1], and the interval is the central
    ``conf`` percentile range of the result. ``seed`` fixes the draws.

    ``eval_cluster_ids`` -- when several eval verdicts come from one trace (a span- or
    process-level judge), the trace id of each verdict. The verdicts are then not independent, and
    the measured rate's posterior is taken at the effective sample size ``n / deff``, with ``deff``
    from :func:`cluster_design_effect` (the effective-sample-size approach of Korn and Graubard
    1998), so the interval widens by the clustering the sample shows.
    The design effect used is returned. ``cal_pos_design_effect`` and ``cal_neg_design_effect`` do
    the same for calibration gold whose items cluster by trace (the record's
    ``tpr_design_effect`` / ``tnr_design_effect``); at 1 the calibration items are independent.

    When the interval collapses to the single point 0 (or 1), almost every draw has the measured
    rate beyond the judge's false-positive floor (or true-positive ceiling).
    :class:`~calibrated_agent_evaluation.calibrate.errors.ErrorRatesDoNotTransferError` is raised
    then, since the sample contradicts the error rates measured on the calibration gold.

    A draw whose denominator ``TPR - (1 - TNR)`` is not positive has no corrected rate. Such draws
    are left out and counted. Because a left-out draw could have
    belonged in either tail, each tail's percentile moves outward by the left-out share ``f``:
    the bounds are the identifiable draws' quantiles at ``(a - f) / (1 - f)`` and
    ``(1 - a) / (1 - f)``, with ``a = (1 - conf) / 2``. When ``f > a`` a tail is not determined
    and :class:`NonIdentifiableRateError` is raised instead of an interval.

    A boundary count (no false positive, every positive caught, or a measured count of 0 or of
    every verdict) takes the limit it governs with that rate at its boundary, the modified
    Jeffreys rule described in the module docstring.

    Args:
        cal_pos: The judge's verdicts (1 = FAIL) on the hunted true-FAIL class; their mean is TPR.
        cal_neg: The judge's verdicts (1 = FAIL) on the hunted true-PASS class; ``1 - mean`` is TNR.
        eval_preds: The judge's verdicts on the separate rate sample; their mean is the measured
            rate.
        conf: The interval's two-sided coverage target.
        n_resamples: The number of Monte Carlo draws.
        seed: Fixes the draws.
        eval_cluster_ids: The trace id of each eval verdict, when one trace contributes several.
        cal_pos_design_effect: The design effect of the clustered true-FAIL calibration class.
        cal_neg_design_effect: The design effect of the clustered true-PASS calibration class.

    Returns:
        The interval, the share of draws with no corrected rate, and the design effect used.

    Raises:
        StatisticsInputError: A sample is empty or holds a value other than 0 or 1, or ``conf``
            or ``n_resamples`` is out of range.
        NonIdentifiableRateError: Too many draws have no corrected rate for a tail to be determined.
        ErrorRatesDoNotTransferError: The interval collapses to 0 or to 1.
    """
    if len(cal_pos) == 0 or len(cal_neg) == 0 or len(eval_preds) == 0:
        raise StatisticsInputError(
            "rg_interval: cal_pos, cal_neg, and eval_preds must all be non-empty"
        )
    if not 0.0 < conf < 1.0:
        raise StatisticsInputError(f"rg_interval: conf must be in (0, 1), got {conf}")
    if n_resamples < 1:
        raise StatisticsInputError(f"rg_interval: n_resamples must be >= 1, got {n_resamples}")
    for name, value in (
        ("cal_pos_design_effect", cal_pos_design_effect),
        ("cal_neg_design_effect", cal_neg_design_effect),
    ):
        if not value >= 1.0:
            raise StatisticsInputError(f"rg_interval: {name} must be >= 1, got {value}")
    tp, n1 = _successes(cal_pos, "cal_pos")
    fp, n0 = _successes(cal_neg, "cal_neg")
    k, n_eval = _successes(eval_preds, "eval_preds")
    deff = 1.0 if eval_cluster_ids is None else cluster_design_effect(eval_preds, eval_cluster_ids)

    rng = np.random.default_rng(seed)
    dp, dn = cal_pos_design_effect, cal_neg_design_effect
    tpr = rng.beta(tp / dp + 0.5, (n1 - tp) / dp + 0.5, n_resamples)
    fpr = rng.beta(fp / dn + 0.5, (n0 - fp) / dn + 0.5, n_resamples)
    measured = rng.beta(k / deff + 0.5, (n_eval - k) / deff + 0.5, n_resamples)
    tail = (1.0 - conf) / 2.0
    counts = f"TPR {tp}/{n1}, 1 - TNR {fp}/{n0}"
    lower, bad_lo = _tail_quantile(
        np.ones(n_resamples) if tp == n1 else tpr,
        fpr,
        np.zeros(n_resamples) if k == 0 else measured,
        tail,
        upper=False,
        counts=counts,
    )
    upper, bad_hi = _tail_quantile(
        tpr,
        np.zeros(n_resamples) if fp == 0 else fpr,
        np.ones(n_resamples) if k == n_eval else measured,
        tail,
        upper=True,
        counts=counts,
    )
    if upper <= 0.0 or lower >= 1.0:
        edge, side = (
            ("0", "below the judge's false-positive floor")
            if upper <= 0.0
            else (
                "1",
                "above the judge's true-positive ceiling",
            )
        )
        raise ErrorRatesDoNotTransferError(
            f"{TRANSFER_REFUSAL}: the measured rate {k}/{n_eval} sits "
            f"{side} ({counts}) in almost every draw, so the corrected rate collapses to exactly "
            f"{edge}. A zero-width interval would claim the true rate is exactly {edge}; what the "
            "data show is that the judge's error rates on the calibration gold do not hold here. "
            "Check that the rate sample comes from the population the judge was calibrated on, "
            "or calibrate the judge on gold drawn from this population"
        )
    return RGInterval(
        lower,
        upper,
        n_draws=n_resamples,
        n_nonidentifiable=max(bad_lo, bad_hi),
        eval_design_effect=deff,
    )

youden_j

youden_j(tpr, tnr)

Return Youden's J = TPR + TNR - 1.

J ranges from 0 (a coin-flip judge, no information) to 1 (perfect). The gate refuses the Rogan-Gladen point correction when J < 0.5 -- below that the denominator TPR - (1 - TNR) = J shrinks toward 0 and the correction divides by ~0 (the widening bound is the honest signal).

Source code in src/calibrated_agent_evaluation/stats/rogan_gladen.py
def youden_j(tpr: float, tnr: float) -> float:
    """Return Youden's J = TPR + TNR - 1.

    J ranges from 0 (a coin-flip judge, no information) to 1 (perfect). The gate refuses the
    Rogan-Gladen point correction when J < 0.5 -- below that the denominator ``TPR - (1 - TNR) = J``
    shrinks toward 0 and the correction divides by ~0 (the widening bound is the honest signal).
    """
    _check_prob(tpr, "tpr")
    _check_prob(tnr, "tnr")
    return tpr + tnr - 1.0