calibrated_agent_evaluation.stats¶
calibrated_agent_evaluation.stats ¶
Statistics used by the calibration gate and the later stages.
Wilson lower bounds and Holm/BY multiplicity come from statsmodels. The Rogan-Gladen interval (Jeffreys posteriors), Youden's J and Cohen's kappa are implemented here, kappa to avoid a large scikit-learn dependency. Saturation (Good-Turing, Kish), McNemar and drift live in their own modules in this package. Pure functions, no vendor trace type, no I/O.
McNemarStat
dataclass
¶
The pure McNemar computation (the improve-stage McNemarResult wraps it).
statistic is min(b, c) for the exact/mid-p path (the binomial statistic statsmodels
reports) or the uncorrected chi-square (b - c)^2 / (b + c) for the asymptotic path.
n_discordant is b + c.
cohen_kappa ¶
Return Cohen's κ between two equal-length label sequences.
κ = (p_o - p_e) / (1 - p_e), where p_o is observed agreement and p_e is the agreement
expected by chance from the marginals. Returns 1.0 for perfect agreement; 0.0 when observed
equals chance. When p_e == 1 (both raters gave a single constant label) κ is undefined --
returns 1.0 if they fully agree, else 0.0 (the standard degenerate-case convention).
Raises StatisticsInputError on empty or unequal-length inputs.
Source code in src/calibrated_agent_evaluation/stats/agreement.py
minority_fraction ¶
Return the fraction of the least common label value, or 0.0 for fewer than two values.
Source code in src/calibrated_agent_evaluation/stats/agreement.py
should_hide_raw_agreement ¶
Return whether raw percent agreement should be hidden for this label set.
True when the minority class is below IMBALANCE_THRESHOLD -- under imbalance raw
agreement flatters a do-nothing judge, so the report surfaces κ instead. A set with a single
label value is the extreme case (the other class has 0% of the labels) and is hidden: a judge
that answers FAIL to everything scores 100% raw agreement on an all-FAIL set. An empty set has
nothing to hide and returns False; the callers refuse an empty pairing themselves.
Source code in src/calibrated_agent_evaluation/stats/agreement.py
jensen_shannon ¶
Jensen-Shannon divergence (0..1, base 2) of two frequency/probability vectors.
scipy.spatial.distance.jensenshannon returns the JS distance (the square root of the
divergence); this squares it back so the drift report reads a divergence bounded in [0, 1].
p and q need not be normalized -- jensenshannon normalizes each to a probability
vector -- but they must be the same length and not both all-zero (which yields NaN).
Raises StatisticsInputError on an empty / mismatched / non-numeric input or a
non-finite result (an all-zero vector) -- never a raw exception, never a NaN.
Source code in src/calibrated_agent_evaluation/stats/drift.py
permutation_p ¶
One-sided permutation p-value that statistic(baseline, fresh) is no larger than chance.
The pooled sample is relabelled n_perm times with numpy.random.default_rng(seed)
(group sizes kept), and p = (1 + #{null >= observed}) / (1 + n_perm), which is never 0 and
is a valid p-value at any n. statistic receives float arrays and returns a float.
Raises StatisticsInputError on empty / non-numeric input or n_perm < 1.
Source code in src/calibrated_agent_evaluation/stats/drift.py
psi ¶
Population Stability Index between two raw-value samples over baseline-quantile bins.
PSI = sum_i (f_i - b_i) * ln(f_i / b_i) over the bin proportions b (baseline) and
f (fresh), each computed from counts plus 0.5. Bins are the baseline's quantiles (deciles
for bins=10), merged until each holds 5 baseline values. 0 for identical samples.
Raises StatisticsInputError on an empty / non-numeric / non-finite input, on
bins < 2, when either side has fewer than 5 * bins values, or on a non-finite result.
Source code in src/calibrated_agent_evaluation/stats/drift.py
psi_permutation_p ¶
Permutation p-value for psi, recutting the bins on each relabelled baseline.
Refuses the inputs psi refuses.
Source code in src/calibrated_agent_evaluation/stats/drift.py
wasserstein ¶
1-D Wasserstein (earth-mover) distance between two raw-value samples, in the data's units.
Wraps scipy.stats.wasserstein_distance (verified surface). 0 for identical samples,
growing with the shift in location/shape. Use wasserstein_normalized for anything
thresholded. Raises StatisticsInputError on an empty / non-numeric / non-finite input
or a non-finite result -- never a raw scipy exception.
Source code in src/calibrated_agent_evaluation/stats/drift.py
wasserstein_normalized ¶
Wasserstein distance divided by the pooled sample's interquartile range.
Scale-free: multiplying both samples by a constant leaves it unchanged, so one threshold means the same thing for a [0, 1] score and for a token count. Falls back to the pooled standard deviation when the IQR is 0, and returns 0.0 when every value is identical.
Source code in src/calibrated_agent_evaluation/stats/drift.py
mcnemar_paired ¶
Run McNemar's paired test over a 2x2 contingency table.
table is [[a, b], [c, d]]; the discordant cells b (before-fail/after-pass) and c
(before-pass/after-fail) are what the test counts. exact selects the variant: None
(default) auto-selects the exact path when b + c < 25 and the asymptotic path otherwise;
True forces exact; False forces asymptotic. midp=True applies the mid-p adjustment
to the exact path (ignored on the asymptotic path).
Raises StatisticsInputError on a non-2x2 / negative table, or on a zero-discordant
table (b + c == 0) -- McNemar is undefined with no discordant pairs, so this refuses rather
than fabricate a p-value (the caller maps it to an explicit undefined result).
Source code in src/calibrated_agent_evaluation/stats/mcnemar.py
newcombe_paired_diff ¶
Newcombe's (1998) hybrid-score interval for a paired difference of proportions (method 10).
For the paired 2x2 table [[a, b], [c, d]] with p1 = (a + b)/n and p2 = (a + c)/n,
returns (p1 - p2, lower, upper) at two-sided level 1 - alpha. The limits combine the
two marginal Wilson score intervals (l1, u1), (l2, u2) with a correlation correction::
lower = (p1 - p2) - sqrt((p1 - l1)^2 - 2 phi (p1 - l1)(u2 - p2) + (u2 - p2)^2)
upper = (p1 - p2) + sqrt((u1 - p1)^2 - 2 phi (u1 - p1)(p2 - l2) + (p2 - l2)^2)
where phi = max(ad - bc - n/2, 0) / sqrt((a+b)(c+d)(a+c)(b+d)) when ad > bc, else
(ad - bc) / sqrt(...), and 0 when the denominator is 0 (Newcombe 1998, Statistics in
Medicine 17:2635-2650, p. 2639; as tabulated in the NCSS "Two Correlated Proportions"
procedure). Unlike the matched-pairs Wald interval it does not collapse to zero width when a
discordant cell is empty, and it stays inside [-1, 1].
Raises StatisticsInputError on a negative cell or an empty table.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
a
|
int
|
Pairs where both observations are successes. |
required |
b
|
int
|
Pairs where only the first is a success. |
required |
c
|
int
|
Pairs where only the second is a success. |
required |
d
|
int
|
Pairs where neither is a success. |
required |
alpha
|
float
|
The two-sided level of the interval. |
0.05
|
Returns:
| Type | Description |
|---|---|
tuple[float, float, float]
|
|
Raises:
| Type | Description |
|---|---|
StatisticsInputError
|
A cell is negative or the table is empty. |
Source code in src/calibrated_agent_evaluation/stats/proportions.py
wilson_lower ¶
Return the one-sided 95%-by-default Wilson score lower bound for k of n successes.
alpha is the two-sided level passed to proportion_confint; the returned value is that
interval's lower edge, which equals the one-sided lower bound at confidence 1 - alpha/2
(so the default alpha=0.10 gives the 95% one-sided lower bound the gate uses).
Raises StatisticsInputError on n <= 0 or k outside [0, n] -- a proportion
from a non-count is a caller bug, surfaced as a typed error rather than a NaN from statsmodels.
k and n may be fractional effective counts (k / deff of n / deff) for clustered
items.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
k
|
float
|
The number of successes. |
required |
n
|
float
|
The number of trials. |
required |
alpha
|
float
|
The two-sided level; 0.10 gives the one-sided 95% bound. |
0.1
|
Returns:
| Type | Description |
|---|---|
float
|
The lower bound, in [0, 1]. |
Raises:
| Type | Description |
|---|---|
StatisticsInputError
|
|
Source code in src/calibrated_agent_evaluation/stats/proportions.py
wilson_upper ¶
Return the one-sided 95%-by-default Wilson score upper bound for k of n successes.
The mirror of wilson_lower, reading the same Wilson score interval's upper edge at the
same alpha convention.
Which edge is conservative depends on which direction is bad, so the two are not interchangeable. A rate that must clear a floor (TPR, TNR: large is good) is gated on the lower bound. A rate that must stay under a ceiling (bias magnitude: small is good) is gated on the upper bound; gating a ceiling on the lower bound reads as "promote unless we are confident the judge IS biased", which inverts the burden of proof a safety gate exists to hold.
Raises StatisticsInputError on n <= 0 or k outside [0, n].
Source code in src/calibrated_agent_evaluation/stats/proportions.py
rg_interval ¶
rg_interval(
cal_pos,
cal_neg,
eval_preds,
*,
conf=0.95,
n_resamples=9999,
seed=0,
eval_cluster_ids=None,
cal_pos_design_effect=1.0,
cal_neg_design_effect=1.0,
)
Two-sided interval for the Rogan-Gladen corrected rate from three independent samples.
TPR, 1 - TNR and measured each get an independent Jeffreys posterior
Beta(x + 1/2, n - x + 1/2). n_resamples Monte Carlo draws of the three are pushed
through the Rogan-Gladen inversion, clipped into [0, 1], and the interval is the central
conf percentile range of the result. seed fixes the draws.
eval_cluster_ids -- when several eval verdicts come from one trace (a span- or
process-level judge), the trace id of each verdict. The verdicts are then not independent, and
the measured rate's posterior is taken at the effective sample size n / deff, with deff
from cluster_design_effect (the effective-sample-size approach of Korn and Graubard
1998), so the interval widens by the clustering the sample shows.
The design effect used is returned. cal_pos_design_effect and cal_neg_design_effect do
the same for calibration gold whose items cluster by trace (the record's
tpr_design_effect / tnr_design_effect); at 1 the calibration items are independent.
When the interval collapses to the single point 0 (or 1), almost every draw has the measured
rate beyond the judge's false-positive floor (or true-positive ceiling).
ErrorRatesDoNotTransferError is raised
then, since the sample contradicts the error rates measured on the calibration gold.
A draw whose denominator TPR - (1 - TNR) is not positive has no corrected rate. Such draws
are left out and counted. Because a left-out draw could have
belonged in either tail, each tail's percentile moves outward by the left-out share f:
the bounds are the identifiable draws' quantiles at (a - f) / (1 - f) and
(1 - a) / (1 - f), with a = (1 - conf) / 2. When f > a a tail is not determined
and NonIdentifiableRateError is raised instead of an interval.
A boundary count (no false positive, every positive caught, or a measured count of 0 or of every verdict) takes the limit it governs with that rate at its boundary, the modified Jeffreys rule described in the module docstring.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
cal_pos
|
Sequence[int] | NDArray[integer]
|
The judge's verdicts (1 = FAIL) on the hunted true-FAIL class; their mean is TPR. |
required |
cal_neg
|
Sequence[int] | NDArray[integer]
|
The judge's verdicts (1 = FAIL) on the hunted true-PASS class; |
required |
eval_preds
|
Sequence[int] | NDArray[integer]
|
The judge's verdicts on the separate rate sample; their mean is the measured rate. |
required |
conf
|
float
|
The interval's two-sided coverage target. |
0.95
|
n_resamples
|
int
|
The number of Monte Carlo draws. |
9999
|
seed
|
int
|
Fixes the draws. |
0
|
eval_cluster_ids
|
Sequence[str] | None
|
The trace id of each eval verdict, when one trace contributes several. |
None
|
cal_pos_design_effect
|
float
|
The design effect of the clustered true-FAIL calibration class. |
1.0
|
cal_neg_design_effect
|
float
|
The design effect of the clustered true-PASS calibration class. |
1.0
|
Returns:
| Type | Description |
|---|---|
RGInterval
|
The interval, the share of draws with no corrected rate, and the design effect used. |
Raises:
| Type | Description |
|---|---|
StatisticsInputError
|
A sample is empty or holds a value other than 0 or 1, or |
NonIdentifiableRateError
|
Too many draws have no corrected rate for a tail to be determined. |
ErrorRatesDoNotTransferError
|
The interval collapses to 0 or to 1. |
Source code in src/calibrated_agent_evaluation/stats/rogan_gladen.py
169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 | |
youden_j ¶
Return Youden's J = TPR + TNR - 1.
J ranges from 0 (a coin-flip judge, no information) to 1 (perfect). The gate refuses the
Rogan-Gladen point correction when J < 0.5 -- below that the denominator TPR - (1 - TNR) = J
shrinks toward 0 and the correction divides by ~0 (the widening bound is the honest signal).