calibrated_agent_evaluation.calibrate¶
calibrated_agent_evaluation.calibrate ¶
Check a judge against human labels, and correct the failure rate it reports.
Works on pairs of a human gold label and the judge's verdict on the same trace, and answers three questions: is this judge good enough to trust (both its true-positive and true-negative rates must clear the gate at the low end of their 95% range), what is the real failure rate (the Rogan-Gladen correction, with a range that carries the uncertainty of both the gold set and the scored sample), and when must the judge be checked again (version and fingerprint pinning, and single-use test sets). Running a judge is the judge runtime's job, not this package's.
Reads normalized Label objects only; the numeric work lives in
stats. The engine and the planner load lazily because they need
the stats extra, while the record types (Interval, Bound) and the errors need nothing
beyond the base install.
CalibrationError ¶
Bases: CalibratedAgentEvaluationDataError
Base class for every calibration-engine refusal.
CalibrationStoreError ¶
Bases: CalibrationError
A calibration-record file read or write failed.
Wraps the underlying OSError so a filesystem failure surfaces inside the typed family
rather than as a raw OS exception escaping the store's public surface.
CalibrationVersionError ¶
Bases: CalibrationError
A stored calibration record's schema_version != the running CALIB_SCHEMA_VERSION.
Refuse-loudly rather than coercing the data (the no-silent-coercion precedent).
DuplicateGoldError ¶
Bases: UnpairedObservationError
One (trace_id, span_id) target carries more than one gold label.
Two raters' labels on one target are one piece of evidence, not two: counting both doubles the
sample size behind the bounds and can certify a judge the single-label set refuses.
duplicates names the repeated targets.
Source code in src/calibrated_agent_evaluation/calibrate/errors.py
ErrorRatesDoNotTransferError ¶
Bases: NonIdentifiableRateError
The rate sample contradicts the judge's calibrated error rates.
Almost every draw puts the measured rate below the judge's false-positive floor (or above its true-positive ceiling), so the corrected rate clips to a single point. A zero-width interval there would claim the true rate is exactly 0 (or 1); what the data show is that the error rates measured on the calibration gold do not hold on this sample.
FewShotEvaluationError ¶
Bases: CalibrationError
Certification/TPR/TNR metrics were requested over the few-shot pool.
The few-shot pool is never an evaluation set; evaluating on it inflates the numbers.
InfeasiblePlanError ¶
Bases: CalibrationError
The calibration planner was asked for a target no finite sample size can reach.
tau >= p_hat: the expected judge accuracy never clears the threshold, so the lower bound
cannot either -- refused rather than searching unboundedly.
LockedTestReuseError ¶
Bases: CalibrationError
A locked test set that was already consumed once was re-evaluated (single-use).
NonIdentifiableRateError ¶
Bases: CalibrationError
The measured rate lies outside the Rogan-Gladen identifiable region.
measured < 1 - TNR or the denominator TPR - (1 - TNR)) <= 0; the corrected point is
refused and the bound is reported rather than silently clamping to 0/1.
RateSampleTooSmallError ¶
Bases: CalibrationError
A corrected rate was requested from a rate sample smaller than the measured minimum.
The interval was measured to under-cover with too few evaluation traces at low prevalence, so
a rate sample below MIN_RATE_SAMPLE is
refused. n is the sample's size (distinct traces when verdicts are clustered) and
minimum the size needed.
Source code in src/calibrated_agent_evaluation/calibrate/errors.py
RateUnsafeError ¶
Bases: CalibrationError
A corrected rate was requested from a record certified on gold below the coverage floor.
The certification stands (the Wilson bound is valid at any n), but below the coverage floor the corrected rate's interval was measured to cover the true rate less often than 95%, so an interval built from smaller gold is refused. At or above the floor coverage is close to 95% but not everywhere: a class with about 2 to 3 errors still under-covers slightly, a known limit of the current interval.
SampleMisuseError ¶
Bases: CalibrationError
The prevalence-corrected rate was requested from the hunted fail-heavy set.
The true rate is measured on a separate production-representative sample -- two samples, two jobs. Reading it off the hunted set would report a wildly inflated rate. Raised too when the rate sample shares traces with the calibration gold.
StatisticsInputError ¶
Bases: CalibrationError
A pure statistics primitive received an out-of-domain numeric input.
n <= 0, k outside [0, n], or a probability outside [0, 1] -- guarded so a bad
count surfaces as a typed error rather than a NaN leaking out of scipy/statsmodels.
UncalibratedJudgeError ¶
Bases: CalibrationError
A corrected rate or promotion was requested for a judge with no CERTIFIED calibration.
UnpairedObservationError ¶
Bases: CalibrationError
An observation could not be paired to exactly one human gold + one judge verdict.
Raised rather than silently dropping the observation (which would bias TPR/TNR).
UnratifiedGoldError ¶
Bases: CalibrationError
A calibration gold set contains a label not produced/ratified by a human.
Certifying a judge against another judge turns every confidence bound into theater -- refused.
WeakJudgeError ¶
Bases: CalibrationError
Youden's J < 0.5 -- the Rogan-Gladen point correction is refused.
The widening interval is still returned as the honest signal; only the point rate accessor raises, so a near-random judge cannot be "rescued" by a correction that divides by ~0.
InMemoryLockedTestLedger ¶
A LockedTestLedger held in memory, optionally seeded with consumed sets.
Refuses a set whose id, content digest, or any trace was already consumed, except a strict
superset of sets opened with allow_top_up for the same judge fingerprint. Not safe
for concurrent consume calls on one instance.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
consumed
|
Iterable[str]
|
Locked-set ids already used, for example from stored calibration records. |
()
|
consumed_trace_ids
|
Iterable[str]
|
Trace ids already used in a locked test, for example the
|
()
|
Source code in src/calibrated_agent_evaluation/calibrate/ledger.py
from_records
classmethod
¶
A ledger holding every set records consumed, with refused-short sets left open.
Pass the records for one failure mode. A refused set already topped up by a later record (one holding all its traces) is closed, so it cannot be grown twice.
Source code in src/calibrated_agent_evaluation/calibrate/ledger.py
is_consumed ¶
allow_top_up ¶
Open the consumed set set_id to a top-up by the judge with judge_fingerprint.
Raises:
| Type | Description |
|---|---|
LockedTestReuseError
|
|
Source code in src/calibrated_agent_evaluation/calibrate/ledger.py
consume ¶
Mark locked consumed, refusing a reused id, reused content or a shared trace.
A set holding every trace of open sets for the same judge_fingerprint, plus at least
one fresh trace, is their top-up: it is accepted and the looks are counted.
Returns:
| Type | Description |
|---|---|
int
|
How many looks the judge has had at this gold, 1 for a fresh set. |
Raises:
| Type | Description |
|---|---|
LockedTestReuseError
|
The set, its content or one of its traces was already used. |
Source code in src/calibrated_agent_evaluation/calibrate/ledger.py
LockedTestLedger ¶
Bases: Protocol
Records which locked test sets have been evaluated.
consume ¶
Mark locked consumed and return how many looks the judge has had at its gold.
1 for a fresh set; one more for each refused set it tops up. Raises
LockedTestReuseError if the set, its content or a trace was already used.
Source code in src/calibrated_agent_evaluation/calibrate/ledger.py
allow_top_up ¶
Let a superset of the consumed set set_id be consumed by the same judge.
ConfusionCounts
dataclass
¶
2x2 confusion counts with FAIL as the positive class.
tp -- gold FAIL, verdict FAIL (caught failure). fn -- gold FAIL, verdict PASS
(missed).
tn -- gold PASS, verdict PASS (correct clear). fp -- gold PASS, verdict FAIL
(false alarm).
Bound
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
A gated rate with its 95% one-sided lower confidence bound.
The gate reads lb95, never point -- the point estimate at the threshold does
not certify. Carrying both in one object is how the "no bare number" rule is made structural.
lb95 is a one-sided 95% Wilson lower bound (sided="lower"), not the lower edge of a
two-sided 95% interval; n is the class size it was computed over. For gold whose items
cluster (several spans of one trace) the bound is taken at the effective size and method
says so.
Config:
frozen:Trueextra:forbid
Fields:
Validators:
CalibrationRecord
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
The reproducible gate evidence.
Pins the judge's exact (model, prompt_version) -- a change to either invalidates the record.
Carries the gated bounds, Youden's J, the corrected-rate interval, and the CERTIFIED/REFUSED
result. Certification is an intersection-union test per judge (both lower bounds must clear),
and certifying several judges is not a joint claim, so the record carries no multiplicity
family. References its sample members by trace_id even though the
record lives outside the trace store.
Config:
frozen:Trueextra:forbid
Fields:
-
calib_id(str) -
judge_id(str) -
model(str) -
prompt_version(str) -
hunted_set_id(str) -
n_pos(int) -
n_neg(int) -
tpr(Bound) -
tnr(Bound) -
tpr_design_effect(float) -
tnr_design_effect(float) -
youden_j(float) -
result(CalibResult) -
rate_set_id(str | None) -
corrected_rate(Interval | None) -
locked_test_consumed(bool) -
bias_battery_ref(str | None) -
schema_version(str) -
trace_ids(tuple[str, ...]) -
planned_per_class(int | None) -
planned_coverage_floor(int | None) -
observed_coverage_floor(int | None) -
observed_floor_unmeasured(bool) -
judge_fingerprint(str | None) -
looks(int) -
agreement_alpha(Interval | None) -
provenance(ProvenanceStamp)
coverage_floor
property
¶
The floor that applies: the larger of the planned and the observed floor, if any.
effective_n_pos
property
¶
The FAIL-class gold size the bounds were taken at: n_pos / tpr_design_effect.
effective_n_neg
property
¶
The PASS-class gold size the bounds were taken at: n_neg / tnr_design_effect.
below_coverage_floor
property
¶
Whether either class of the gold is smaller than the coverage floor that applies.
Such a record certifies (the Wilson bound is valid at any n), but the corrected rate's
interval is not known to cover nominally at that size, so corrected_rate refuses it.
A record whose observed rates have no measured floor is below it at any size. Clustered
gold is compared at its effective size.
CalibrationSet
dataclass
¶
A split-tagged set of paired observations.
split tags the provenance: "few_shot" (never evaluated), "dev" (iterate
freely, never certifies), "locked_test" (the only source of a CERTIFIED verdict,
consumed exactly once). set_id names the set; the ledger also keys on its content, so a
renamed copy is still a used set.
Interval
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
A two-sided confidence interval for the Rogan-Gladen corrected rate.
point is None when the correction was refused (J<0.5, or non-identifiable);
refusal_reason records which. The (lo, hi) bound is always populated and is the
honest signal even when the point is refused.
method, level, sided and n say what kind of interval this is, so a 95%
two-sided interval and a 95% one-sided bound can never be confused downstream. n is the
sample the interval was computed over (for a two-sample difference, both samples together).
Config:
frozen:Trueextra:forbid
Fields:
-
point(float | None) -
lo(float) -
hi(float) -
refusal_reason(str | None) -
method(str) -
level(float) -
sided(Sided) -
n(int)
Validators:
require_point ¶
Return the corrected point rate, or raise the typed refusal that produced None.
The point-rate accessor: a caller wanting the scalar must go through this,
so a refused correction surfaces as a typed WeakJudgeError /
NonIdentifiableRateError rather than a silent None leaking into a report. The
(lo, hi) bound stays readable as the signal.
Source code in src/calibrated_agent_evaluation/calibrate/record.py
LabelPair
dataclass
¶
One paired observation: a human gold label and the judge's verdict for the same item.
Both reference the same (trace_id, span_id); the engine derives the 2x2 confusion counts
from a set of these, treating outcome == "FAIL" as the positive class.
CalibrationPlan
dataclass
¶
The per-class sizes plan_calibration derives, and the one to hunt.
certification certifies the judge; coverage_floor is the smallest size at which the
corrected rate's interval covers nominally at the planned accuracy; per_class is the
larger, the size to hunt, or the rate-safe size for a plan made with rate_safe=True. A
judge certified on fewer than coverage_floor per class is certified, but its corrected rate
is refused. double_coded is double_coded_per_class for gold two or more raters
will code, and per_class is then at least that; None for one rater.
locked_set_digest ¶
A hash of the set's targets and their gold outcomes, independent of order and set_id.
Source code in src/calibrated_agent_evaluation/calibrate/ledger.py
open_for_top_up ¶
Whether record's set may be grown: REFUSED short of its plan, by a fingerprinted judge.
A judge refused at its planned size, or certified, was measured on as much gold as was asked for, so more gold on top is a second test, not the first one completed.
Source code in src/calibrated_agent_evaluation/calibrate/ledger.py
pair_observations ¶
Reduce paired observations to 2x2 confusion counts.
Raises UnratifiedGoldError if any gold label is not kind == "human",
UnpairedObservationError if a pair's gold and verdict do not reference the same
(trace_id, span_id), and DuplicateGoldError if a target appears in more than one
pair. All are refusals, never silent drops.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pairs
|
Sequence[LabelPair]
|
The (judge verdict, gold label) pairs. |
required |
Returns:
| Type | Description |
|---|---|
ConfusionCounts
|
The 2x2 confusion counts. |
Raises:
| Type | Description |
|---|---|
UnratifiedGoldError
|
A gold label is not human. |
UnpairedObservationError
|
A verdict and its gold label name different targets. |
DuplicateGoldError
|
A target appears in more than one pair. |
Source code in src/calibrated_agent_evaluation/calibrate/pairing.py
refuse_duplicate_targets ¶
Refuse a gold set in which any (trace_id, span_id) target appears more than once.
Raises:
| Type | Description |
|---|---|
DuplicateGoldError
|
A target appears more than once; the error names the repeats. |
Source code in src/calibrated_agent_evaluation/calibrate/pairing.py
dump_records ¶
Write records to calibration_record.{csv,json} under path.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
records
|
Sequence[CalibrationRecord]
|
The records to persist. |
required |
path
|
str | Path
|
The directory to write into. |
required |
fmt
|
str
|
Which file a later |
'csv'
|
Raises:
| Type | Description |
|---|---|
CalibrationStoreError
|
A bad |
Source code in src/calibrated_agent_evaluation/calibrate/store.py
load_records ¶
Read calibration records back from path; a missing file is an empty store.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str | Path
|
The directory |
required |
fmt
|
str
|
|
'csv'
|
Returns:
| Type | Description |
|---|---|
list[CalibrationRecord]
|
The stored records. |
Raises:
| Type | Description |
|---|---|
CalibrationVersionError
|
A stored |
CalibrationStoreError
|
Any read, parse or validation failure. |
Source code in src/calibrated_agent_evaluation/calibrate/store.py
certifies ¶
Return whether record still certifies judge as it is now.
False when the record is not CERTIFIED or is not pinned to the judge (pin_mismatch):
a model, prompt version or prompt content change requires re-calibration.
Source code in src/calibrated_agent_evaluation/calibrate/engine.py
certify ¶
Run the locked-test gate and return a CERTIFIED/REFUSED CalibrationRecord.
Refuses (never certifies) unless locked.split == "locked_test" -- a dev set never certifies
and the few-shot pool is never evaluated. Consumes locked from ledger and refuses a set
whose id, content or any trace was already consumed (single-use). Gold must be human, one
label per target (enforced in pairing). CERTIFIED iff both lower bounds clear
GATE_THRESHOLD.
plan -- the CalibrationPlan the gold
was hunted against. Its per-class size and coverage floor are recorded, and gold below the floor
is marked (CalibrationRecord.below_coverage_floor) without changing the result: the judge
still certifies, and corrected_rate refuses an interval from the record.
A certified judge's floor is also planned at the TPR and TNR it showed, with or without a
plan, and the larger of the floors applies; a judge whose accuracy differs from the plan
can need more gold.
Each judge's certification stands alone. Certifying several judges in one run is not a joint claim that all of them are good, so no correction across judges is applied and none is recorded.
Promotion is an intersection-union test: a judge deficient in TPR or in TNR is certified with
probability at most about 5% (slightly more at some n, since Wilson coverage oscillates),
with no per-judge correction needed for using two bounds. The two bounds are not a joint 95%
region: at TPR = TNR = 0.80 and 80 per class their simultaneous coverage is 0.890.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
judge
|
Judge
|
The judge identity (id, model, prompt version) the record pins to. |
required |
locked
|
CalibrationSet
|
The locked test set of (judge verdict, human gold) pairs. |
required |
ledger
|
LockedTestLedger
|
The record of locked sets already used; |
required |
calib_id
|
str | None
|
The record id; generated when omitted. |
None
|
plan
|
CalibrationPlan | None
|
The calibration plan the gold was collected against. |
None
|
sampling
|
Mapping[str, float] | None
|
The sampling parameters the judge's verdicts were drawn with, recorded on the provenance stamp (empty: none were sent). |
None
|
agreement_alpha
|
Interval | None
|
The gold raters' agreement, recorded on the record when measured. |
None
|
A REFUSED set short of plan's per-class size is opened in ledger for a top-up by the
same judge: a later set holding all its traces plus fresh ones may be tested, and the record
counts the looks (looks) so the retest is disclosed.
Returns:
| Type | Description |
|---|---|
CalibrationRecord
|
A CERTIFIED or REFUSED calibration record. |
Raises:
| Type | Description |
|---|---|
FewShotEvaluationError
|
|
LockedTestReuseError
|
|
UnratifiedGoldError
|
A gold label is not human. |
UnpairedObservationError
|
A verdict and its gold label name different targets. |
DuplicateGoldError
|
A target carries more than one gold label. |
Source code in src/calibrated_agent_evaluation/calibrate/engine.py
156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 | |
corrected_rate ¶
corrected_rate(
record,
rate_sample_verdicts,
rate_set_id,
*,
rate_trace_ids,
seed=0,
n_resamples=DEFAULT_N_RESAMPLES,
rate_cluster_ids=None,
)
Return a copy of record with the Rogan-Gladen corrected rate + CI attached.
seed and n_resamples (the Monte Carlo draw count) drive the interval and are stamped
on the returned record's provenance, so the interval can be recomputed from the stored
record alone.
Refuses for an uncertified judge, and with RateUnsafeError for a record whose gold is
below its coverage floor, the larger of the planned and the observed one,
or whose observed rates have no measured floor, and with RateSampleTooSmallError for a
rate sample of fewer than MIN_RATE_SAMPLE
traces (distinct traces when rate_cluster_ids is given). When J<0.5 the corrected point is
refused (the interval is still returned as the signal); when the rate is non-identifiable the
point is refused and the bound reported. When more than one tail's share of the interval's draws
have no corrected rate, the interval is refused too: (lo, hi) = (0, 1) with a
refusal_reason starting "non-identifiable interval:". The interval always stays finite
and in [0,1].
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
record
|
CalibrationRecord
|
A CERTIFIED calibration record for the judge. |
required |
rate_sample_verdicts
|
Sequence[int]
|
The judge's verdicts (1 = FAIL, 0 = PASS) on a separate production-representative sample; their mean is the raw rate. |
required |
rate_set_id
|
str
|
The id of that sample; it must differ from |
required |
rate_trace_ids
|
Sequence[str]
|
The trace id of each verdict, in the same order. Required, because the rate sample must be checked against the calibration gold: hunted traces re-entering it break the independence the interval assumes. |
required |
seed
|
int
|
Fixes the Monte Carlo draws behind the interval. |
0
|
n_resamples
|
int
|
The number of Monte Carlo draws. |
DEFAULT_N_RESAMPLES
|
rate_cluster_ids
|
Sequence[str] | None
|
The trace id of each verdict, when one trace contributes several and the interval should account for the clustering. |
None
|
Returns:
| Type | Description |
|---|---|
CalibrationRecord
|
A copy of |
Raises:
| Type | Description |
|---|---|
UncalibratedJudgeError
|
|
RateUnsafeError
|
The record's gold is below its coverage floor. |
SampleMisuseError
|
The rate sample is empty, is the hunted calibration set, shares traces with the calibration gold, or its trace ids do not match its verdicts. |
RateSampleTooSmallError
|
The rate sample has fewer than |
Source code in src/calibrated_agent_evaluation/calibrate/engine.py
327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 | |
fingerprint_of ¶
The judge's content fingerprint, or None for a judge that carries none.
Source code in src/calibrated_agent_evaluation/calibrate/engine.py
pin_mismatch ¶
Why record is not pinned to judge as it is now, or None when it is.
record is any record pinning a judge (calibration, bias battery): it carries judge_id,
model, prompt_version and judge_fingerprint. A record made for another judge id
never pins this one. A judge with a fingerprint matches only a record stamped with the same
one, so an in-place prompt edit under an unchanged prompt_version is caught, and a record
stamped before fingerprints existed proves nothing about the current prompt. A record stamped
with a fingerprint never pins a judge that carries none, since nothing shows that judge's
content is the one measured. A judge without a fingerprint matches an unstamped record on
(judge_id, model, prompt_version) alone.
Source code in src/calibrated_agent_evaluation/calibrate/engine.py
coverage_floor ¶
Return the smallest per-class n at which the corrected-rate interval covers nominally.
Read from COVERAGE_FLOOR, which was measured at a set of p_hat values. Between two
measured values the larger of their floors applies; below the lowest measured value (0.85) its
floor applies, an extrapolation that rests on coverage improving as the per-class error count
grows. Depends on p_hat only: coverage was measured across prevalence 0.02 to 0.50 and
taken at its worst, and prevalence is what the rate sample estimates, so it cannot be an input.
Raises InfeasiblePlanError when p_hat is outside (0, 1] or no measured size
covers nominally there.
Source code in src/calibrated_agent_evaluation/calibrate/plan.py
double_coded_per_class ¶
The per-class gold that holds minimum_overlap double-coded traces, 65 at the default.
The calibration gold is balanced per class, so the agreement gate's overlap splits evenly across FAIL and PASS (rounded up).
Source code in src/calibrated_agent_evaluation/calibrate/plan.py
gate_fails_allowed ¶
The most FAIL verdicts among n that still clear the run gate; -1 when none do.
Source code in src/calibrated_agent_evaluation/calibrate/plan.py
gate_set_size ¶
The traces per failure mode the run gate needs to pass a suite that passes as expected.
calibrated-agent-evaluation run passes a mode when the 95% one-sided Wilson lower bound of
its pass rate reaches threshold, the same bound the calibration gate reads. So the size is
wilson_calibration_size at expected_pass: the smallest n from which a suite
passing expected_pass of the time (count rounded down) clears the gate, at n and at
every larger size, so adding traces never turns a suite as good as expected red.
Raises:
| Type | Description |
|---|---|
InfeasiblePlanError
|
|
Source code in src/calibrated_agent_evaluation/calibrate/plan.py
plan_calibration ¶
The sizes behind plan_calibration_size, each kept; see that function.
rate_safe -- for a caller who wants a corrected rate: per_class is then
rate_safe_calibration_size, and certification and coverage_floor are the same
as without it.
raters -- how many raters will code the gold. From two up the agreement gate measures
them on the gold itself, so per_class is raised to double_coded_per_class when it
is smaller.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
p_hat
|
float
|
The accuracy you expect the judge to have on each class (or on TPR, with
|
required |
tau
|
float
|
The lower bound each class must clear, normally |
required |
alpha
|
float
|
The two-sided level of the Wilson bound; 0.10 gives a one-sided 95% bound. |
0.1
|
tnr_hat
|
float | None
|
The expected TNR when it differs from |
None
|
rate_safe
|
bool
|
Size the plan for a corrected rate as well as certification. |
False
|
raters
|
int
|
How many raters will code the gold (1 or more). |
1
|
Returns:
| Type | Description |
|---|---|
CalibrationPlan
|
The per-class label count, the certification size, the coverage floor and, for |
CalibrationPlan
|
double-coded gold, the per-class size the agreement overlap needs. |
Raises:
| Type | Description |
|---|---|
InfeasiblePlanError
|
No label count can clear |
Source code in src/calibrated_agent_evaluation/calibrate/plan.py
plan_calibration_size ¶
Return the per-class n to hunt: enough to certify the judge and to estimate the rate.
The larger of wilson_calibration_size (the gate's own bound, which depends on tau
and alpha) and coverage_floor (the corrected-rate interval's measured minimum,
which depends on the expected accuracy only). The certification size is computed first, so a
tau the expected accuracy cannot reach is refused as before; the floor can only raise a
feasible plan or refuse it, never make an infeasible one feasible.
tnr_hat -- for a judge expected to be more accurate on one class than the other, its
expected TNR; p_hat is then its expected TPR. Both classes are hunted at the returned
size. The gate must certify both rates, so the certification size is taken at the lower of the
two; the floor is taken at the higher, the rule the coverage simulations checked (see the
module docstring). Omitted, the judge is
expected to be equally accurate on both classes and the plan is the one for p_hat alone.
raters -- from two up, the gold is also the raters' agreement overlap, and the size is at
least double_coded_per_class (65 per class at the defaults).
Raises InfeasiblePlanError when either job has no answer.
Source code in src/calibrated_agent_evaluation/calibrate/plan.py
rate_safe_at ¶
Whether n per class is rate-safe at every accuracy that would certify at n.
Certification marks a judge rate-unsafe when its gold is below the coverage floor at
the TPR and TNR it showed. At n per class the gate certifies a class on k correct when
the Wilson lower bound of k of n reaches tau, so a judge can show any such k/n.
n is rate-safe when the floor at every one of them is at most n. Accuracies above the
highest one with a measured floor are skipped: no size is known to make the interval cover
there, so no plan can help, and certify marks those records rate-unsafe at any size. With the
committed table every accuracy has a measured floor, so nothing is skipped.
Source code in src/calibrated_agent_evaluation/calibrate/plan.py
rate_safe_calibration_size ¶
The smallest per-class n to hunt when the caller wants a corrected rate.
At least the certification-only plan (plan_calibration_size), and rate-safe
(rate_safe_at) there and at every larger size, so a judge more accurate than planned
does not make its own record rate-unsafe. Every size from the largest measured floor up is
rate-safe, since no certifying accuracy with a measured floor needs more, so the search runs
down from there. With the committed table this is 162 per class at every planned accuracy from
0.88 to 0.99 at tau=0.80, because below 162 some count of errors always gives an accuracy
between 0.98 and 0.99, whose floor is 162; at 0.85 certifying alone needs 208.
Raises InfeasiblePlanError where the certification-only plan does: tau the
expected accuracy cannot reach, or a planned rate with no measured floor.
Source code in src/calibrated_agent_evaluation/calibrate/plan.py
wilson_calibration_size ¶
Return the smallest per-class n from which the gate's bound at p_hat clears tau.
The certification size alone. plan_calibration_size is what to hunt, because it also
covers the corrected rate's interval.
p_hat -- the judge accuracy you expect to observe (e.g. 0.90).
tau -- the gate threshold the lower bound must clear (e.g. 0.80).
alpha -- the two-sided level whose lower edge is the one-sided bound (default 0.10 -> 95%
LB).
The returned n holds from there on: for every n' >= n, a judge that gets
floor(p_hat * n') of n' right clears the bound. The first n that clears is not
enough, because the bound is a sawtooth in n: at p_hat=0.90 it first clears at 48, then
51 to 54 fall short again (51 labels at 90% is 45 correct, whose bound is 0.788). Collecting
more labels than planned must never un-certify a judge as accurate as planned.
The search is finite. With z the one-sided normal quantile and k = floor(p_hat * n), so
k / n >= p_hat - 1 / n, the Wilson lower bound is at least
(p_hat - 1 / n - z / (2 sqrt(n))) / (1 + z^2 / n), using k (n - k) / n^2 <= 1 / 4. That
clears tau once p_hat - tau >= z / (2 sqrt(n)) + (1 + tau z^2) / n, whose right side
falls as n grows, so every size from the first n meeting it clears the bound. The search
checks the sizes below that one.
Raises InfeasiblePlanError when tau >= p_hat (the point estimate never clears the
threshold, so its lower bound cannot either) or when the guarantee needs more than
100,000 per class.