calibrated_agent_evaluation.bias¶
calibrated_agent_evaluation.bias ¶
The bias-check promotion gate -- the third judge-trust gate.
Runs the CALM bias battery (must-pass: length/self-preference/sycophancy; advisory: distraction/
bandwagon/compassion-fade; positional pairwise-only) over controlled probe items. Each probe is
run with a placebo repeat, and its magnitude is the flip rate beyond the placebo's, bounded above
by Newcombe's paired-difference interval. It refuses promotion of a judge that passed calibration
but fails a must-pass bias -- filling the bias_battery_ref slot left on the calibration record.
Pure methodology plane: no vendor SDK type appears here. The judge is run through an injected
ProbeRunner port. The shipped concrete runner is
JudgeProbeRunner, which reaches the model through
execute/'s JudgeModelBackend seam, so bias/ itself imports no vendor SDK.
BiasName
module-attribute
¶
BiasName = Literal[
"length",
"self_preference",
"sycophancy",
"distraction",
"bandwagon",
"compassion_fade",
"positional",
]
BiasedJudgeError ¶
Bases: BiasError
A must-pass bias's one-sided 95% upper bound exceeds its model-relative threshold.
The judge may have passed calibration; the bias gate is orthogonal and refuses promotion.
Upper, not lower: bias magnitude must stay under a ceiling, so the conservative bound is the
upper edge. See ProbeResult for why the lower bound was wrong here.
Source code in src/calibrated_agent_evaluation/bias/errors.py
BiasError ¶
Bases: CalibratedAgentEvaluationDataError
Base class for every bias-gate refusal.
Carries the optional EnforcementDecision
that recorded the refusal, so a hard refusal both raises (control flow) and records (the
audit trail) -- a caller catching the error reads err.decision for the audit record. Reuses
the enforcement decision type rather than inventing a parallel one.
Source code in src/calibrated_agent_evaluation/bias/errors.py
BiasStoreError ¶
Bases: BiasError
A bias-battery record file read or write failed.
Wraps the underlying OSError / json.JSONDecodeError so a filesystem/parse failure
surfaces inside the typed family rather than as a raw stdlib exception escaping the store.
Source code in src/calibrated_agent_evaluation/bias/errors.py
EmptyProbeError ¶
Bases: BiasError
A must-pass probe scored zero trials.
Fail-closed, and the counterpart to MissingBiasThresholdError: a configured threshold
with nothing measured against it certifies nothing. Zero trials used to yield
magnitude_lb95=0.0 and therefore passed=True unconditionally, so an unrun must-pass
probe silently cleared the gate -- absence of evidence read as evidence of absence.
Source code in src/calibrated_agent_evaluation/bias/errors.py
IncompleteBatteryError ¶
Bases: BiasError
The battery has no probes, or lacks a must-pass probe for a must-pass bias.
Fail-closed: a CLEAR verdict claims every must-pass bias was measured and stayed under its
threshold. An empty battery, or one missing the sycophancy probe, used to come back CLEAR with
nothing measured. missing names the must-pass biases with no must-pass probe.
Source code in src/calibrated_agent_evaluation/bias/errors.py
MissingBiasBatteryError ¶
Bases: BiasError
Promotion was checked for a judge whose bias_battery_ref is null.
No battery has been run, so bias is unproven -- refuse rather than promote on absence.
Source code in src/calibrated_agent_evaluation/bias/errors.py
MissingBiasThresholdError ¶
Bases: BiasError
No model-relative threshold is configured for a must-pass bias.
Fail-closed: an unconfigured model must not pass on a borrowed/permissive default -- bias magnitude varies ~10x across judge models ("enforcement, not enablement").
Source code in src/calibrated_agent_evaluation/bias/errors.py
ProbeExecutionError ¶
Bases: BiasError
A judge probe run failed.
Wraps the executor ExecutionError (chained from exc) so an execution failure surfaces
inside the typed family; run_battery lets it propagate rather than recording a CLEAR
verdict from partial results (no silent partial pass).
Source code in src/calibrated_agent_evaluation/bias/errors.py
StaleBiasBatteryError ¶
Bases: BiasError
The referenced battery does not match the judge under promotion.
Either its pinned (model, prompt_version) differs (a changed judge invalidates the
battery), or the calibration record's bias_battery_ref does not name this battery. Treated
as absent -- a stale battery cannot certify a changed judge.
Source code in src/calibrated_agent_evaluation/bias/errors.py
BiasProbe
dataclass
¶
A controlled probe for one CALM bias.
trials is a tuple of trials; each trial is a tuple of ProbeInput\s whose fair
verdicts should agree. is_must_pass marks whether an exceedance can drive the overall
verdict to BIASED. pairwise marks the positional probe, which uses
order-swap-and-tie instead of plain disagreement.
score ¶
Run every trial and its placebo through runner and tally the disagreements.
A trial disagrees when the judge's verdicts across its inputs differ. For the pairwise
probe, a disagreement between the two orderings is resolved as a tie (order-swap-and-tie)
and counted in tie_count too. The placebo reuses the trial's own verdict on its first
input and adds len(trial) - 1 repeats of that input, the only extra calls. Any
ProbeExecutionError from runner propagates (no partial scoring).
Source code in src/calibrated_agent_evaluation/bias/probe.py
ProbeInput
dataclass
¶
ProbeRunner ¶
Bases: Protocol
The port run_battery depends on.
The shipped adapter, JudgeProbeRunner, drives the
judge runtime over an injected model backend and wraps a failure as
ProbeExecutionError.
bias/ never imports it -- dependency injection keeps the methodology plane vendor-free.
run raises ProbeExecutionError on an execution failure.
run ¶
Judge one probe input with the given model and prompt version.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
judge_model
|
str
|
The model id the judge runs on. |
required |
prompt_version
|
str
|
The judge prompt version being checked. |
required |
probe_input
|
ProbeInput
|
The probe trace and its perturbation. |
required |
Returns:
| Type | Description |
|---|---|
ProbeVerdict
|
The judge's verdict on that input. |
Source code in src/calibrated_agent_evaluation/bias/probe.py
ProbeVerdict
dataclass
¶
The judge's decision on one probe input, as a canonical label.
label is "PASS"/"FAIL" for a single-verdict judge, or the chosen candidate id for a
pairwise judge. The probe compares labels across a trial's inputs; a fair judge returns the same
label for inputs that differ only in the biasing dimension.
BiasBatteryRecord
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
The reproducible bias-gate evidence.
Pins the judge's exact (model, prompt_version) -- a change to either makes this record
"absent" for the changed judge, the same version-invalidation discipline as the
CalibrationRecord. Carries every probe's inputs/magnitude/threshold/verdict so a promotion
decision is reconstructable from stored artifacts alone. Referenced by
CalibrationRecord.bias_battery_ref (a judge-bound artifact referenced by id).
Config:
frozen:Trueextra:forbid
Fields:
-
battery_id(str) -
judge_id(str) -
model(str) -
prompt_version(str) -
is_pairwise(bool) -
results(tuple[ProbeResult, ...]) -
verdict(BiasVerdict) -
schema_version(str) -
failed_biases(tuple[BiasName, ...]) -
judge_fingerprint(str | None) -
provenance(ProvenanceStamp)
ProbeResult
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
One bias probe's outcome.
The gate reads magnitude_ub95, a one-sided 95% upper bound on how much more often the
judge's verdicts disagree across the biasing dimension than across a placebo that varies
nothing, never the point magnitude: a bias is cleared only when we are confident it is
small. The placebo nets out the judge's own sampling noise, which is not bias.
It read the lower bound until 2026-09-28, rationalized as "a judge is declared biased only when we are confident the bias is real". That is the right conservatism for a rate that must clear a floor and the wrong one for a rate that must stay under a ceiling. Bias magnitude is a badness measure, so gating its lower bound made the gate maximally permissive -- it meant "promote unless we are confident the judge IS biased", inverting the burden of proof, and it contradicted the bias gate's own rule, which gates the measured magnitude.
A probe whose inputs declare a direction also carries the directional test: how much more
often the perturbed input moves the verdict toward the bias-favored label than the placebo
repeat does (direction_magnitude), its one-sided 95% upper bound (direction_ub95) and
the bound it must stay under (direction_threshold). direction_passed is None when
the probe declares no direction.
passed is magnitude_ub95 <= threshold and, when the probe has a direction, the
directional test passing too. It is None when the probe did not run (not_applicable)
or no threshold applied (an unconfigured advisory bias). Only a probe with is_must_pass
can drive the overall verdict to BIASED.
Config:
frozen:Trueextra:forbid
Fields:
-
bias(BiasName) -
status(ProbeStatus) -
n(int) -
biased_count(int) -
magnitude(float) -
magnitude_ub95(float) -
threshold(float | None) -
passed(bool | None) -
is_must_pass(bool) -
tie_count(int) -
detail(str) -
placebo_count(int) -
both_count(int) -
direction_n(int) -
direction_count(int) -
direction_placebo_count(int) -
direction_both_count(int) -
direction_magnitude(float | None) -
direction_ub95(float | None) -
direction_threshold(float | None) -
direction_passed(bool | None)
BiasThresholds
dataclass
¶
Per-model, per-bias maximum-allowed magnitude (the upper bound must not exceed it).
has_threshold ¶
Whether a threshold is configured for (model, bias) (gates advisory biases).
threshold_for ¶
Return the configured threshold for (model, bias).
Raises MissingBiasThresholdError when bias is a must-pass bias with no
configured threshold for model (fail-closed). For an advisory bias, callers
gate on has_threshold first; calling this for an unconfigured advisory bias also
raises (the caller bug of asking for a threshold that was never set).
Source code in src/calibrated_agent_evaluation/bias/thresholds.py
run_battery ¶
run_battery(
judge,
runner,
thresholds,
probes,
*,
pairwise=False,
battery_id=None,
sampling=None,
)
Run the bias battery for judge and return a pinned CLEAR/BIASED record.
The positional probe runs only on a pairwise path; on a single-verdict path it is recorded as
not_applicable. Must-pass exceedances (a must-pass probe whose magnitude_ub95 exceeds
its threshold, or whose direction_ub95 exceeds DIRECTION_THRESHOLD) drive the
verdict to BIASED; advisory exceedances are recorded but never
change the verdict. sampling is the sampling parameters the runner's judge sends (see
sampling), recorded on the provenance
stamp. Every trial is run with its placebo, so a battery makes len(trial) - 1 extra judge
calls per trial. Raises
MissingBiasThresholdError or
ProbeExecutionError rather than returning a
partial CLEAR, and IncompleteBatteryError when
probes is empty, or when the verdict would be CLEAR while a must-pass bias
(MUST_PASS) had no must-pass probe run. A measured must-pass exceedance is still
reported BIASED from a partial battery.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
judge
|
Judge
|
The judge identity the record pins to. |
required |
runner
|
ProbeRunner
|
Runs the judge on one probe input. |
required |
thresholds
|
BiasThresholds
|
The per-bias thresholds. |
required |
probes
|
Sequence[BiasProbe]
|
The probes to run. |
required |
pairwise
|
bool
|
The judge compares two answers, so the positional probe applies. |
False
|
battery_id
|
str | None
|
The record id; generated when omitted. |
None
|
sampling
|
Mapping[str, float] | None
|
The sampling parameters the judge sends, recorded on the provenance stamp. |
None
|
Returns:
| Type | Description |
|---|---|
BiasBatteryRecord
|
The CLEAR or BIASED battery record. |
Raises:
| Type | Description |
|---|---|
IncompleteBatteryError
|
No probes, or a CLEAR verdict with a must-pass bias unmeasured. |
MissingBiasThresholdError
|
A probe's bias has no threshold. |
ProbeExecutionError
|
The runner failed on a probe input. |
Source code in src/calibrated_agent_evaluation/bias/battery.py
200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 | |
enforce_bias_gate ¶
Refuse promotion unless the judge has a fresh, CLEAR bias battery.
Refuses when (a) bias_battery_ref is null -> MissingBiasBatteryError;
(b) the referenced battery's (model, prompt_version) differs from judge or its id does
not match the record's ref -> StaleBiasBatteryError (treated as absent); (c) the
battery verdict is BIASED -> BiasedJudgeError. Returns cleanly otherwise. Each
refusal carries a recorded hard_refusal decision.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
calibration_record
|
CalibrationRecord
|
The judge's calibration record, which references its battery. |
required |
judge
|
Judge
|
The judge being promoted. |
required |
battery
|
BiasBatteryRecord
|
The bias battery record. |
required |
Raises:
| Type | Description |
|---|---|
MissingBiasBatteryError
|
The calibration record references no battery. |
StaleBiasBatteryError
|
The battery is for another model, prompt version or record. |
BiasedJudgeError
|
The battery's verdict is BIASED. |
Source code in src/calibrated_agent_evaluation/bias/gate.py
link_battery_to_calibration ¶
Return a copy of calibration_record with bias_battery_ref set to it.
Frozen dataclass -> dataclasses.replace. Set for both CLEAR and BIASED batteries so a run
is always auditable via the calibration record.
Source code in src/calibrated_agent_evaluation/bias/gate.py
load_probe_set ¶
Load every *.json probe file under directory, sorted by file name.
Raises ProbeExecutionError when the directory holds no probe files, or any file is
unreadable or missing bias / must_pass / trials (an empty battery would certify
nothing, so it is refused here rather than reaching the gate).
Source code in src/calibrated_agent_evaluation/bias/loader.py
load_thresholds ¶
Load per-model, per-bias thresholds from a {"by_model": {model: {bias: float}}} file.
Raises ProbeExecutionError on an unreadable file or a non-numeric threshold.
Source code in src/calibrated_agent_evaluation/bias/loader.py
order_swap_and_tie ¶
Return the consistent winner across both orderings, or TIE when they disagree.
label_ab is the judge's pick with the candidates in one order; label_ba with the order
swapped. A fair judge picks the same underlying candidate regardless of order, so a
disagreement is an order effect -- neutralized as a tie.
Source code in src/calibrated_agent_evaluation/bias/mitigate.py
dump_battery ¶
Write records to bias_battery_record.{csv,json} under path.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
records
|
Sequence[BiasBatteryRecord]
|
The battery records to persist. |
required |
path
|
str | Path
|
The directory to write into. |
required |
fmt
|
str
|
Which file a later |
'csv'
|
Raises:
| Type | Description |
|---|---|
BiasStoreError
|
A bad |
Source code in src/calibrated_agent_evaluation/bias/store.py
load_battery ¶
Read bias battery records back from path; a missing file is an empty store.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str | Path
|
The directory |
required |
fmt
|
str
|
|
'csv'
|
Returns:
| Type | Description |
|---|---|
list[BiasBatteryRecord]
|
The stored records. |
Raises:
| Type | Description |
|---|---|
BiasStoreError
|
A schema-version mismatch, or any read, parse or validation failure. |