Skip to content

calibrated_agent_evaluation.bias

calibrated_agent_evaluation.bias

The bias-check promotion gate -- the third judge-trust gate.

Runs the CALM bias battery (must-pass: length/self-preference/sycophancy; advisory: distraction/ bandwagon/compassion-fade; positional pairwise-only) over controlled probe items. Each probe is run with a placebo repeat, and its magnitude is the flip rate beyond the placebo's, bounded above by Newcombe's paired-difference interval. It refuses promotion of a judge that passed calibration but fails a must-pass bias -- filling the bias_battery_ref slot left on the calibration record.

Pure methodology plane: no vendor SDK type appears here. The judge is run through an injected ProbeRunner port. The shipped concrete runner is JudgeProbeRunner, which reaches the model through execute/'s JudgeModelBackend seam, so bias/ itself imports no vendor SDK.

ADVISORY module-attribute

ADVISORY = frozenset(
    {"distraction", "bandwagon", "compassion_fade"}
)

MUST_PASS module-attribute

MUST_PASS = frozenset(
    {"length", "self_preference", "sycophancy"}
)

BiasName module-attribute

BiasName = Literal[
    "length",
    "self_preference",
    "sycophancy",
    "distraction",
    "bandwagon",
    "compassion_fade",
    "positional",
]

BiasVerdict module-attribute

BiasVerdict = Literal['CLEAR', 'BIASED']

ProbeStatus module-attribute

ProbeStatus = Literal['run', 'not_applicable']

BIAS_SCHEMA_VERSION module-attribute

BIAS_SCHEMA_VERSION = '4'

BiasedJudgeError

BiasedJudgeError(message, *, decision=None)

Bases: BiasError

A must-pass bias's one-sided 95% upper bound exceeds its model-relative threshold.

The judge may have passed calibration; the bias gate is orthogonal and refuses promotion.

Upper, not lower: bias magnitude must stay under a ceiling, so the conservative bound is the upper edge. See ProbeResult for why the lower bound was wrong here.

Source code in src/calibrated_agent_evaluation/bias/errors.py
def __init__(self, message: str, *, decision: EnforcementDecision | None = None) -> None:
    super().__init__(message)
    self.decision = decision

BiasError

BiasError(message, *, decision=None)

Bases: CalibratedAgentEvaluationDataError

Base class for every bias-gate refusal.

Carries the optional EnforcementDecision that recorded the refusal, so a hard refusal both raises (control flow) and records (the audit trail) -- a caller catching the error reads err.decision for the audit record. Reuses the enforcement decision type rather than inventing a parallel one.

Source code in src/calibrated_agent_evaluation/bias/errors.py
def __init__(self, message: str, *, decision: EnforcementDecision | None = None) -> None:
    super().__init__(message)
    self.decision = decision

decision instance-attribute

decision = decision

BiasStoreError

BiasStoreError(message, *, decision=None)

Bases: BiasError

A bias-battery record file read or write failed.

Wraps the underlying OSError / json.JSONDecodeError so a filesystem/parse failure surfaces inside the typed family rather than as a raw stdlib exception escaping the store.

Source code in src/calibrated_agent_evaluation/bias/errors.py
def __init__(self, message: str, *, decision: EnforcementDecision | None = None) -> None:
    super().__init__(message)
    self.decision = decision

EmptyProbeError

EmptyProbeError(message, *, decision=None)

Bases: BiasError

A must-pass probe scored zero trials.

Fail-closed, and the counterpart to MissingBiasThresholdError: a configured threshold with nothing measured against it certifies nothing. Zero trials used to yield magnitude_lb95=0.0 and therefore passed=True unconditionally, so an unrun must-pass probe silently cleared the gate -- absence of evidence read as evidence of absence.

Source code in src/calibrated_agent_evaluation/bias/errors.py
def __init__(self, message: str, *, decision: EnforcementDecision | None = None) -> None:
    super().__init__(message)
    self.decision = decision

IncompleteBatteryError

IncompleteBatteryError(
    message, *, missing=(), decision=None
)

Bases: BiasError

The battery has no probes, or lacks a must-pass probe for a must-pass bias.

Fail-closed: a CLEAR verdict claims every must-pass bias was measured and stayed under its threshold. An empty battery, or one missing the sycophancy probe, used to come back CLEAR with nothing measured. missing names the must-pass biases with no must-pass probe.

Source code in src/calibrated_agent_evaluation/bias/errors.py
def __init__(
    self,
    message: str,
    *,
    missing: tuple[str, ...] = (),
    decision: EnforcementDecision | None = None,
) -> None:
    super().__init__(message, decision=decision)
    self.missing = missing

missing instance-attribute

missing = missing

MissingBiasBatteryError

MissingBiasBatteryError(message, *, decision=None)

Bases: BiasError

Promotion was checked for a judge whose bias_battery_ref is null.

No battery has been run, so bias is unproven -- refuse rather than promote on absence.

Source code in src/calibrated_agent_evaluation/bias/errors.py
def __init__(self, message: str, *, decision: EnforcementDecision | None = None) -> None:
    super().__init__(message)
    self.decision = decision

MissingBiasThresholdError

MissingBiasThresholdError(message, *, decision=None)

Bases: BiasError

No model-relative threshold is configured for a must-pass bias.

Fail-closed: an unconfigured model must not pass on a borrowed/permissive default -- bias magnitude varies ~10x across judge models ("enforcement, not enablement").

Source code in src/calibrated_agent_evaluation/bias/errors.py
def __init__(self, message: str, *, decision: EnforcementDecision | None = None) -> None:
    super().__init__(message)
    self.decision = decision

ProbeExecutionError

ProbeExecutionError(message, *, decision=None)

Bases: BiasError

A judge probe run failed.

Wraps the executor ExecutionError (chained from exc) so an execution failure surfaces inside the typed family; run_battery lets it propagate rather than recording a CLEAR verdict from partial results (no silent partial pass).

Source code in src/calibrated_agent_evaluation/bias/errors.py
def __init__(self, message: str, *, decision: EnforcementDecision | None = None) -> None:
    super().__init__(message)
    self.decision = decision

StaleBiasBatteryError

StaleBiasBatteryError(message, *, decision=None)

Bases: BiasError

The referenced battery does not match the judge under promotion.

Either its pinned (model, prompt_version) differs (a changed judge invalidates the battery), or the calibration record's bias_battery_ref does not name this battery. Treated as absent -- a stale battery cannot certify a changed judge.

Source code in src/calibrated_agent_evaluation/bias/errors.py
def __init__(self, message: str, *, decision: EnforcementDecision | None = None) -> None:
    super().__init__(message)
    self.decision = decision

BiasProbe dataclass

BiasProbe(bias, is_must_pass, trials, pairwise=False)

A controlled probe for one CALM bias.

trials is a tuple of trials; each trial is a tuple of ProbeInput\s whose fair verdicts should agree. is_must_pass marks whether an exceedance can drive the overall verdict to BIASED. pairwise marks the positional probe, which uses order-swap-and-tie instead of plain disagreement.

bias instance-attribute

bias

is_must_pass instance-attribute

is_must_pass

trials instance-attribute

trials

pairwise class-attribute instance-attribute

pairwise = field(default=False)

score

score(runner, judge_model, prompt_version)

Run every trial and its placebo through runner and tally the disagreements.

A trial disagrees when the judge's verdicts across its inputs differ. For the pairwise probe, a disagreement between the two orderings is resolved as a tie (order-swap-and-tie) and counted in tie_count too. The placebo reuses the trial's own verdict on its first input and adds len(trial) - 1 repeats of that input, the only extra calls. Any ProbeExecutionError from runner propagates (no partial scoring).

Source code in src/calibrated_agent_evaluation/bias/probe.py
def score(self, runner: ProbeRunner, judge_model: str, prompt_version: str) -> ProbeTally:
    """Run every trial and its placebo through ``runner`` and tally the disagreements.

    A trial disagrees when the judge's verdicts across its inputs differ. For the pairwise
    probe, a disagreement between the two orderings is resolved as a tie (order-swap-and-tie)
    and counted in ``tie_count`` too. The placebo reuses the trial's own verdict on its first
    input and adds ``len(trial) - 1`` repeats of that input, the only extra calls. Any
    ``ProbeExecutionError`` from ``runner`` propagates (no partial scoring).
    """
    biased = placebo = both = ties = 0
    directed = d_trial = d_placebo = d_both = 0
    for trial in self.trials:
        labels = [
            runner.run(judge_model, prompt_version, probe_input).label for probe_input in trial
        ]
        repeats = [
            runner.run(judge_model, prompt_version, replace(trial[0], repeat=i)).label
            for i in range(1, len(trial))
        ]
        if self.pairwise and len(labels) == 2:
            trial_flip = order_swap_and_tie(labels[0], labels[1]) == TIE
            ties += trial_flip
        else:
            trial_flip = len(set(labels)) > 1
        placebo_flip = len({labels[0], *repeats}) > 1
        biased += trial_flip
        placebo += placebo_flip
        both += trial_flip and placebo_flip
        pointed = [(j, inp.toward) for j, inp in enumerate(trial) if j and inp.toward]
        if pointed:
            trial_move = any(labels[0] != t and labels[j] == t for j, t in pointed)
            placebo_move = any(labels[0] != t and repeats[j - 1] == t for j, t in pointed)
            directed += 1
            d_trial += trial_move
            d_placebo += placebo_move
            d_both += trial_move and placebo_move
    return ProbeTally(
        n=len(self.trials),
        biased_count=biased,
        placebo_count=placebo,
        both_count=both,
        tie_count=ties,
        direction_n=directed,
        direction_count=d_trial,
        direction_placebo_count=d_placebo,
        direction_both_count=d_both,
    )

ProbeInput dataclass

ProbeInput(bias, probe_id, payload, repeat=0, toward=None)

One judge invocation's controlled content (vendor-free).

bias instance-attribute

bias

probe_id instance-attribute

probe_id

payload instance-attribute

payload

repeat class-attribute instance-attribute

repeat = 0

toward class-attribute instance-attribute

toward = None

ProbeRunner

Bases: Protocol

The port run_battery depends on.

The shipped adapter, JudgeProbeRunner, drives the judge runtime over an injected model backend and wraps a failure as ProbeExecutionError. bias/ never imports it -- dependency injection keeps the methodology plane vendor-free. run raises ProbeExecutionError on an execution failure.

run

run(judge_model, prompt_version, probe_input)

Judge one probe input with the given model and prompt version.

Parameters:

Name Type Description Default
judge_model str

The model id the judge runs on.

required
prompt_version str

The judge prompt version being checked.

required
probe_input ProbeInput

The probe trace and its perturbation.

required

Returns:

Type Description
ProbeVerdict

The judge's verdict on that input.

Source code in src/calibrated_agent_evaluation/bias/probe.py
def run(self, judge_model: str, prompt_version: str, probe_input: ProbeInput) -> ProbeVerdict:
    """Judge one probe input with the given model and prompt version.

    Args:
        judge_model: The model id the judge runs on.
        prompt_version: The judge prompt version being checked.
        probe_input: The probe trace and its perturbation.

    Returns:
        The judge's verdict on that input.
    """
    ...

ProbeVerdict dataclass

ProbeVerdict(label, detail='')

The judge's decision on one probe input, as a canonical label.

label is "PASS"/"FAIL" for a single-verdict judge, or the chosen candidate id for a pairwise judge. The probe compares labels across a trial's inputs; a fair judge returns the same label for inputs that differ only in the biasing dimension.

label instance-attribute

label

detail class-attribute instance-attribute

detail = ''

BiasBatteryRecord pydantic-model

Bases: CalibratedAgentEvaluationModel

The reproducible bias-gate evidence.

Pins the judge's exact (model, prompt_version) -- a change to either makes this record "absent" for the changed judge, the same version-invalidation discipline as the CalibrationRecord. Carries every probe's inputs/magnitude/threshold/verdict so a promotion decision is reconstructable from stored artifacts alone. Referenced by CalibrationRecord.bias_battery_ref (a judge-bound artifact referenced by id).

Config:

  • frozen: True
  • extra: forbid

Fields:

battery_id pydantic-field

battery_id

judge_id pydantic-field

judge_id

model pydantic-field

model

prompt_version pydantic-field

prompt_version

is_pairwise pydantic-field

is_pairwise

results pydantic-field

results

verdict pydantic-field

verdict

schema_version pydantic-field

schema_version = BIAS_SCHEMA_VERSION

failed_biases pydantic-field

failed_biases

judge_fingerprint pydantic-field

judge_fingerprint = None

provenance pydantic-field

provenance

ProbeResult pydantic-model

Bases: CalibratedAgentEvaluationModel

One bias probe's outcome.

The gate reads magnitude_ub95, a one-sided 95% upper bound on how much more often the judge's verdicts disagree across the biasing dimension than across a placebo that varies nothing, never the point magnitude: a bias is cleared only when we are confident it is small. The placebo nets out the judge's own sampling noise, which is not bias.

It read the lower bound until 2026-09-28, rationalized as "a judge is declared biased only when we are confident the bias is real". That is the right conservatism for a rate that must clear a floor and the wrong one for a rate that must stay under a ceiling. Bias magnitude is a badness measure, so gating its lower bound made the gate maximally permissive -- it meant "promote unless we are confident the judge IS biased", inverting the burden of proof, and it contradicted the bias gate's own rule, which gates the measured magnitude.

A probe whose inputs declare a direction also carries the directional test: how much more often the perturbed input moves the verdict toward the bias-favored label than the placebo repeat does (direction_magnitude), its one-sided 95% upper bound (direction_ub95) and the bound it must stay under (direction_threshold). direction_passed is None when the probe declares no direction.

passed is magnitude_ub95 <= threshold and, when the probe has a direction, the directional test passing too. It is None when the probe did not run (not_applicable) or no threshold applied (an unconfigured advisory bias). Only a probe with is_must_pass can drive the overall verdict to BIASED.

Config:

  • frozen: True
  • extra: forbid

Fields:

bias pydantic-field

bias

status pydantic-field

status

n pydantic-field

n

biased_count pydantic-field

biased_count

magnitude pydantic-field

magnitude

magnitude_ub95 pydantic-field

magnitude_ub95

threshold pydantic-field

threshold

passed pydantic-field

passed

is_must_pass pydantic-field

is_must_pass

tie_count pydantic-field

tie_count = 0

detail pydantic-field

detail = ''

placebo_count pydantic-field

placebo_count = 0

both_count pydantic-field

both_count = 0

direction_n pydantic-field

direction_n = 0

direction_count pydantic-field

direction_count = 0

direction_placebo_count pydantic-field

direction_placebo_count = 0

direction_both_count pydantic-field

direction_both_count = 0

direction_magnitude pydantic-field

direction_magnitude = None

direction_ub95 pydantic-field

direction_ub95 = None

direction_threshold pydantic-field

direction_threshold = None

direction_passed pydantic-field

direction_passed = None

BiasThresholds dataclass

BiasThresholds(by_model)

Per-model, per-bias maximum-allowed magnitude (the upper bound must not exceed it).

by_model instance-attribute

by_model

has_threshold

has_threshold(model, bias)

Whether a threshold is configured for (model, bias) (gates advisory biases).

Source code in src/calibrated_agent_evaluation/bias/thresholds.py
def has_threshold(self, model: str, bias: BiasName) -> bool:
    """Whether a threshold is configured for ``(model, bias)`` (gates advisory biases)."""
    return bias in self.by_model.get(model, {})

threshold_for

threshold_for(model, bias)

Return the configured threshold for (model, bias).

Raises MissingBiasThresholdError when bias is a must-pass bias with no configured threshold for model (fail-closed). For an advisory bias, callers gate on has_threshold first; calling this for an unconfigured advisory bias also raises (the caller bug of asking for a threshold that was never set).

Source code in src/calibrated_agent_evaluation/bias/thresholds.py
def threshold_for(self, model: str, bias: BiasName) -> float:
    """Return the configured threshold for ``(model, bias)``.

    Raises :class:`MissingBiasThresholdError` when ``bias`` is a must-pass bias with no
    configured threshold for ``model`` (fail-closed). For an advisory bias, callers
    gate on :meth:`has_threshold` first; calling this for an unconfigured advisory bias also
    raises (the caller bug of asking for a threshold that was never set).
    """
    model_thresholds = self.by_model.get(model, {})
    if bias in model_thresholds:
        return model_thresholds[bias]
    if bias in MUST_PASS:
        raise MissingBiasThresholdError(
            f"no bias threshold configured for must-pass bias {bias!r} on model {model!r}; "
            "refusing to produce a verdict (fail-closed). Configure a per-model "
            "threshold before running the battery."
        )
    raise MissingBiasThresholdError(
        f"no threshold configured for advisory bias {bias!r} on model {model!r}; "
        "check has_threshold() before calling threshold_for()"
    )

run_battery

run_battery(
    judge,
    runner,
    thresholds,
    probes,
    *,
    pairwise=False,
    battery_id=None,
    sampling=None,
)

Run the bias battery for judge and return a pinned CLEAR/BIASED record.

The positional probe runs only on a pairwise path; on a single-verdict path it is recorded as not_applicable. Must-pass exceedances (a must-pass probe whose magnitude_ub95 exceeds its threshold, or whose direction_ub95 exceeds DIRECTION_THRESHOLD) drive the verdict to BIASED; advisory exceedances are recorded but never change the verdict. sampling is the sampling parameters the runner's judge sends (see sampling), recorded on the provenance stamp. Every trial is run with its placebo, so a battery makes len(trial) - 1 extra judge calls per trial. Raises MissingBiasThresholdError or ProbeExecutionError rather than returning a partial CLEAR, and IncompleteBatteryError when probes is empty, or when the verdict would be CLEAR while a must-pass bias (MUST_PASS) had no must-pass probe run. A measured must-pass exceedance is still reported BIASED from a partial battery.

Parameters:

Name Type Description Default
judge Judge

The judge identity the record pins to.

required
runner ProbeRunner

Runs the judge on one probe input.

required
thresholds BiasThresholds

The per-bias thresholds.

required
probes Sequence[BiasProbe]

The probes to run.

required
pairwise bool

The judge compares two answers, so the positional probe applies.

False
battery_id str | None

The record id; generated when omitted.

None
sampling Mapping[str, float] | None

The sampling parameters the judge sends, recorded on the provenance stamp.

None

Returns:

Type Description
BiasBatteryRecord

The CLEAR or BIASED battery record.

Raises:

Type Description
IncompleteBatteryError

No probes, or a CLEAR verdict with a must-pass bias unmeasured.

MissingBiasThresholdError

A probe's bias has no threshold.

ProbeExecutionError

The runner failed on a probe input.

Source code in src/calibrated_agent_evaluation/bias/battery.py
def run_battery(
    judge: Judge,
    runner: ProbeRunner,
    thresholds: BiasThresholds,
    probes: Sequence[BiasProbe],
    *,
    pairwise: bool = False,
    battery_id: str | None = None,
    sampling: Mapping[str, float] | None = None,
) -> BiasBatteryRecord:
    """Run the bias battery for ``judge`` and return a pinned CLEAR/BIASED record.

    The positional probe runs only on a pairwise path; on a single-verdict path it is recorded as
    ``not_applicable``. Must-pass exceedances (a must-pass probe whose ``magnitude_ub95`` exceeds
    its threshold, or whose ``direction_ub95`` exceeds :data:`DIRECTION_THRESHOLD`) drive the
    verdict to ``BIASED``; advisory exceedances are recorded but never
    change the verdict. ``sampling`` is the sampling parameters the runner's judge sends (see
    :meth:`calibrated_agent_evaluation.judges.ExecutableJudge.sampling`), recorded on the provenance
    stamp. Every trial is run with its placebo, so a battery makes ``len(trial) - 1`` extra judge
    calls per trial. Raises
    :class:`~calibrated_agent_evaluation.bias.errors.MissingBiasThresholdError` or
    :class:`~calibrated_agent_evaluation.bias.errors.ProbeExecutionError` rather than returning a
    partial CLEAR, and :class:`~calibrated_agent_evaluation.bias.errors.IncompleteBatteryError` when
    ``probes`` is empty, or when the verdict would be CLEAR while a must-pass bias
    (:data:`MUST_PASS`) had no must-pass probe run. A measured must-pass exceedance is still
    reported BIASED from a partial battery.

    Args:
        judge: The judge identity the record pins to.
        runner: Runs the judge on one probe input.
        thresholds: The per-bias thresholds.
        probes: The probes to run.
        pairwise: The judge compares two answers, so the positional probe applies.
        battery_id: The record id; generated when omitted.
        sampling: The sampling parameters the judge sends, recorded on the provenance stamp.

    Returns:
        The CLEAR or BIASED battery record.

    Raises:
        IncompleteBatteryError: No probes, or a CLEAR verdict with a must-pass bias unmeasured.
        MissingBiasThresholdError: A probe's bias has no threshold.
        ProbeExecutionError: The runner failed on a probe input.
    """
    if not probes:
        raise IncompleteBatteryError(
            "the bias battery has no probes, so nothing was measured and no verdict can be "
            f"given; a verdict needs a must-pass probe for each of {sorted(MUST_PASS)}. Point the "
            "battery at a probe directory that holds them",
            missing=tuple(sorted(MUST_PASS)),
        )
    results = [
        _dispatch_probe(probe, runner, judge, thresholds, pairwise=pairwise) for probe in probes
    ]

    failed_biases: tuple[BiasName, ...] = tuple(
        r.bias
        for r in results
        if r.is_must_pass and r.passed is False  # nosemgrep: is-function-without-parentheses
    )
    if not failed_biases:
        _refuse_unearned_clear(probes, results)
    verdict: BiasVerdict = "BIASED" if failed_biases else "CLEAR"
    return BiasBatteryRecord(
        battery_id=battery_id or uuid.uuid4().hex,
        judge_id=judge.judge_id,
        model=judge.model,
        prompt_version=judge.prompt_version,
        judge_fingerprint=fingerprint_of(judge),
        is_pairwise=pairwise,
        results=tuple(results),
        verdict=verdict,
        failed_biases=failed_biases,
        provenance=capture_provenance(
            thresholds={
                **{r.bias: r.threshold for r in results if r.threshold is not None},
                **{
                    f"{r.bias}_direction": r.direction_threshold
                    for r in results
                    if r.direction_threshold is not None
                },
            },
            sampling=sampling,
        ),
    )

enforce_bias_gate

enforce_bias_gate(calibration_record, judge, battery)

Refuse promotion unless the judge has a fresh, CLEAR bias battery.

Refuses when (a) bias_battery_ref is null -> MissingBiasBatteryError; (b) the referenced battery's (model, prompt_version) differs from judge or its id does not match the record's ref -> StaleBiasBatteryError (treated as absent); (c) the battery verdict is BIASED -> BiasedJudgeError. Returns cleanly otherwise. Each refusal carries a recorded hard_refusal decision.

Parameters:

Name Type Description Default
calibration_record CalibrationRecord

The judge's calibration record, which references its battery.

required
judge Judge

The judge being promoted.

required
battery BiasBatteryRecord

The bias battery record.

required

Raises:

Type Description
MissingBiasBatteryError

The calibration record references no battery.

StaleBiasBatteryError

The battery is for another model, prompt version or record.

BiasedJudgeError

The battery's verdict is BIASED.

Source code in src/calibrated_agent_evaluation/bias/gate.py
def enforce_bias_gate(
    calibration_record: CalibrationRecord, judge: Judge, battery: BiasBatteryRecord
) -> None:
    """Refuse promotion unless the judge has a fresh, CLEAR bias battery.

    Refuses when (a) ``bias_battery_ref`` is null -> :class:`MissingBiasBatteryError`;
    (b) the referenced battery's ``(model, prompt_version)`` differs from ``judge`` or its id does
    not match the record's ref -> :class:`StaleBiasBatteryError` (treated as absent); (c) the
    battery verdict is ``BIASED`` -> :class:`BiasedJudgeError`. Returns cleanly otherwise. Each
    refusal carries a recorded ``hard_refusal`` decision.

    Args:
        calibration_record: The judge's calibration record, which references its battery.
        judge: The judge being promoted.
        battery: The bias battery record.

    Raises:
        MissingBiasBatteryError: The calibration record references no battery.
        StaleBiasBatteryError: The battery is for another model, prompt version or record.
        BiasedJudgeError: The battery's verdict is BIASED.
    """
    if calibration_record.bias_battery_ref is None:
        raise MissingBiasBatteryError(
            f"judge {judge.judge_id!r} has no bias battery (bias_battery_ref is null); "
            "refusing to promote on unproven bias. Run the bias battery first.",
            decision=_refusal("BIAS-2", "no bias battery run", judge, battery),
        )
    mismatch = pin_mismatch(battery, judge)
    if mismatch is not None or calibration_record.bias_battery_ref != battery.battery_id:
        why = mismatch or (
            f"the calibration record references battery {calibration_record.bias_battery_ref!r}, "
            f"not {battery.battery_id!r}"
        )
        raise StaleBiasBatteryError(
            f"the bias battery for judge {judge.judge_id!r} is stale and treated as absent: {why}. "
            "Run the bias battery again for the current judge.",
            decision=_refusal("BIAS-3", "stale bias battery", judge, battery),
        )
    if battery.verdict == "BIASED":
        raise BiasedJudgeError(
            f"judge {judge.judge_id!r} failed the bias gate on must-pass bias(es) "
            f"{list(battery.failed_biases)}; refusing promotion despite calibration. Fix the "
            "judge prompt and run the bias battery again.",
            decision=_refusal("BIAS-1", "biased judge", judge, battery),
        )
link_battery_to_calibration(calibration_record, battery)

Return a copy of calibration_record with bias_battery_ref set to it.

Frozen dataclass -> dataclasses.replace. Set for both CLEAR and BIASED batteries so a run is always auditable via the calibration record.

Source code in src/calibrated_agent_evaluation/bias/gate.py
def link_battery_to_calibration(
    calibration_record: CalibrationRecord, battery: BiasBatteryRecord
) -> CalibrationRecord:
    """Return a copy of ``calibration_record`` with ``bias_battery_ref`` set to it.

    Frozen dataclass -> ``dataclasses.replace``. Set for both CLEAR and BIASED batteries so a run
    is always auditable via the calibration record.
    """
    return replace(calibration_record, bias_battery_ref=battery.battery_id)

load_probe_set

load_probe_set(directory)

Load every *.json probe file under directory, sorted by file name.

Raises ProbeExecutionError when the directory holds no probe files, or any file is unreadable or missing bias / must_pass / trials (an empty battery would certify nothing, so it is refused here rather than reaching the gate).

Source code in src/calibrated_agent_evaluation/bias/loader.py
def load_probe_set(directory: str | Path) -> list[BiasProbe]:
    """Load every ``*.json`` probe file under ``directory``, sorted by file name.

    Raises :class:`ProbeExecutionError` when the directory holds no probe files, or any file is
    unreadable or missing ``bias`` / ``must_pass`` / ``trials`` (an empty battery would certify
    nothing, so it is refused here rather than reaching the gate).
    """
    root = Path(directory)
    files = sorted(root.glob("*.json")) if root.is_dir() else []
    if not files:
        raise ProbeExecutionError(f"no probe files (*.json) found under {root}")
    return [_probe_from_doc(_read_json(f), f) for f in files]

load_thresholds

load_thresholds(path)

Load per-model, per-bias thresholds from a {"by_model": {model: {bias: float}}} file.

Raises ProbeExecutionError on an unreadable file or a non-numeric threshold.

Source code in src/calibrated_agent_evaluation/bias/loader.py
def load_thresholds(path: str | Path) -> BiasThresholds:
    """Load per-model, per-bias thresholds from a ``{"by_model": {model: {bias: float}}}`` file.

    Raises :class:`ProbeExecutionError` on an unreadable file or a non-numeric threshold.
    """
    file = Path(path)
    doc = _read_json(file)
    try:
        items = [(str(m), dict(b)) for m, b in doc["by_model"].items()]
    except (KeyError, TypeError, AttributeError, ValueError) as exc:
        raise _malformed(
            file, f"expected {{'by_model': {{model: {{bias: float}}}}}}: {exc!r}"
        ) from exc
    by_model: dict[str, dict[BiasName, float]] = {}
    for model, biases in items:
        row: dict[BiasName, float] = {}
        for bias, value in biases.items():
            if isinstance(value, bool) or not isinstance(value, int | float):
                raise _malformed(file, f"threshold for {model}/{bias} is {value!r}, not a number")
            row[cast(BiasName, bias)] = float(value)
        by_model[model] = row
    return BiasThresholds(by_model=by_model)

order_swap_and_tie

order_swap_and_tie(label_ab, label_ba)

Return the consistent winner across both orderings, or TIE when they disagree.

label_ab is the judge's pick with the candidates in one order; label_ba with the order swapped. A fair judge picks the same underlying candidate regardless of order, so a disagreement is an order effect -- neutralized as a tie.

Source code in src/calibrated_agent_evaluation/bias/mitigate.py
def order_swap_and_tie(label_ab: str, label_ba: str) -> str:
    """Return the consistent winner across both orderings, or :data:`TIE` when they disagree.

    ``label_ab`` is the judge's pick with the candidates in one order; ``label_ba`` with the order
    swapped. A fair judge picks the same underlying candidate regardless of order, so a
    disagreement is an order effect -- neutralized as a tie.
    """
    return label_ab if label_ab == label_ba else TIE

dump_battery

dump_battery(records, path, fmt='csv')

Write records to bias_battery_record.{csv,json} under path.

Parameters:

Name Type Description Default
records Sequence[BiasBatteryRecord]

The battery records to persist.

required
path str | Path

The directory to write into.

required
fmt str

Which file a later load_battery reads back (both are always written).

'csv'

Raises:

Type Description
BiasStoreError

A bad fmt or any filesystem failure.

Source code in src/calibrated_agent_evaluation/bias/store.py
def dump_battery(records: Sequence[BiasBatteryRecord], path: str | Path, fmt: str = "csv") -> None:
    """Write ``records`` to ``bias_battery_record.{csv,json}`` under ``path``.

    Args:
        records: The battery records to persist.
        path: The directory to write into.
        fmt: Which file a later :func:`load_battery` reads back (both are always written).

    Raises:
        BiasStoreError: A bad ``fmt`` or any filesystem failure.
    """
    _STORE.dump(records, path, fmt)

load_battery

load_battery(path, fmt='csv')

Read bias battery records back from path; a missing file is an empty store.

Parameters:

Name Type Description Default
path str | Path

The directory dump_battery wrote.

required
fmt str

"csv" or "json".

'csv'

Returns:

Type Description
list[BiasBatteryRecord]

The stored records.

Raises:

Type Description
BiasStoreError

A schema-version mismatch, or any read, parse or validation failure.

Source code in src/calibrated_agent_evaluation/bias/store.py
def load_battery(path: str | Path, fmt: str = "csv") -> list[BiasBatteryRecord]:
    """Read bias battery records back from ``path``; a missing file is an empty store.

    Args:
        path: The directory :func:`dump_battery` wrote.
        fmt: ``"csv"`` or ``"json"``.

    Returns:
        The stored records.

    Raises:
        BiasStoreError: A schema-version mismatch, or any read, parse or validation failure.
    """
    return _STORE.load(path, fmt)