Skip to content

calibrated_agent_evaluation.enforce

calibrated_agent_evaluation.enforce

The tiered enforcement / lint-policy layer -- "enforcement, not enablement".

The methodology-plane surface that makes the right habit the default and the wrong habit the hard path, tiering each intervention to the strength of the evidence: hard refusals only where a pattern is provably invalid, warnings + nudges where it is merely directional, and display defaults that shape output. It reads -- never recomputes -- the calibration verdict to gate scoring, and reuses the shared cohen_kappa for the imbalance display.

Pure methodology plane: no vendor SDK type appears here, and no new runtime dependency is introduced. Every decision is captured as an auditable EnforcementDecision. It does NOT run judges, parse CLI flags, or compute the calibration statistics.

Tier module-attribute

Tier = Literal["hard_refusal", "warning", "display_default"]

Scale module-attribute

Scale = Literal['binary', 'likert']

GENERIC_GENERATION_METRICS module-attribute

GENERIC_GENERATION_METRICS = frozenset(
    {"rouge", "bertscore", "bleu", "meteor", "helpfulness"}
)

ENFORCE_SCHEMA_VERSION module-attribute

ENFORCE_SCHEMA_VERSION = '1'

ChecklistResult dataclass

ChecklistResult(overall, non_gating)

The collapsed verdict plus the non-gating sub-metrics.

overall is FAIL iff at least one critical item is FAIL; non_gating records every non-critical item's verdict (tracked, never gating).

overall instance-attribute

overall

non_gating instance-attribute

non_gating

EnforcementContext dataclass

EnforcementContext(ci=False, allow_likert=False)

The mode/capabilities a lint or gate call runs under.

ci -- while true, generic generation-quality metrics are hard-refused. allow_likert -- while true, a single Likert axis is permitted with a warning; disabled by default.

ci class-attribute instance-attribute

ci = False

allow_likert class-attribute instance-attribute

allow_likert = False

EnforcementDecision pydantic-model

Bases: CalibratedAgentEvaluationModel

One recorded enforcement decision.

rule is the governing id (e.g. "", ""); tier is exactly one of Tier; inputs are the evaluated inputs (non-empty); outcome is "refused"/"warned"/"shaped"; message is the human-facing explanation; nudge is the structural steer where applicable.

Config:

  • frozen: True
  • extra: forbid

Fields:

Validators:

  • _check_invariants

rule pydantic-field

rule

tier pydantic-field

tier

inputs pydantic-field

inputs

outcome pydantic-field

outcome

message pydantic-field

message

nudge pydantic-field

nudge = None

schema_version pydantic-field

schema_version = ENFORCE_SCHEMA_VERSION

CritiqueOrderError

CritiqueOrderError(message, *, decision=None)

Bases: EnforcementError

A judge output schema omits critique or orders verdict before it.

The binary/critique-first discipline is structural; critique precedes verdict.

Source code in src/calibrated_agent_evaluation/enforce/errors.py
def __init__(self, message: str, *, decision: EnforcementDecision | None = None) -> None:
    super().__init__(message)
    self.decision = decision

EnforcementError

EnforcementError(message, *, decision=None)

Bases: CalibratedAgentEvaluationDataError

Base class for every enforcement-layer refusal.

Carries the optional EnforcementDecision that recorded the refusal, so a hard refusal both raises (control flow) and records (the audit trail) -- a caller catching the error reads err.decision for the audit record.

Source code in src/calibrated_agent_evaluation/enforce/errors.py
def __init__(self, message: str, *, decision: EnforcementDecision | None = None) -> None:
    super().__init__(message)
    self.decision = decision

decision instance-attribute

decision = decision

EnforcementStoreError

EnforcementStoreError(message, *, decision=None)

Bases: EnforcementError

A decision-store file read or write failed.

Wraps the underlying OSError / json.JSONDecodeError so a filesystem/parse failure surfaces inside the typed family rather than as a raw stdlib exception escaping the store's public surface ("wrap external errors at the boundary").

Source code in src/calibrated_agent_evaluation/enforce/errors.py
def __init__(self, message: str, *, decision: EnforcementDecision | None = None) -> None:
    super().__init__(message)
    self.decision = decision

GenericMetricInCIError

GenericMetricInCIError(message, *, decision=None)

Bases: EnforcementError

A generic generation-quality metric was used as a CI gate.

ROUGE/BERTScore/generic-"helpfulness" are exploration-only; refused while the context is in CI mode.

Source code in src/calibrated_agent_evaluation/enforce/errors.py
def __init__(self, message: str, *, decision: EnforcementDecision | None = None) -> None:
    super().__init__(message)
    self.decision = decision

LikertNotAllowedError

LikertNotAllowedError(message, *, decision=None)

Bases: EnforcementError

A Likert-scaled axis appears while the context's allow_likert capability is disabled.

Likert is available only behind the explicit opt-in, which is disabled by default.

Source code in src/calibrated_agent_evaluation/enforce/errors.py
def __init__(self, message: str, *, decision: EnforcementDecision | None = None) -> None:
    super().__init__(message)
    self.decision = decision

MultiAxisLikertError

MultiAxisLikertError(message, *, decision=None)

Bases: EnforcementError

A judge definition scores more than one non-binary (Likert) axis in a single call.

The specifically-attested-harmful pattern -- a hard lint-time refusal.

Source code in src/calibrated_agent_evaluation/enforce/errors.py
def __init__(self, message: str, *, decision: EnforcementDecision | None = None) -> None:
    super().__init__(message)
    self.decision = decision

UncalibratedScoreError

UncalibratedScoreError(message, *, decision=None)

Bases: EnforcementError

A score or corrected rate was requested for a judge with no certifying calibration.

The calibration verdict is read from the calibration record (certifies); a non-CERTIFIED record, or a record whose pinned (model, prompt_version) no longer matches the judge, refuses the score.

Source code in src/calibrated_agent_evaluation/enforce/errors.py
def __init__(self, message: str, *, decision: EnforcementDecision | None = None) -> None:
    super().__init__(message)
    self.decision = decision

WeightedCollapseError

WeightedCollapseError(message, *, decision=None)

Bases: EnforcementError

A checklist judge declares a weighted-threshold collapse rule.

Only critical/non-critical tagging is offered; tunable weights reintroduce the vibe-scoring the binary rule exists to kill.

Source code in src/calibrated_agent_evaluation/enforce/errors.py
def __init__(self, message: str, *, decision: EnforcementDecision | None = None) -> None:
    super().__init__(message)
    self.decision = decision

Axis pydantic-model

Bases: CalibratedAgentEvaluationModel

One scoring axis of a judge. scale defaults to binary.

Config:

  • frozen: True
  • extra: forbid

Fields:

name pydantic-field

name

scale pydantic-field

scale = 'binary'

ChecklistItem pydantic-model

Bases: CalibratedAgentEvaluationModel

One checklist item, tagged critical or non-critical at authoring time.

A critical item's "no" collapses the whole judge to FAIL; a non-critical item's "no" is tracked as a non-gating quality sub-metric.

Config:

  • frozen: True
  • extra: forbid

Fields:

name pydantic-field

name

critical pydantic-field

critical

JudgeDefinition pydantic-model

Bases: CalibratedAgentEvaluationModel

An authored judge definition -- the lint target.

output_fields is the ordered field list of the judge's forced-structured output; the critique-first discipline requires a non-empty "critique" ordered before "verdict" (checked by the linter). scorer names the scoring mechanism (e.g. "rouge", "exact_match"); lint classifies it as generic-generation vs structured-target for the CI ban. checklist items carry critical tags; weighted_collapse is refused at lint.

Config:

  • frozen: True
  • extra: forbid

Fields:

judge_id pydantic-field

judge_id

axes pydantic-field

axes

output_fields pydantic-field

output_fields

scorer pydantic-field

scorer = None

checklist pydantic-field

checklist = ()

weighted_collapse pydantic-field

weighted_collapse = False

schema_version pydantic-field

schema_version = ENFORCE_SCHEMA_VERSION

AgreementDisplay dataclass

AgreementDisplay(
    scalar_name,
    value,
    raw_agreement_hidden,
    confusion_cells=None,
    undefined_reason=None,
)

What to surface for an agreement computation.

Under imbalance: scalar_name == "cohen_kappa" with confusion_cells populated and raw_agreement_hidden true. Otherwise: scalar_name == "raw_agreement" with the raw fraction and no cells. value is the scalar with its 95% interval, or None with undefined_reason set when κ is 0/0.

scalar_name instance-attribute

scalar_name

value instance-attribute

value

raw_agreement_hidden instance-attribute

raw_agreement_hidden

confusion_cells class-attribute instance-attribute

confusion_cells = None

undefined_reason class-attribute instance-attribute

undefined_reason = None

collapse_checklist

collapse_checklist(defn, item_verdicts)

Collapse per-item verdicts to one overall outcome.

item_verdicts maps each checklist item name to its PASS/FAIL verdict. Any critical item at FAIL -> overall FAIL; otherwise overall PASS. Non-critical items are returned as sub-metrics and never affect overall. An item named in the definition but absent from item_verdicts is treated as unresolved and, if critical, fails closed (a missing critical verdict cannot certify a PASS).

Source code in src/calibrated_agent_evaluation/enforce/checklist.py
def collapse_checklist(
    defn: JudgeDefinition, item_verdicts: Mapping[str, Outcome]
) -> ChecklistResult:
    """Collapse per-item verdicts to one overall outcome.

    ``item_verdicts`` maps each checklist item name to its ``PASS``/``FAIL`` verdict. Any critical
    item at ``FAIL`` -> overall ``FAIL``; otherwise overall ``PASS``. Non-critical items are
    returned as sub-metrics and never affect ``overall``. An item named in the definition but
    absent from ``item_verdicts`` is treated as unresolved and, if critical, fails closed (a
    missing critical verdict cannot certify a PASS).
    """
    overall: Outcome = "PASS"
    non_gating: dict[str, Outcome] = {}
    for item in defn.checklist:
        verdict = item_verdicts.get(item.name)
        if item.critical:
            # Fail closed: a critical item that is FAIL or unresolved collapses to FAIL.
            if verdict != "PASS":
                overall = "FAIL"
        elif verdict is not None:
            non_gating[item.name] = verdict
    return ChecklistResult(overall=overall, non_gating=non_gating)

enforce_score_gate

enforce_score_gate(record, judge)

Refuse a score for an uncertified judge.

Returns cleanly when certifies(record, judge) holds (a certified record whose pinned (model, prompt_version) still matches the judge). Otherwise records a hard_refusal EnforcementDecision and raises UncalibratedScoreError carrying it -- no score is computed or displayed.

Source code in src/calibrated_agent_evaluation/enforce/gate.py
def enforce_score_gate(record: CalibrationRecord, judge: Judge) -> None:
    """Refuse a score for an uncertified judge.

    Returns cleanly when ``certifies(record, judge)`` holds (a certified record whose pinned
    ``(model, prompt_version)`` still matches the judge). Otherwise records a ``hard_refusal``
    :class:`EnforcementDecision` and raises :class:`UncalibratedScoreError` carrying it -- no score
    is computed or displayed.
    """
    # Imported here: the calibration engine needs the [stats] extra, and the warning below must not.
    from calibrated_agent_evaluation.calibrate import certifies
    from calibrated_agent_evaluation.calibrate.engine import pin_mismatch

    if certifies(record, judge):
        return
    decision = EnforcementDecision(
        rule="ENF-1",
        tier="hard_refusal",
        inputs={
            "judge_id": judge.judge_id,
            "model": judge.model,
            "prompt_version": judge.prompt_version,
            "record_result": record.result,
            "record_model": record.model,
            "record_prompt_version": record.prompt_version,
        },
        outcome="refused",
        message=(
            f"judge {judge.judge_id!r} has no certifying calibration, so no score is computed: "
            + (
                pin_mismatch(record, judge)
                or f"its calibration record {record.calib_id!r} is {record.result}; calibrate the "
                "judge against gold labels first"
            )
        ),
    )
    raise UncalibratedScoreError(decision.message, decision=decision)

warn_error_analysis_first

warn_error_analysis_first(judge_id, human_label_count)

Warn that error analysis precedes scoring when no human labels exist yet.

Returns a warning decision when human_label_count == 0 (observe -> open-code -> cluster -> evaluate comes before pick-metric -> score); returns None when labels already exist.

Source code in src/calibrated_agent_evaluation/enforce/gate.py
def warn_error_analysis_first(judge_id: str, human_label_count: int) -> EnforcementDecision | None:
    """Warn that error analysis precedes scoring when no human labels exist yet.

    Returns a ``warning`` decision when ``human_label_count == 0`` (observe -> open-code -> cluster
    -> evaluate comes before pick-metric -> score); returns ``None`` when labels already exist.
    """
    if human_label_count > 0:
        return None
    return EnforcementDecision(
        rule="PRIN-3",
        tier="warning",
        inputs={"judge_id": judge_id, "human_label_count": str(human_label_count)},
        outcome="warned",
        message=(
            f"scoring judge {judge_id!r} before any human labels exist: error analysis "
            "(observe -> open-code -> cluster -> evaluate) precedes scoring."
        ),
        nudge="Run error analysis and record human labels before requesting a score.",
    )

critique_before_verdict

critique_before_verdict(defn)

Return whether output_fields contains critique strictly before verdict.

The pure predicate behind the critique-first refusal: a definition missing critique, missing verdict, or ordering verdict first fails it. The linter turns a False here into a typed CritiqueOrderError.

Source code in src/calibrated_agent_evaluation/enforce/judgedef.py
def critique_before_verdict(defn: JudgeDefinition) -> bool:
    """Return whether ``output_fields`` contains ``critique`` strictly before ``verdict``.

    The pure predicate behind the critique-first refusal: a definition missing ``critique``, missing
    ``verdict``, or ordering ``verdict`` first fails it. The linter turns a ``False`` here into a
    typed :class:`~calibrated_agent_evaluation.enforce.errors.CritiqueOrderError`.
    """
    fields = defn.output_fields
    if "critique" not in fields or "verdict" not in fields:
        return False
    return fields.index("critique") < fields.index("verdict")

likert_axes

likert_axes(defn)

Return the non-binary (Likert) axes of defn.

Source code in src/calibrated_agent_evaluation/enforce/judgedef.py
def likert_axes(defn: JudgeDefinition) -> tuple[Axis, ...]:
    """Return the non-binary (Likert) axes of ``defn``."""
    return tuple(ax for ax in defn.axes if ax.scale == "likert")

lint_judge_definition

lint_judge_definition(defn, ctx)

Lint defn under ctx; return warning/display decisions or raise on a hard refusal.

Hard refusals (multi-Likert, Likert-not-allowed, critique-order, weighted-collapse, generic-metric-in-CI) raise a typed error carrying the recorded decision. Non-fatal outcomes (Likert-allowed, exploration-only, multi-criteria nudge) are returned as EnforcementDecisions in evaluation order.

Source code in src/calibrated_agent_evaluation/enforce/lint.py
def lint_judge_definition(
    defn: JudgeDefinition, ctx: EnforcementContext
) -> list[EnforcementDecision]:
    """Lint ``defn`` under ``ctx``; return warning/display decisions or raise on a hard refusal.

    Hard refusals (multi-Likert, Likert-not-allowed, critique-order, weighted-collapse,
    generic-metric-in-CI) raise a typed error carrying the recorded decision. Non-fatal outcomes
    (Likert-allowed, exploration-only, multi-criteria nudge) are returned as
    :class:`EnforcementDecision`s in evaluation order.
    """
    decisions: list[EnforcementDecision] = []

    # --- Likert axes: multiple axes are always invalid; a single axis is opt-in ---------------
    likert = likert_axes(defn)
    if len(likert) > 1:
        _refuse(
            MultiAxisLikertError,
            "ENF-2",
            {"judge_id": defn.judge_id, "likert_axes": ",".join(ax.name for ax in likert)},
            f"judge {defn.judge_id!r} scores {len(likert)} non-binary (Likert) axes in one call "
            f"({', '.join(ax.name for ax in likert)}); this is refused at lint time. "
            "Decompose into N single-axis binary judges or a single-call binary checklist.",
        )
    if len(likert) == 1 and not ctx.allow_likert:
        _refuse(
            LikertNotAllowedError,
            "ENF-6",
            {"judge_id": defn.judge_id, "likert_axis": likert[0].name},
            f"judge {defn.judge_id!r} scores {likert[0].name!r} on a Likert (non-binary) scale; "
            "Likert is refused unless the EnforcementContext sets allow_likert. Rewrite the axis "
            "as one binary PASS/FAIL question with a critique.",
        )

    # --- Critique-first --------------------------------------------------------
    if not critique_before_verdict(defn):
        _refuse(
            CritiqueOrderError,
            "PRIN-4",
            {"judge_id": defn.judge_id, "output_fields": ",".join(defn.output_fields)},
            f"judge {defn.judge_id!r} output schema must place a non-empty 'critique' before "
            f"'verdict'; got fields {list(defn.output_fields)} (critique-first).",
        )

    # --- Checklist weighted-threshold collapse is refused -----------------------------
    if defn.weighted_collapse:
        _refuse(
            WeightedCollapseError,
            "ENF-4",
            {"judge_id": defn.judge_id, "weighted_collapse": "true"},
            f"judge {defn.judge_id!r} declares a weighted-threshold collapse rule; only "
            "critical/non-critical tagging is offered -- tunable weights reintroduce vibe-scoring. "
            "Tag each item critical or non-critical instead.",
        )

    # --- Generic generation-quality scorer: banned in CI, exploration-only otherwise --
    if defn.scorer is not None and defn.scorer.lower() in GENERIC_GENERATION_METRICS:
        if ctx.ci:
            _refuse(
                GenericMetricInCIError,
                "ENF-5",
                {"judge_id": defn.judge_id, "scorer": defn.scorer, "ci": "true"},
                f"judge {defn.judge_id!r} uses the generic generation-quality metric "
                f"{defn.scorer!r} as a CI gate; generic metrics are banned in --ci and labeled "
                "exploration-only. Use a domain-specific or structured-target scorer.",
            )
        decisions.append(
            EnforcementDecision(
                rule="ENF-5",
                tier="display_default",
                inputs={"judge_id": defn.judge_id, "scorer": defn.scorer, "ci": "false"},
                outcome="shaped",
                message=(
                    f"scorer {defn.scorer!r} is a generic generation-quality metric: "
                    "exploration-only, not a CI gate."
                ),
            )
        )

    # --- Single Likert axis, allowed with a warning (display default) --------------------
    if len(likert) == 1 and ctx.allow_likert:
        decisions.append(
            EnforcementDecision(
                rule="ENF-6",
                tier="display_default",
                inputs={"judge_id": defn.judge_id, "likert_axis": likert[0].name},
                outcome="warned",
                message=(
                    f"judge {defn.judge_id!r} uses a Likert axis {likert[0].name!r} behind "
                    "allow_likert; 1-5 scoring is noisy -- prefer binary."
                ),
                nudge="Consider a binary axis or a single-call binary checklist instead.",
            )
        )

    # --- General multi-criteria (>1 binary axis): warn + nudge, never refuse ----------
    binary_axes = tuple(ax for ax in defn.axes if ax.scale == "binary")
    if len(binary_axes) > 1:
        decisions.append(
            EnforcementDecision(
                rule="ENF-3",
                tier="warning",
                inputs={
                    "judge_id": defn.judge_id,
                    "binary_axes": ",".join(ax.name for ax in binary_axes),
                },
                outcome="warned",
                message=(
                    f"judge {defn.judge_id!r} scores {len(binary_axes)} criteria in one call; "
                    "general multi-criteria judging is allowed but directional."
                ),
                nudge=(
                    "Prefer N single-axis judges or a single-call binary checklist "
                    "(each item binary, tagged critical/non-critical)."
                ),
            )
        )

    return decisions

lint_scorer_lanes

lint_scorer_lanes(scorers, ctx)

Refuse, in CI, a gate over verdicts that a generic generation-quality metric produced.

scorers names the scorers whose verdicts a run would gate (the scorer name in each recorded lane). In CI a generic metric among them raises GenericMetricInCIError carrying the decision; outside CI each one is returned as an exploration-only display_default decision. Structured-target scorers pass with no decision.

Source code in src/calibrated_agent_evaluation/enforce/lint.py
def lint_scorer_lanes(scorers: Sequence[str], ctx: EnforcementContext) -> list[EnforcementDecision]:
    """Refuse, in CI, a gate over verdicts that a generic generation-quality metric produced.

    ``scorers`` names the scorers whose verdicts a run would gate (the scorer name in each
    recorded lane). In CI a generic metric among them raises :class:`GenericMetricInCIError`
    carrying the decision; outside CI each one is returned as an exploration-only
    ``display_default`` decision. Structured-target scorers pass with no decision.
    """
    generic = sorted({s for s in scorers if s.lower() in GENERIC_GENERATION_METRICS})
    if not generic:
        return []
    names = ", ".join(repr(s) for s in generic)
    if ctx.ci:
        _refuse(
            GenericMetricInCIError,
            "ENF-5",
            {"scorers": ",".join(generic), "ci": "true"},
            f"the scorer verdicts include ones from {names}, a generic generation-quality "
            "metric; such metrics track human judgment poorly on open-ended output, so they "
            "cannot gate CI. Gate on a structured-target scorer (exact match, a JSON or numeric "
            "comparison) or a calibrated judge, and keep the generic metric for exploration.",
        )
    return [
        EnforcementDecision(
            rule="ENF-5",
            tier="display_default",
            inputs={"scorer": name, "ci": "false"},
            outcome="shaped",
            message=(
                f"scorer {name!r} is a generic generation-quality metric: exploration-only, it "
                "cannot gate CI."
            ),
        )
        for name in generic
    ]

dump_decisions

dump_decisions(decisions, path, fmt='csv')

Write decisions to enforcement_decision.{csv,json} under path.

Parameters:

Name Type Description Default
decisions Sequence[EnforcementDecision]

The decisions to persist.

required
path str | Path

The directory to write into.

required
fmt str

Which file a later load_decisions reads back (both are always written).

'csv'

Raises:

Type Description
EnforcementStoreError

A bad fmt or any filesystem failure.

Source code in src/calibrated_agent_evaluation/enforce/store.py
def dump_decisions(
    decisions: Sequence[EnforcementDecision], path: str | Path, fmt: str = "csv"
) -> None:
    """Write ``decisions`` to ``enforcement_decision.{csv,json}`` under ``path``.

    Args:
        decisions: The decisions to persist.
        path: The directory to write into.
        fmt: Which file a later :func:`load_decisions` reads back (both are always written).

    Raises:
        EnforcementStoreError: A bad ``fmt`` or any filesystem failure.
    """
    _DECISIONS.dump(decisions, path, fmt)

dump_definitions

dump_definitions(definitions, path, fmt='csv')

Write definitions to judge_definition.{csv,json} under path.

Parameters:

Name Type Description Default
definitions Sequence[JudgeDefinition]

The judge definitions to persist.

required
path str | Path

The directory to write into.

required
fmt str

Which file a later load_definitions reads back (both are always written).

'csv'

Raises:

Type Description
EnforcementStoreError

A bad fmt or any filesystem failure.

Source code in src/calibrated_agent_evaluation/enforce/store.py
def dump_definitions(
    definitions: Sequence[JudgeDefinition], path: str | Path, fmt: str = "csv"
) -> None:
    """Write ``definitions`` to ``judge_definition.{csv,json}`` under ``path``.

    Args:
        definitions: The judge definitions to persist.
        path: The directory to write into.
        fmt: Which file a later :func:`load_definitions` reads back (both are always written).

    Raises:
        EnforcementStoreError: A bad ``fmt`` or any filesystem failure.
    """
    _DEFINITIONS.dump(definitions, path, fmt)

load_decisions

load_decisions(path, fmt='csv')

Read enforcement decisions back from path; a missing file is an empty store.

Parameters:

Name Type Description Default
path str | Path

The directory dump_decisions wrote.

required
fmt str

"csv" or "json".

'csv'

Returns:

Type Description
list[EnforcementDecision]

The stored decisions.

Raises:

Type Description
EnforcementStoreError

A schema-version mismatch, or any read, parse or validation failure.

Source code in src/calibrated_agent_evaluation/enforce/store.py
def load_decisions(path: str | Path, fmt: str = "csv") -> list[EnforcementDecision]:
    """Read enforcement decisions back from ``path``; a missing file is an empty store.

    Args:
        path: The directory :func:`dump_decisions` wrote.
        fmt: ``"csv"`` or ``"json"``.

    Returns:
        The stored decisions.

    Raises:
        EnforcementStoreError: A schema-version mismatch, or any read, parse or validation failure.
    """
    return _DECISIONS.load(path, fmt)

load_definitions

load_definitions(path, fmt='csv')

Read judge definitions back from path; a missing file is an empty store.

Parameters:

Name Type Description Default
path str | Path

The directory dump_definitions wrote.

required
fmt str

"csv" or "json".

'csv'

Returns:

Type Description
list[JudgeDefinition]

The stored definitions.

Raises:

Type Description
EnforcementStoreError

A schema-version mismatch, or any read, parse or validation failure.

Source code in src/calibrated_agent_evaluation/enforce/store.py
def load_definitions(path: str | Path, fmt: str = "csv") -> list[JudgeDefinition]:
    """Read judge definitions back from ``path``; a missing file is an empty store.

    Args:
        path: The directory :func:`dump_definitions` wrote.
        fmt: ``"csv"`` or ``"json"``.

    Returns:
        The stored definitions.

    Raises:
        EnforcementStoreError: A schema-version mismatch, or any read, parse or validation failure.
    """
    return _DEFINITIONS.load(path, fmt)

agreement_display

agreement_display(gold, verdict)

Select the agreement scalar to surface, hiding raw agreement under imbalance.

When the gold labels are class-imbalanced (per should_hide_raw_agreement), returns Cohen's κ with confusion cells and raw_agreement_hidden=True; otherwise returns the raw percent-agreement scalar.

Source code in src/calibrated_agent_evaluation/enforce/display.py
def agreement_display(gold: Sequence[object], verdict: Sequence[object]) -> AgreementDisplay:
    """Select the agreement scalar to surface, hiding raw agreement under imbalance.

    When the gold labels are class-imbalanced (per ``should_hide_raw_agreement``), returns
    Cohen's κ with confusion cells and ``raw_agreement_hidden=True``; otherwise returns the raw
    percent-agreement scalar.
    """
    if should_hide_raw_agreement(gold):
        cells = _confusion_cells(gold, verdict)
        try:
            kappa = cohen_kappa_interval(gold, verdict)
        except UndefinedAgreementError as exc:
            return AgreementDisplay(
                scalar_name="cohen_kappa",
                value=None,
                raw_agreement_hidden=True,
                confusion_cells=cells,
                undefined_reason=str(exc),
            )
        return AgreementDisplay(
            scalar_name="cohen_kappa",
            value=kappa,
            raw_agreement_hidden=True,
            confusion_cells=cells,
        )
    return AgreementDisplay(
        scalar_name="raw_agreement",
        value=_raw_agreement(gold, verdict),
        raw_agreement_hidden=False,
    )