calibrated_agent_evaluation.enforce¶
calibrated_agent_evaluation.enforce ¶
The tiered enforcement / lint-policy layer -- "enforcement, not enablement".
The methodology-plane surface that makes the right habit the default and the wrong habit the hard
path, tiering each intervention to the strength of the evidence: hard refusals only
where a pattern is provably invalid, warnings + nudges where it is merely
directional, and display defaults that shape output. It reads --
never recomputes -- the calibration verdict to gate scoring, and reuses the shared
cohen_kappa for the imbalance display.
Pure methodology plane: no vendor SDK type appears here, and no new runtime dependency is
introduced. Every decision is captured as an auditable
EnforcementDecision. It does NOT run judges,
parse CLI flags, or compute the calibration statistics.
GENERIC_GENERATION_METRICS
module-attribute
¶
ChecklistResult
dataclass
¶
EnforcementContext
dataclass
¶
The mode/capabilities a lint or gate call runs under.
ci -- while true, generic generation-quality metrics are hard-refused.
allow_likert -- while true, a single Likert axis is permitted with a warning;
disabled by default.
EnforcementDecision
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
One recorded enforcement decision.
rule is the governing id (e.g. "", ""); tier is exactly one of
Tier; inputs are the evaluated inputs (non-empty); outcome is
"refused"/"warned"/"shaped"; message is the human-facing explanation; nudge
is the structural steer where applicable.
Config:
frozen:Trueextra:forbid
Fields:
-
rule(str) -
tier(Tier) -
inputs(dict[str, str]) -
outcome(str) -
message(str) -
nudge(str | None) -
schema_version(str)
Validators:
-
_check_invariants
CritiqueOrderError ¶
Bases: EnforcementError
A judge output schema omits critique or orders verdict before it.
The binary/critique-first discipline is structural; critique precedes verdict.
Source code in src/calibrated_agent_evaluation/enforce/errors.py
EnforcementError ¶
Bases: CalibratedAgentEvaluationDataError
Base class for every enforcement-layer refusal.
Carries the optional EnforcementDecision
that recorded the refusal, so a hard refusal both raises (control flow) and records (the
audit trail) -- a caller catching the error reads err.decision for the audit record.
Source code in src/calibrated_agent_evaluation/enforce/errors.py
EnforcementStoreError ¶
Bases: EnforcementError
A decision-store file read or write failed.
Wraps the underlying OSError / json.JSONDecodeError so a filesystem/parse failure
surfaces inside the typed family rather than as a raw stdlib exception escaping the store's
public surface ("wrap external errors at the boundary").
Source code in src/calibrated_agent_evaluation/enforce/errors.py
GenericMetricInCIError ¶
Bases: EnforcementError
A generic generation-quality metric was used as a CI gate.
ROUGE/BERTScore/generic-"helpfulness" are exploration-only; refused while the context is in CI mode.
Source code in src/calibrated_agent_evaluation/enforce/errors.py
LikertNotAllowedError ¶
Bases: EnforcementError
A Likert-scaled axis appears while the context's allow_likert capability is disabled.
Likert is available only behind the explicit opt-in, which is disabled by default.
Source code in src/calibrated_agent_evaluation/enforce/errors.py
MultiAxisLikertError ¶
Bases: EnforcementError
A judge definition scores more than one non-binary (Likert) axis in a single call.
The specifically-attested-harmful pattern -- a hard lint-time refusal.
Source code in src/calibrated_agent_evaluation/enforce/errors.py
UncalibratedScoreError ¶
Bases: EnforcementError
A score or corrected rate was requested for a judge with no certifying calibration.
The calibration verdict is read from the calibration record (certifies); a non-CERTIFIED
record, or a record whose pinned (model, prompt_version) no longer matches the judge,
refuses the score.
Source code in src/calibrated_agent_evaluation/enforce/errors.py
WeightedCollapseError ¶
Bases: EnforcementError
A checklist judge declares a weighted-threshold collapse rule.
Only critical/non-critical tagging is offered; tunable weights reintroduce the vibe-scoring the binary rule exists to kill.
Source code in src/calibrated_agent_evaluation/enforce/errors.py
Axis
pydantic-model
¶
ChecklistItem
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
One checklist item, tagged critical or non-critical at authoring time.
A critical item's "no" collapses the whole judge to FAIL; a non-critical item's "no" is
tracked as a non-gating quality sub-metric.
Config:
frozen:Trueextra:forbid
Fields:
JudgeDefinition
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
An authored judge definition -- the lint target.
output_fields is the ordered field list of the judge's forced-structured output; the
critique-first discipline requires a non-empty "critique" ordered before "verdict"
(checked by the linter). scorer names the scoring mechanism (e.g. "rouge",
"exact_match"); lint classifies it as
generic-generation vs structured-target for the CI ban. checklist items carry critical tags;
weighted_collapse is refused at lint.
Config:
frozen:Trueextra:forbid
Fields:
-
judge_id(str) -
axes(tuple[Axis, ...]) -
output_fields(tuple[str, ...]) -
scorer(str | None) -
checklist(tuple[ChecklistItem, ...]) -
weighted_collapse(bool) -
schema_version(str)
AgreementDisplay
dataclass
¶
AgreementDisplay(
scalar_name,
value,
raw_agreement_hidden,
confusion_cells=None,
undefined_reason=None,
)
What to surface for an agreement computation.
Under imbalance: scalar_name == "cohen_kappa" with confusion_cells populated and
raw_agreement_hidden true. Otherwise: scalar_name == "raw_agreement" with the raw
fraction and no cells. value is the scalar with its 95% interval, or None with
undefined_reason set when κ is 0/0.
collapse_checklist ¶
Collapse per-item verdicts to one overall outcome.
item_verdicts maps each checklist item name to its PASS/FAIL verdict. Any critical
item at FAIL -> overall FAIL; otherwise overall PASS. Non-critical items are
returned as sub-metrics and never affect overall. An item named in the definition but
absent from item_verdicts is treated as unresolved and, if critical, fails closed (a
missing critical verdict cannot certify a PASS).
Source code in src/calibrated_agent_evaluation/enforce/checklist.py
enforce_score_gate ¶
Refuse a score for an uncertified judge.
Returns cleanly when certifies(record, judge) holds (a certified record whose pinned
(model, prompt_version) still matches the judge). Otherwise records a hard_refusal
EnforcementDecision and raises UncalibratedScoreError carrying it -- no score
is computed or displayed.
Source code in src/calibrated_agent_evaluation/enforce/gate.py
warn_error_analysis_first ¶
Warn that error analysis precedes scoring when no human labels exist yet.
Returns a warning decision when human_label_count == 0 (observe -> open-code -> cluster
-> evaluate comes before pick-metric -> score); returns None when labels already exist.
Source code in src/calibrated_agent_evaluation/enforce/gate.py
critique_before_verdict ¶
Return whether output_fields contains critique strictly before verdict.
The pure predicate behind the critique-first refusal: a definition missing critique, missing
verdict, or ordering verdict first fails it. The linter turns a False here into a
typed CritiqueOrderError.
Source code in src/calibrated_agent_evaluation/enforce/judgedef.py
likert_axes ¶
lint_judge_definition ¶
Lint defn under ctx; return warning/display decisions or raise on a hard refusal.
Hard refusals (multi-Likert, Likert-not-allowed, critique-order, weighted-collapse,
generic-metric-in-CI) raise a typed error carrying the recorded decision. Non-fatal outcomes
(Likert-allowed, exploration-only, multi-criteria nudge) are returned as
EnforcementDecisions in evaluation order.
Source code in src/calibrated_agent_evaluation/enforce/lint.py
60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 | |
lint_scorer_lanes ¶
Refuse, in CI, a gate over verdicts that a generic generation-quality metric produced.
scorers names the scorers whose verdicts a run would gate (the scorer name in each
recorded lane). In CI a generic metric among them raises GenericMetricInCIError
carrying the decision; outside CI each one is returned as an exploration-only
display_default decision. Structured-target scorers pass with no decision.
Source code in src/calibrated_agent_evaluation/enforce/lint.py
dump_decisions ¶
Write decisions to enforcement_decision.{csv,json} under path.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
decisions
|
Sequence[EnforcementDecision]
|
The decisions to persist. |
required |
path
|
str | Path
|
The directory to write into. |
required |
fmt
|
str
|
Which file a later |
'csv'
|
Raises:
| Type | Description |
|---|---|
EnforcementStoreError
|
A bad |
Source code in src/calibrated_agent_evaluation/enforce/store.py
dump_definitions ¶
Write definitions to judge_definition.{csv,json} under path.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
definitions
|
Sequence[JudgeDefinition]
|
The judge definitions to persist. |
required |
path
|
str | Path
|
The directory to write into. |
required |
fmt
|
str
|
Which file a later |
'csv'
|
Raises:
| Type | Description |
|---|---|
EnforcementStoreError
|
A bad |
Source code in src/calibrated_agent_evaluation/enforce/store.py
load_decisions ¶
Read enforcement decisions back from path; a missing file is an empty store.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str | Path
|
The directory |
required |
fmt
|
str
|
|
'csv'
|
Returns:
| Type | Description |
|---|---|
list[EnforcementDecision]
|
The stored decisions. |
Raises:
| Type | Description |
|---|---|
EnforcementStoreError
|
A schema-version mismatch, or any read, parse or validation failure. |
Source code in src/calibrated_agent_evaluation/enforce/store.py
load_definitions ¶
Read judge definitions back from path; a missing file is an empty store.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str | Path
|
The directory |
required |
fmt
|
str
|
|
'csv'
|
Returns:
| Type | Description |
|---|---|
list[JudgeDefinition]
|
The stored definitions. |
Raises:
| Type | Description |
|---|---|
EnforcementStoreError
|
A schema-version mismatch, or any read, parse or validation failure. |
Source code in src/calibrated_agent_evaluation/enforce/store.py
agreement_display ¶
Select the agreement scalar to surface, hiding raw agreement under imbalance.
When the gold labels are class-imbalanced (per should_hide_raw_agreement), returns
Cohen's κ with confusion cells and raw_agreement_hidden=True; otherwise returns the raw
percent-agreement scalar.