calibrated_agent_evaluation.cluster¶
calibrated_agent_evaluation.cluster ¶
Clustering, saturation & coverage engine.
Public API re-exports; import from calibrated_agent_evaluation.cluster.
ClusteringError ¶
Bases: CalibratedAgentEvaluationDataError
Invalid input or state in the clustering pipeline.
UnratifiedCodebookError ¶
Bases: ClusteringError
Attempt to use a draft codebook without human ratification.
Codebook
dataclass
¶
A versioned, human-ratified catalog of failure modes.
Only a Codebook with ratified_by set may tag Labels.
CoverageResult
dataclass
¶
CoverageResult(
cell_counts=dict(),
blind_spots=(),
total_cells=0,
covered_cells=0,
unclassified=(),
)
Output of coverage_report(): surfaces taxonomy blind spots.
unclassified are observed FAIL cells whose taxonomy_cell does not match any grid cell
(an unknown/extra dimension value). They are surfaced, not silently dropped; a genuinely
observed failure that falls outside the declared grid is a signal (the grid is incomplete or
the trace is mis-tagged), not something to under-count into a phantom blind spot.
DraftCluster
dataclass
¶
DraftCodebook
dataclass
¶
FailureMode
dataclass
¶
SaturationResult
dataclass
¶
SaturationResult(
saturated,
coverage,
total_failures,
total_modes,
singletons,
reason=None,
chao1=None,
)
Output of saturation_check(): answers 'should I keep labeling?'.
coverage is the Good-Turing coverage with its one-sided 95% Esty lower bound (the gate
reads coverage.lb95). chao1 is the Chao1 richness estimate with its two-sided 95%
log-transformed interval, None when nothing was observed.
ratify ¶
ratify(
draft,
*,
ratified_by,
merges=None,
splits=None,
renames=None,
version="1",
parent_version=None,
descriptions=None,
)
Transform a DraftCodebook into a ratified Codebook.
Raises ClusteringError if version <= parent_version (monotonicity).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
draft
|
DraftCodebook
|
The draft codebook to ratify. |
required |
ratified_by
|
str
|
Who ratified it. |
required |
merges
|
dict[str, list[str]] | None
|
New mode name to the draft modes merged into it. |
None
|
splits
|
dict[str, list[list[int]]] | None
|
Draft mode to the example index groups it splits into. |
None
|
renames
|
dict[str, str] | None
|
Draft mode name to its new name. |
None
|
version
|
str
|
The ratified codebook's version. |
'1'
|
parent_version
|
str | None
|
The version this one replaces. |
None
|
descriptions
|
Mapping[str, str] | None
|
Mode name to its definition, applied after merges and renames: a mode
whose final name is a key gets that value as its description, and every other mode
keeps the one it has (a rename keeps the old mode's). Keys that name no mode are
unused, so a whole vocabulary can be passed. |
None
|
Returns:
| Type | Description |
|---|---|
Codebook
|
The ratified codebook. |
Raises:
| Type | Description |
|---|---|
ClusteringError
|
|
Source code in src/calibrated_agent_evaluation/cluster/codebook.py
18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 | |
tag_label ¶
Tag a Label with a failure mode from a ratified Codebook.
Raises UnratifiedCodebookError if codebook is a DraftCodebook. Raises ClusteringError if mode is not in the codebook.
Source code in src/calibrated_agent_evaluation/cluster/codebook.py
coverage_report ¶
Report per-cell observation counts and blind spots.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
grid
|
TaxonomyGrid
|
The taxonomy dimensions and their values. |
required |
observations
|
list[dict[str, str]]
|
The taxonomy cell of each FAIL-labeled trace. |
required |
Returns:
| Type | Description |
|---|---|
CoverageResult
|
The count per cell, the cells with no failures, and observations outside the grid. |
Source code in src/calibrated_agent_evaluation/cluster/coverage.py
build_draft ¶
Assemble a DraftCodebook from cluster assignments.
Groups critiques by assignment label, optionally asks the namer for names.
Raises:
| Type | Description |
|---|---|
ClusteringError
|
The namer returns a different number of names than there are clusters, or returns any name more than once. |
Source code in src/calibrated_agent_evaluation/cluster/draft.py
hierarchical_cluster ¶
Cluster embeddings via Ward's linkage + distance-based cut.
Returns a list of cluster assignments (0-indexed integers), one per embedding.
Requires the [stats] extra (scipy).
Source code in src/calibrated_agent_evaluation/cluster/group.py
chao1_estimate ¶
Chao1 lower-bound species-richness estimator (Chao 1984).
Returns (estimated_total_modes, modes_found, fraction_discovered). Zero-count modes are not
observed modes and do not enter modes_found.
Source code in src/calibrated_agent_evaluation/cluster/saturation.py
chao1_interval ¶
Chao1 with Chao's (1987) log-transformed two-sided 95% interval.
With T = Chao1 - S_obs and K = exp(z sqrt(log(1 + Var/T^2))) the interval is
[S_obs + T/K, S_obs + T*K] (eq. 13), so it never falls below S_obs. The variance is
eq. 5 (f2 > 0) or eq. 7 (f2 = 0). When T = 0 (f1 = 0, or f1 = 1 with
f2 = 0) the log transform is undefined and eq. 14 is used instead, built on Var(S_obs).
Raises ClusteringError when nothing was observed.
Source code in src/calibrated_agent_evaluation/cluster/saturation.py
good_turing_coverage ¶
Good-Turing coverage point estimate: 1 - singletons/total.
Returns a float in [0.0, 1.0], and 0.0 when nothing was observed. Use
good_turing_coverage_bound for the number a gate may read.
Source code in src/calibrated_agent_evaluation/cluster/saturation.py
good_turing_coverage_bound ¶
Good-Turing coverage with its one-sided 95% lower bound (Esty 1983).
lb95 = C - z_0.95 * sqrt((f1'/n (1 - f1'/n) + 2 f2/n) / n), clipped to [0, 1], with
f1' = max(f1, 1) in the variance. Without the floor, no singletons and no doubletons give a
variance of zero and a bound equal to the point 1.0, so 30 labels split [15, 15] read as proven
saturation; a sample that saw no singleton still cannot rule out an unseen mode, the same
reasoning behind Chao's f1 (f1 - 1) form when f2 = 0.
Raises ClusteringError when nothing was observed: there is no coverage to bound.
Source code in src/calibrated_agent_evaluation/cluster/saturation.py
saturation_check ¶
Full saturation verdict.
Saturated only when total >= min_failures, at least min_modes distinct modes were
observed, and the Good-Turing coverage one-sided 95% lower bound clears
coverage_threshold. A point estimate at the threshold does not certify, and a single mode
holding every observation (coverage exactly 1.0, nothing learned about the landscape) is never
saturated. Zero-count modes are dropped first.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
counts
|
list[int]
|
How many labeled failures each failure mode has. |
required |
min_failures
|
int
|
The fewest labeled failures that can count as saturated. |
30
|
coverage_threshold
|
float
|
The lower bound the Good-Turing coverage must clear. |
0.95
|
min_modes
|
int
|
The fewest distinct modes that can count as saturated. |
2
|
Returns:
| Type | Description |
|---|---|
SaturationResult
|
The verdict, the coverage bound and the Chao1 estimate of the total number of modes. |