Skip to content

calibrated_agent_evaluation.cluster

calibrated_agent_evaluation.cluster

Clustering, saturation & coverage engine.

Public API re-exports; import from calibrated_agent_evaluation.cluster.

ClusteringError

Bases: CalibratedAgentEvaluationDataError

Invalid input or state in the clustering pipeline.

UnratifiedCodebookError

Bases: ClusteringError

Attempt to use a draft codebook without human ratification.

Codebook dataclass

Codebook(
    version,
    modes,
    ratified_by,
    ratified_at,
    parent_version=None,
)

A versioned, human-ratified catalog of failure modes.

Only a Codebook with ratified_by set may tag Labels.

version instance-attribute

version

modes instance-attribute

modes

ratified_by instance-attribute

ratified_by

ratified_at instance-attribute

ratified_at

parent_version class-attribute instance-attribute

parent_version = None

CoverageResult dataclass

CoverageResult(
    cell_counts=dict(),
    blind_spots=(),
    total_cells=0,
    covered_cells=0,
    unclassified=(),
)

Output of coverage_report(): surfaces taxonomy blind spots.

unclassified are observed FAIL cells whose taxonomy_cell does not match any grid cell (an unknown/extra dimension value). They are surfaced, not silently dropped; a genuinely observed failure that falls outside the declared grid is a signal (the grid is incomplete or the trace is mis-tagged), not something to under-count into a phantom blind spot.

cell_counts class-attribute instance-attribute

cell_counts = field(default_factory=dict)

blind_spots class-attribute instance-attribute

blind_spots = ()

total_cells class-attribute instance-attribute

total_cells = 0

covered_cells class-attribute instance-attribute

covered_cells = 0

unclassified class-attribute instance-attribute

unclassified = ()

DraftCluster dataclass

DraftCluster(name, member_indices, member_critiques)

One proposed failure-mode cluster before human ratification.

name instance-attribute

name

member_indices instance-attribute

member_indices

member_critiques instance-attribute

member_critiques

DraftCodebook dataclass

DraftCodebook(
    clusters, embedding_model=None, created_at=None
)

The output of clustering: NOT usable for tagging.

clusters instance-attribute

clusters

embedding_model class-attribute instance-attribute

embedding_model = None

created_at class-attribute instance-attribute

created_at = None

FailureMode dataclass

FailureMode(name, description='', example_critiques=())

One named failure mode in a ratified codebook.

name instance-attribute

name

description class-attribute instance-attribute

description = ''

example_critiques class-attribute instance-attribute

example_critiques = ()

SaturationResult dataclass

SaturationResult(
    saturated,
    coverage,
    total_failures,
    total_modes,
    singletons,
    reason=None,
    chao1=None,
)

Output of saturation_check(): answers 'should I keep labeling?'.

coverage is the Good-Turing coverage with its one-sided 95% Esty lower bound (the gate reads coverage.lb95). chao1 is the Chao1 richness estimate with its two-sided 95% log-transformed interval, None when nothing was observed.

saturated instance-attribute

saturated

coverage instance-attribute

coverage

total_failures instance-attribute

total_failures

total_modes instance-attribute

total_modes

singletons instance-attribute

singletons

reason class-attribute instance-attribute

reason = None

chao1 class-attribute instance-attribute

chao1 = None

ratify

ratify(
    draft,
    *,
    ratified_by,
    merges=None,
    splits=None,
    renames=None,
    version="1",
    parent_version=None,
    descriptions=None,
)

Transform a DraftCodebook into a ratified Codebook.

Raises ClusteringError if version <= parent_version (monotonicity).

Parameters:

Name Type Description Default
draft DraftCodebook

The draft codebook to ratify.

required
ratified_by str

Who ratified it.

required
merges dict[str, list[str]] | None

New mode name to the draft modes merged into it.

None
splits dict[str, list[list[int]]] | None

Draft mode to the example index groups it splits into.

None
renames dict[str, str] | None

Draft mode name to its new name.

None
version str

The ratified codebook's version.

'1'
parent_version str | None

The version this one replaces.

None
descriptions Mapping[str, str] | None

Mode name to its definition, applied after merges and renames: a mode whose final name is a key gets that value as its description, and every other mode keeps the one it has (a rename keeps the old mode's). Keys that name no mode are unused, so a whole vocabulary can be passed. None attaches nothing.

None

Returns:

Type Description
Codebook

The ratified codebook.

Raises:

Type Description
ClusteringError

version doesn't advance past parent_version; splits is given; two draft clusters share a name; a merge or rename names an unknown mode; or a merge or rename targets a name that, when it is applied, a surviving mode holds which the operation does not consume (combine two modes with merges=, which may list its own target).

Source code in src/calibrated_agent_evaluation/cluster/codebook.py
def ratify(
    draft: DraftCodebook,
    *,
    ratified_by: str,
    merges: dict[str, list[str]] | None = None,
    splits: dict[str, list[list[int]]] | None = None,
    renames: dict[str, str] | None = None,
    version: str = "1",
    parent_version: str | None = None,
    descriptions: Mapping[str, str] | None = None,
) -> Codebook:
    """Transform a DraftCodebook into a ratified Codebook.

    Raises ClusteringError if version <= parent_version (monotonicity).

    Args:
        draft: The draft codebook to ratify.
        ratified_by: Who ratified it.
        merges: New mode name to the draft modes merged into it.
        splits: Draft mode to the example index groups it splits into.
        renames: Draft mode name to its new name.
        version: The ratified codebook's version.
        parent_version: The version this one replaces.
        descriptions: Mode name to its definition, applied after merges and renames: a mode
            whose final name is a key gets that value as its description, and every other mode
            keeps the one it has (a rename keeps the old mode's). Keys that name no mode are
            unused, so a whole vocabulary can be passed. ``None`` attaches nothing.

    Returns:
        The ratified codebook.

    Raises:
        ClusteringError: ``version`` doesn't advance past ``parent_version``; ``splits`` is
            given; two draft clusters share a name; a merge or rename names an unknown mode;
            or a merge or rename targets a name that, when it is applied, a surviving mode
            holds which the operation does not consume (combine two modes with ``merges=``,
            which may list its own target).
    """
    if parent_version is not None and not _version_advances(version, parent_version):
        raise ClusteringError(
            f"Codebook version {version!r} must sort strictly after "
            f"parent version {parent_version!r} (monotonicity)"
        )

    if splits is not None:
        # Permanent: a split needs a new cut of the embeddings, which
        # clustering owns; ratify only records the human's call on clusters that already exist.
        raise ClusteringError(
            "ratify does not split clusters: re-run hierarchical_cluster with a lower threshold "
            "to cut them finer, then merge or rename"
        )

    # Two clusters under one name would collapse into the last of them and drop the others'
    # critiques, so the draft is refused instead.
    counts = Counter(cluster.name for cluster in draft.clusters)
    repeated = [name for name, count in counts.items() if count > 1]
    if repeated:
        first = repeated[0]
        second = f"{first} #2"
        raise ClusteringError(
            f"draft has more than one cluster named {', '.join(repr(n) for n in repeated)}; "
            f"ratifying would keep one and drop the others' critiques. Give each cluster its own "
            f"name (for example {second!r}) and, if they are one mode, combine them with "
            f"merges={{{first!r}: {[first, second]!r}}}"
        )

    # Start with all draft clusters as modes
    modes_by_name: dict[str, FailureMode] = {}
    for cluster in draft.clusters:
        modes_by_name[cluster.name] = FailureMode(
            name=cluster.name,
            example_critiques=cluster.member_critiques[:3],
        )

    # Apply merges: {new_name: [old_name_1, old_name_2]}
    if merges:
        for new_name, old_names in merges.items():
            # Writing the merged mode over a surviving mode the merge does not consume would
            # silently drop that mode and its critiques; a merge may list its own target.
            if new_name in modes_by_name and new_name not in old_names:
                raise ClusteringError(
                    f"merge into {new_name!r} would replace the mode {new_name!r}, which the "
                    f"merge does not list, and drop its critiques; to combine them, list it "
                    f"among the merged modes: merges={{{new_name!r}: {[new_name, *old_names]!r}}}"
                )
            examples: list[str] = []
            for old_name in old_names:
                if old_name not in modes_by_name:
                    raise ClusteringError(
                        f"merge references unknown mode {old_name!r}; "
                        f"available: {list(modes_by_name.keys())}"
                    )
                examples.extend(modes_by_name.pop(old_name).example_critiques)
            modes_by_name[new_name] = FailureMode(
                name=new_name,
                example_critiques=tuple(examples[:5]),
            )

    # Apply renames: {old_name: new_name}
    if renames:
        for old_name, new_name in renames.items():
            if old_name not in modes_by_name:
                raise ClusteringError(
                    f"rename references unknown mode {old_name!r}; "
                    f"available: {list(modes_by_name.keys())}"
                )
            # Renaming onto another surviving mode's name would replace that mode and drop its
            # critiques; combining two modes is a merge, so the rename is refused.
            if new_name != old_name and new_name in modes_by_name:
                raise ClusteringError(
                    f"rename of {old_name!r} to {new_name!r} would replace the mode "
                    f"{new_name!r} and drop its critiques; to combine the two modes, merge them "
                    f"instead: merges={{{new_name!r}: {[new_name, old_name]!r}}}"
                )
            old_mode = modes_by_name.pop(old_name)
            modes_by_name[new_name] = FailureMode(
                name=new_name,
                description=old_mode.description,
                example_critiques=old_mode.example_critiques,
            )

    modes = tuple(modes_by_name.values())
    if descriptions is not None:
        modes = tuple(
            replace(mode, description=descriptions[mode.name])
            if mode.name in descriptions
            else mode
            for mode in modes
        )

    return Codebook(
        version=version,
        modes=modes,
        ratified_by=ratified_by,
        ratified_at=datetime.now(tz=UTC),
        parent_version=parent_version,
    )

tag_label

tag_label(label, mode, codebook)

Tag a Label with a failure mode from a ratified Codebook.

Raises UnratifiedCodebookError if codebook is a DraftCodebook. Raises ClusteringError if mode is not in the codebook.

Source code in src/calibrated_agent_evaluation/cluster/codebook.py
def tag_label(label: Label, mode: str, codebook: Codebook | DraftCodebook) -> Label:
    """Tag a Label with a failure mode from a ratified Codebook.

    Raises UnratifiedCodebookError if codebook is a DraftCodebook.
    Raises ClusteringError if mode is not in the codebook.
    """
    if not isinstance(codebook, Codebook):
        raise UnratifiedCodebookError(
            "Codebook must be human-ratified before use; call ratify() first"
        )

    mode_names = [m.name for m in codebook.modes]
    if mode not in mode_names:
        raise ClusteringError(
            f"Mode {mode!r} not found in codebook version {codebook.version}; "
            f"available: {mode_names}"
        )

    return replace(label, failure_mode=mode, codebook_version=codebook.version)

coverage_report

coverage_report(grid, observations)

Report per-cell observation counts and blind spots.

Parameters:

Name Type Description Default
grid TaxonomyGrid

The taxonomy dimensions and their values.

required
observations list[dict[str, str]]

The taxonomy cell of each FAIL-labeled trace.

required

Returns:

Type Description
CoverageResult

The count per cell, the cells with no failures, and observations outside the grid.

Source code in src/calibrated_agent_evaluation/cluster/coverage.py
def coverage_report(
    grid: TaxonomyGrid,
    observations: list[dict[str, str]],
) -> CoverageResult:
    """Report per-cell observation counts and blind spots.

    Args:
        grid: The taxonomy dimensions and their values.
        observations: The taxonomy cell of each FAIL-labeled trace.

    Returns:
        The count per cell, the cells with no failures, and observations outside the grid.
    """
    dim_names = sorted(grid.dimensions.keys())
    dim_values = [grid.dimensions[d] for d in dim_names]

    # Build all cells as canonical key strings
    all_cells: list[dict[str, str]] = []
    for combo in product(*dim_values):
        cell = dict(zip(dim_names, combo, strict=True))
        all_cells.append(cell)

    def _cell_key(cell: dict[str, str]) -> str:
        return "|".join(f"{k}={cell.get(k, '')}" for k in dim_names)

    # Count observations per cell. An observation whose taxonomy_cell is not a grid cell (an
    # unknown/extra dimension value) is recorded as UNCLASSIFIED, never silently discarded;
    # dropping it would under-count coverage and mislabel a genuinely-observed cell as a blind spot.
    cell_counts: dict[str, int] = {_cell_key(c): 0 for c in all_cells}
    unclassified: list[dict[str, str]] = []
    for obs in observations:
        key = _cell_key(obs)
        if key in cell_counts:
            cell_counts[key] += 1
        else:
            unclassified.append(obs)

    blind_spots: list[dict[str, str]] = []
    for cell in all_cells:
        if cell_counts[_cell_key(cell)] == 0:
            blind_spots.append(cell)

    covered = sum(1 for v in cell_counts.values() if v > 0)

    return CoverageResult(
        cell_counts=cell_counts,
        blind_spots=tuple(blind_spots),
        total_cells=len(all_cells),
        covered_cells=covered,
        unclassified=tuple(unclassified),
    )

build_draft

build_draft(
    critiques, embeddings, assignments, *, namer=None
)

Assemble a DraftCodebook from cluster assignments.

Groups critiques by assignment label, optionally asks the namer for names.

Raises:

Type Description
ClusteringError

The namer returns a different number of names than there are clusters, or returns any name more than once.

Source code in src/calibrated_agent_evaluation/cluster/draft.py
def build_draft(
    critiques: list[str],
    embeddings: list[list[float]],  # noqa: ARG001
    assignments: list[int],
    *,
    namer: DraftNamer | None = None,
) -> DraftCodebook:
    """Assemble a DraftCodebook from cluster assignments.

    Groups critiques by assignment label, optionally asks the namer for names.

    Raises:
        ClusteringError: The namer returns a different number of names than there are
            clusters, or returns any name more than once.
    """
    groups: dict[int, list[int]] = {}
    for idx, label in enumerate(assignments):
        groups.setdefault(label, []).append(idx)

    # Assemble in a single canonical (sorted-by-label) order so the namer sees clusters in the
    # SAME order the DraftClusters are built; otherwise scipy's non-sorted labels would attach
    # each proposed name to the wrong cluster.
    sorted_groups = sorted(groups.items())
    cluster_critique_lists = [[critiques[i] for i in indices] for _label, indices in sorted_groups]

    if namer is not None:
        names = namer.name_clusters(cluster_critique_lists)
        # A namer (e.g. an LLM) that returns the wrong number of names would, under a lenient
        # zip, silently DROP the unnamed clusters and every critique in them; the exact
        # failure-mode-loss this module exists to prevent. Refuse loudly instead (no silent
        # truncation).
        if len(names) != len(sorted_groups):
            raise ClusteringError(
                f"namer returned {len(names)} name(s) for {len(sorted_groups)} cluster(s); "
                f"every cluster must be named or its critiques would be dropped"
            )
        # Two clusters under one name would collapse into a single mode at ratification and
        # lose the other cluster's critiques, so a repeated name is refused, not built.
        repeated = [name for name, count in Counter(names).items() if count > 1]
        if repeated:
            raise ClusteringError(
                f"namer gave more than one cluster the name(s) "
                f"{', '.join(repr(name) for name in repeated)}; every cluster needs its own "
                f"name or ratification would collapse them into one mode"
            )
    else:
        names = [f"mode_{i}" for i in range(len(sorted_groups))]

    clusters: list[DraftCluster] = []
    for (_label, indices), name in zip(sorted_groups, names, strict=True):
        clusters.append(
            DraftCluster(
                name=name,
                member_indices=tuple(indices),
                member_critiques=tuple(critiques[i] for i in indices),
            )
        )

    return DraftCodebook(clusters=tuple(clusters))

hierarchical_cluster

hierarchical_cluster(embeddings, *, threshold=1.0)

Cluster embeddings via Ward's linkage + distance-based cut.

Returns a list of cluster assignments (0-indexed integers), one per embedding. Requires the [stats] extra (scipy).

Source code in src/calibrated_agent_evaluation/cluster/group.py
def hierarchical_cluster(
    embeddings: list[list[float]],
    *,
    threshold: float = 1.0,
) -> list[int]:
    """Cluster embeddings via Ward's linkage + distance-based cut.

    Returns a list of cluster assignments (0-indexed integers), one per embedding.
    Requires the ``[stats]`` extra (scipy).
    """
    if len(embeddings) < 2:
        raise ClusteringError(f"Clustering requires ≥2 FAIL critiques; got {len(embeddings)}")

    try:
        from scipy.cluster.hierarchy import fcluster, linkage
    except ImportError as exc:
        raise ClusteringError(
            f"scipy is required for hierarchical clustering; install with: {install_hint('stats')}"
        ) from exc

    import numpy as np

    matrix = np.array(embeddings, dtype=np.float64)
    z = linkage(matrix, method="ward")
    labels = fcluster(z, t=threshold, criterion="distance")
    # Convert from 1-indexed (scipy) to 0-indexed
    return [int(lbl) - 1 for lbl in labels]

chao1_estimate

chao1_estimate(counts)

Chao1 lower-bound species-richness estimator (Chao 1984).

Returns (estimated_total_modes, modes_found, fraction_discovered). Zero-count modes are not observed modes and do not enter modes_found.

Source code in src/calibrated_agent_evaluation/cluster/saturation.py
def chao1_estimate(counts: list[int]) -> tuple[float, int, float]:
    """Chao1 lower-bound species-richness estimator (Chao 1984).

    Returns ``(estimated_total_modes, modes_found, fraction_discovered)``. Zero-count modes are not
    observed modes and do not enter ``modes_found``.
    """
    observed = _validate_counts(counts)
    if not observed:
        return (0.0, 0, 0.0)

    s_obs = len(observed)
    n1 = sum(1 for c in observed if c == 1)  # singletons
    n2 = sum(1 for c in observed if c == 2)  # doubletons

    if n1 == 0:
        # no singletons -- all modes observed >=2x, no unseen predicted
        estimated = float(s_obs)
    elif n2 == 0:
        # Bias-corrected form when no doubletons
        estimated = s_obs + (n1 * (n1 - 1)) / 2.0
    else:
        estimated = s_obs + (n1 * n1) / (2.0 * n2)

    fraction = s_obs / estimated if estimated > 0 else 1.0
    return (estimated, s_obs, fraction)

chao1_interval

chao1_interval(counts)

Chao1 with Chao's (1987) log-transformed two-sided 95% interval.

With T = Chao1 - S_obs and K = exp(z sqrt(log(1 + Var/T^2))) the interval is [S_obs + T/K, S_obs + T*K] (eq. 13), so it never falls below S_obs. The variance is eq. 5 (f2 > 0) or eq. 7 (f2 = 0). When T = 0 (f1 = 0, or f1 = 1 with f2 = 0) the log transform is undefined and eq. 14 is used instead, built on Var(S_obs).

Raises ClusteringError when nothing was observed.

Source code in src/calibrated_agent_evaluation/cluster/saturation.py
def chao1_interval(counts: list[int]) -> Interval:
    """Chao1 with Chao's (1987) log-transformed two-sided 95% interval.

    With ``T = Chao1 - S_obs`` and ``K = exp(z sqrt(log(1 + Var/T^2)))`` the interval is
    ``[S_obs + T/K, S_obs + T*K]`` (eq. 13), so it never falls below ``S_obs``. The variance is
    eq. 5 (``f2 > 0``) or eq. 7 (``f2 = 0``). When ``T = 0`` (``f1 = 0``, or ``f1 = 1`` with
    ``f2 = 0``) the log transform is undefined and eq. 14 is used instead, built on ``Var(S_obs)``.

    Raises :class:`ClusteringError` when nothing was observed.
    """
    observed = _validate_counts(counts)
    if not observed:
        raise ClusteringError("chao1_interval: no observed modes, nothing to estimate")
    estimated, s_obs, _ = chao1_estimate(observed)
    n = sum(observed)
    f1 = sum(1 for c in observed if c == 1)
    f2 = sum(1 for c in observed if c == 2)
    t = estimated - s_obs
    if t > 0:
        if f2 > 0:
            r = f1 / f2
            var = f2 * (0.5 * r**2 + r**3 + 0.25 * r**4)
        else:
            var = f1 * (f1 - 1) / 2.0 + f1 * (2 * f1 - 1) ** 2 / 4.0 - f1**4 / (4.0 * estimated)
        k = math.exp(_Z_TWO_SIDED_95 * math.sqrt(math.log(1.0 + max(var, 0.0) / (t * t))))
        lo, hi = s_obs + t / k, s_obs + t * k
    else:
        var_obs, p = _var_s_obs(observed)
        centre = s_obs / (1.0 - p)
        half = _Z_TWO_SIDED_95 * math.sqrt(var_obs) / (1.0 - p)
        lo, hi = max(float(s_obs), centre - half), centre + half
    return Interval(
        point=estimated,
        lo=lo,
        hi=max(hi, estimated),
        method=METHOD_CHAO1_LOG,
        level=0.95,
        sided="two-sided",
        n=n,
    )

good_turing_coverage

good_turing_coverage(counts)

Good-Turing coverage point estimate: 1 - singletons/total.

Returns a float in [0.0, 1.0], and 0.0 when nothing was observed. Use good_turing_coverage_bound for the number a gate may read.

Source code in src/calibrated_agent_evaluation/cluster/saturation.py
def good_turing_coverage(counts: list[int]) -> float:
    """Good-Turing coverage point estimate: ``1 - singletons/total``.

    Returns a float in [0.0, 1.0], and 0.0 when nothing was observed. Use
    :func:`good_turing_coverage_bound` for the number a gate may read.
    """
    observed = _validate_counts(counts)
    total = sum(observed)
    if total == 0:
        return 0.0
    singletons = sum(1 for c in observed if c == 1)
    return 1.0 - (singletons / total)

good_turing_coverage_bound

good_turing_coverage_bound(counts)

Good-Turing coverage with its one-sided 95% lower bound (Esty 1983).

lb95 = C - z_0.95 * sqrt((f1'/n (1 - f1'/n) + 2 f2/n) / n), clipped to [0, 1], with f1' = max(f1, 1) in the variance. Without the floor, no singletons and no doubletons give a variance of zero and a bound equal to the point 1.0, so 30 labels split [15, 15] read as proven saturation; a sample that saw no singleton still cannot rule out an unseen mode, the same reasoning behind Chao's f1 (f1 - 1) form when f2 = 0.

Raises ClusteringError when nothing was observed: there is no coverage to bound.

Source code in src/calibrated_agent_evaluation/cluster/saturation.py
def good_turing_coverage_bound(counts: list[int]) -> Bound:
    """Good-Turing coverage with its one-sided 95% lower bound (Esty 1983).

    ``lb95 = C - z_0.95 * sqrt((f1'/n (1 - f1'/n) + 2 f2/n) / n)``, clipped to [0, 1], with
    ``f1' = max(f1, 1)`` in the variance. Without the floor, no singletons and no doubletons give a
    variance of zero and a bound equal to the point 1.0, so 30 labels split [15, 15] read as proven
    saturation; a sample that saw no singleton still cannot rule out an unseen mode, the same
    reasoning behind Chao's ``f1 (f1 - 1)`` form when ``f2 = 0``.

    Raises :class:`ClusteringError` when nothing was observed: there is no coverage to bound.
    """
    observed = _validate_counts(counts)
    n = sum(observed)
    if n == 0:
        raise ClusteringError("good_turing_coverage_bound: no observations, nothing to bound")
    f1 = sum(1 for c in observed if c == 1)
    f2 = sum(1 for c in observed if c == 2)
    point = 1.0 - f1 / n
    f1_var = max(f1, 1)
    var = (f1_var / n * (1.0 - f1_var / n) + 2.0 * f2 / n) / n
    lb = min(1.0, max(0.0, point - _Z_ONE_SIDED_95 * math.sqrt(max(var, 0.0))))
    return Bound(point=point, lb95=lb, n=n, method=METHOD_ESTY)

saturation_check

saturation_check(
    counts,
    *,
    min_failures=30,
    coverage_threshold=0.95,
    min_modes=2,
)

Full saturation verdict.

Saturated only when total >= min_failures, at least min_modes distinct modes were observed, and the Good-Turing coverage one-sided 95% lower bound clears coverage_threshold. A point estimate at the threshold does not certify, and a single mode holding every observation (coverage exactly 1.0, nothing learned about the landscape) is never saturated. Zero-count modes are dropped first.

Parameters:

Name Type Description Default
counts list[int]

How many labeled failures each failure mode has.

required
min_failures int

The fewest labeled failures that can count as saturated.

30
coverage_threshold float

The lower bound the Good-Turing coverage must clear.

0.95
min_modes int

The fewest distinct modes that can count as saturated.

2

Returns:

Type Description
SaturationResult

The verdict, the coverage bound and the Chao1 estimate of the total number of modes.

Source code in src/calibrated_agent_evaluation/cluster/saturation.py
def saturation_check(
    counts: list[int],
    *,
    min_failures: int = 30,
    coverage_threshold: float = 0.95,
    min_modes: int = 2,
) -> SaturationResult:
    """Full saturation verdict.

    Saturated only when ``total >= min_failures``, at least ``min_modes`` distinct modes were
    observed, and the Good-Turing coverage one-sided 95% **lower bound** clears
    ``coverage_threshold``. A point estimate at the threshold does not certify, and a single mode
    holding every observation (coverage exactly 1.0, nothing learned about the landscape) is never
    saturated. Zero-count modes are dropped first.

    Args:
        counts: How many labeled failures each failure mode has.
        min_failures: The fewest labeled failures that can count as saturated.
        coverage_threshold: The lower bound the Good-Turing coverage must clear.
        min_modes: The fewest distinct modes that can count as saturated.

    Returns:
        The verdict, the coverage bound and the Chao1 estimate of the total number of modes.
    """
    observed = _validate_counts(counts)

    total = sum(observed)
    n_modes = len(observed)
    singletons = sum(1 for c in observed if c == 1)
    coverage = (
        good_turing_coverage_bound(observed)
        if total > 0
        else Bound(point=0.0, lb95=0.0, n=0, method=METHOD_ESTY)
    )
    chao1 = chao1_interval(observed) if observed else None

    def _result(saturated: bool, reason: str | None) -> SaturationResult:
        return SaturationResult(
            saturated=saturated,
            coverage=coverage,
            total_failures=total,
            total_modes=n_modes,
            singletons=singletons,
            reason=reason,
            chao1=chao1,
        )

    if total < min_failures:
        return _result(
            False, f"insufficient_observations: {total} < {min_failures}, the minimum failure count"
        )

    if n_modes < min_modes:
        return _result(
            False,
            f"too_few_modes: {n_modes} observed mode(s) < {min_modes}; every observation fell in "
            "one cluster, which says nothing about the unseen modes",
        )

    if coverage.lb95 < coverage_threshold:
        return _result(
            False,
            f"coverage_below_threshold: Good-Turing coverage lower bound {coverage.lb95:.4f} "
            f"(point {coverage.point:.4f}) < {coverage_threshold} ({singletons} singletons in "
            f"{total} observations)",
        )

    return _result(True, None)