Skip to content

calibrated_agent_evaluation.attribution

calibrated_agent_evaluation.attribution

Crew attribution layer.

Makes a multi-agent crew evaluable per-agent, not just at the outcome: - Crew Define artifact -- expected agent set / order / rules per scenario, so a skip is a diff. - Counterfactual walk-back -- validate a recorded culprit/manifestation against the span tree (plus an opt-in LLM proposer the human ratifies). - Two-vocabulary codebook -- failure modes + responsibility boundaries, versioned. - Per-agent corrected rates -- each agent corrected by its own judge's TPR/TNR (reuses the calibration engine). - Relocation primitive -- gate crew improvement on the workflow outcome rate, not one agent's.

Pure methodology plane: reads normalized Trace/Span/Label and reuses the calibration engine; no trace-store/executor vendor type appears here. It does NOT run LLM judges nor cluster critiques into modes.

Public API re-exports from submodules.

SpanAgents module-attribute

SpanAgents = Mapping[tuple[str, str], str]

AttributionStatus module-attribute

AttributionStatus = Literal['single', 'disputed']

Denominator module-attribute

Denominator = Literal[
    "share-of-failures", "rate-over-traffic"
]

RelocationVerdict module-attribute

RelocationVerdict = Literal[
    "improved", "relocation", "regressed", "flat"
]

Vocabulary module-attribute

Vocabulary = Literal[
    "failure_modes", "responsibility_boundaries"
]

AgentRecordMismatchError

Bases: CalibratedAgentEvaluationDataError

Per-agent correction was handed another agent's judge calibration record.

AttributionError

Bases: CalibratedAgentEvaluationDataError

A walk-back that doesn't fit the trace.

It names a span absent from the trace, orders the culprit after the manifestation, or implicates an undeclared handoff seam.

CodebookError

Bases: CalibratedAgentEvaluationDataError

A codebook entry is missing its definition or a positive/boundary example.

CrewDefinitionError

Bases: CalibratedAgentEvaluationDataError

An invalid crew Define artifact, or a scenario with no declared expected process.

AgentRate dataclass

AgentRate(agent, interval, denominator)

One agent's Rogan-Gladen-corrected rate.

interval is the Interval (corrected point and its interval, or a refusal reason) produced by that agent's own judge; rows do not share a fate. denominator states whether this is a share-of-failures or a rate-over-traffic number, so a discovery-pile rate is never quoted as the customer SLA.

agent instance-attribute

agent

interval instance-attribute

interval

denominator instance-attribute

denominator

AgreementReport dataclass

AgreementReport(
    outcome_kappa,
    culprit_kappa,
    n_dual_coded,
    outcome_undefined_reason=None,
    culprit_undefined_reason=None,
    n_culprit_coded=0,
)

Per-axis inter-coder agreement over a dual-coded subset.

A high outcome_kappa with a low culprit_kappa is the most useful multiagent signal; a fuzzy agent/handoff responsibility boundary (a real product bug), not labeler noise.

Each κ carries its 95% interval. An axis on which both coders used one and the same value throughout has κ = 0/0: its field is None and its *_undefined_reason says so, rather than the conventional 1.0 that would read as proven agreement. n_dual_coded counts the traces both coders labeled (the outcome axis); n_culprit_coded the ones both called FAIL (the culprit axis).

outcome_kappa instance-attribute

outcome_kappa

culprit_kappa instance-attribute

culprit_kappa

n_dual_coded instance-attribute

n_dual_coded

outcome_undefined_reason class-attribute instance-attribute

outcome_undefined_reason = None

culprit_undefined_reason class-attribute instance-attribute

culprit_undefined_reason = None

n_culprit_coded class-attribute instance-attribute

n_culprit_coded = 0

Attribution dataclass

Attribution(
    trace_id,
    culprit_spans,
    manifestation_span,
    handoff_implicated,
    status,
    span_defect=False,
)

A validated counterfactual walk-back over the span tree.

culprit_spans has ≥1 entry; more than one ⇒ status="disputed" (an undefined responsibility boundary, recorded; never collapsed to a single culprit). span_defect is the outcome-pass / span-fail flag: a defect recorded independently of the trace outcome, supplying per-span gold.

trace_id instance-attribute

trace_id

culprit_spans instance-attribute

culprit_spans

manifestation_span instance-attribute

manifestation_span

handoff_implicated instance-attribute

handoff_implicated

status instance-attribute

status

span_defect class-attribute instance-attribute

span_defect = False

AttributionSuggestion dataclass

AttributionSuggestion(culprit_span, critique)

An LLM proposal for a crew FAIL: NEVER a Label until a human ratifies it.

critique precedes the (human-ratified) verdict, mirroring the review-UI Suggestion. Whatever produced it, ratify_suggestion refuses it until a named human ratifies it. No proposer ships: the WalkBackProposer protocol and its NoProposer default were removed, since nothing implemented or consumed them and no planned feature owns one.

culprit_span instance-attribute

culprit_span

critique instance-attribute

critique

Codebook dataclass

Codebook(
    version,
    failure_modes=(),
    responsibility_boundaries=(),
    changelog=(),
)

A crew's two versioned vocabularies: failure modes and agent or handoff boundaries.

A version bump flags labels written under the prior version for re-labeling.

version instance-attribute

version

failure_modes class-attribute instance-attribute

failure_modes = ()

responsibility_boundaries class-attribute instance-attribute

responsibility_boundaries = ()

changelog class-attribute instance-attribute

changelog = ()

CodebookEntry dataclass

CodebookEntry(
    name, definition, positive_examples, boundary_examples
)

One versioned vocabulary entry.

It needs a definition, at least one positive example and at least one boundary example, and is registered through register_entry.

name instance-attribute

name

definition instance-attribute

definition

positive_examples instance-attribute

positive_examples

boundary_examples instance-attribute

boundary_examples

CrewDefinition dataclass

CrewDefinition(agents, handoffs, scenarios, rules)

The crew Define artifact.

Built + validated by build_crew_definition (a skipped agent is a diff against scenarios[...].expected_agents; a violated boundary is a rules clause). handoffs are declared seams like "intake->policy-checker".

agents instance-attribute

agents

handoffs instance-attribute

handoffs

scenarios instance-attribute

scenarios

rules instance-attribute

rules

CrewOutcomes dataclass

CrewOutcomes(
    workflow_failures, workflow_total, per_agent_failures
)

A before/after snapshot for the relocation check.

workflow_failures / workflow_total give the crew's workflow outcome rate (the gate metric); per_agent_failures counts each agent's span-level failures over the same slice.

workflow_failures instance-attribute

workflow_failures

workflow_total instance-attribute

workflow_total

per_agent_failures instance-attribute

per_agent_failures

ExpectedProcess dataclass

ExpectedProcess(
    scenario,
    expected_agents,
    terminal_decision,
    payout_allowed,
    escalation_required,
)

Per-scenario expected process: the diff target for the process judge.

expected_agents is the valid ordering (a sequence, not just a set); a skipped/mis-routed agent is only detectable as expected minus actual, so it must exist before any trace.

scenario instance-attribute

scenario

expected_agents instance-attribute

expected_agents

terminal_decision instance-attribute

terminal_decision

payout_allowed instance-attribute

payout_allowed

escalation_required instance-attribute

escalation_required

MandatedRule dataclass

MandatedRule(
    rule_id,
    antecedent,
    consequent_agent,
    deterministic=True,
)

A named responsibility boundary: antecedent ⇒ required consequent agent.

deterministic=False marks a rule the process judge cannot decide by agent set-difference alone; it must be routed to a calibrated sub-judge.

rule_id instance-attribute

rule_id

antecedent instance-attribute

antecedent

consequent_agent instance-attribute

consequent_agent

deterministic class-attribute instance-attribute

deterministic = True

RelocationResult pydantic-model

Bases: CalibratedAgentEvaluationModel

The before/after crew-improvement verdict.

verdict="relocation" when the touched agent's failure rate fell (its interval below 0) but the workflow outcome rate was not shown to improve; a per-agent gain with a flat outcome rate is failure relocation, not a fix. workflow_rate_change and touched_agent_rate_change are the after-minus-before intervals the verdict read; None on results made before they were recorded.

Config:

  • frozen: True
  • extra: forbid

Fields:

workflow_outcome_rate_before pydantic-field

workflow_outcome_rate_before

workflow_outcome_rate_after pydantic-field

workflow_outcome_rate_after

per_agent_failures_before pydantic-field

per_agent_failures_before

per_agent_failures_after pydantic-field

per_agent_failures_after

verdict pydantic-field

verdict

relocated_from pydantic-field

relocated_from

workflow_rate_change pydantic-field

workflow_rate_change = None

touched_agent_rate_change pydantic-field

touched_agent_rate_change = None

attribution_agreement

attribution_agreement(coder_a, coder_b)

Return per-axis (outcome, culprit) inter-coder agreement, or None if none dual-coded.

Labels are trace-level and matched on trace_id. The outcome axis compares outcome over every trace both coders labeled. The culprit axis compares culprit_span over the traces both coders called FAIL: a PASS has no culprit, and counting the PASS traces where both culprits are None would carry the outcome agreement into the culprit axis (80 shared PASS traces and 20 FAILs with culprits picked at random read as culprit kappa 0.54 instead of about 0). Returns:

  • an AgreementReport (outcome_kappa / culprit_kappa / n_dual_coded / n_culprit_coded) when at least one trace was co-labeled, each kappa with its 95% interval or None plus a reason when it is 0/0 or has no traces on that axis;
  • None when the coders share no trace -- there is nothing to agree on, so no figure is fabricated from a single coder (the capability-absent path).

Raises:

Type Description
AttributionError

A coder passed a span-level label or labeled a trace twice.

Source code in src/calibrated_agent_evaluation/attribution/agreement.py
def attribution_agreement(
    coder_a: Sequence[Label],
    coder_b: Sequence[Label],
) -> AgreementReport | None:
    """Return per-axis (outcome, culprit) inter-coder agreement, or ``None`` if none dual-coded.

    Labels are trace-level and matched on ``trace_id``. The outcome axis compares ``outcome`` over
    every trace both coders labeled. The culprit axis compares ``culprit_span`` over the traces
    both coders called FAIL: a PASS has no culprit, and counting the PASS traces where both
    culprits are ``None`` would carry the outcome agreement into the culprit axis (80 shared PASS
    traces and 20 FAILs with culprits picked at random read as culprit kappa 0.54 instead of about
    0). Returns:

      * an :class:`AgreementReport` (``outcome_kappa`` / ``culprit_kappa`` / ``n_dual_coded`` /
        ``n_culprit_coded``) when at least one trace was co-labeled, each kappa with its 95%
        interval or ``None`` plus a reason when it is 0/0 or has no traces on that axis;
      * ``None`` when the coders share no trace -- there is nothing to agree on, so no figure is
        fabricated from a single coder (the capability-absent path).

    Raises:
        AttributionError: A coder passed a span-level label or labeled a trace twice.
    """
    a_by_trace = _by_trace(coder_a, "A")
    b_by_trace = _by_trace(coder_b, "B")
    shared = sorted(a_by_trace.keys() & b_by_trace.keys())
    if not shared:
        return None

    outcome_a = [a_by_trace[t].outcome for t in shared]
    outcome_b = [b_by_trace[t].outcome for t in shared]
    co_fail = [t for t in shared if a_by_trace[t].outcome == b_by_trace[t].outcome == "FAIL"]

    outcome, outcome_reason = _kappa(outcome_a, outcome_b)
    if co_fail:
        culprit, culprit_reason = _kappa(
            [a_by_trace[t].culprit_span for t in co_fail],
            [b_by_trace[t].culprit_span for t in co_fail],
        )
    else:
        culprit, culprit_reason = (
            None,
            "no trace was called FAIL by both coders, so there is no culprit to agree on",
        )
    return AgreementReport(
        outcome_kappa=outcome,
        culprit_kappa=culprit,
        n_dual_coded=len(shared),
        outcome_undefined_reason=outcome_reason,
        culprit_undefined_reason=culprit_reason,
        n_culprit_coded=len(co_fail),
    )

bump_version

bump_version(codebook, new_version, labels, *, change_note)

Return the re-versioned codebook and the labels to flag for re-labeling.

Every Label whose codebook_version differs from new_version (i.e. was written under a superseded version, including None) is returned in the flag list -- a definition change is retroactive, so labeled history must be re-checked against the new definition. Labels already at new_version are not flagged. change_note is appended to the changelog for audit.

Source code in src/calibrated_agent_evaluation/attribution/codebook.py
def bump_version(
    codebook: Codebook,
    new_version: str,
    labels: Sequence[Label],
    *,
    change_note: str,
) -> tuple[Codebook, list[Label]]:
    """Return the re-versioned codebook and the labels to flag for re-labeling.

    Every Label whose ``codebook_version`` differs from ``new_version`` (i.e. was written under a
    superseded version, including ``None``) is returned in the flag list -- a definition change is
    retroactive, so labeled history must be re-checked against the new definition. Labels already at
    ``new_version`` are not flagged. ``change_note`` is appended to the changelog for audit.
    """
    if not new_version:
        raise CodebookError("A codebook version bump requires a non-empty new_version.")
    flagged = [label for label in labels if label.codebook_version != new_version]
    bumped = replace(
        codebook,
        version=new_version,
        changelog=(*codebook.changelog, change_note),
    )
    return bumped, flagged

register_entry

register_entry(codebook, vocabulary, entry)

Return a new codebook with entry appended to the named vocabulary.

Raises CodebookError if the entry has an empty definition, or fewer than one positive or one boundary example -- the examples are what make a category ratifiable, so an entry without them is refused rather than stored half-defined. Also refuses a name already present in the target vocabulary.

Source code in src/calibrated_agent_evaluation/attribution/codebook.py
def register_entry(codebook: Codebook, vocabulary: Vocabulary, entry: CodebookEntry) -> Codebook:
    """Return a new codebook with ``entry`` appended to the named vocabulary.

    Raises :class:`CodebookError` if the entry has an empty definition, or fewer than one positive
    or one boundary example -- the examples are what make a category ratifiable, so an
    entry without them is refused rather than stored half-defined. Also refuses a name already
    present in the target vocabulary.
    """
    _validate_entry(entry)
    _refuse_duplicate_name(codebook, vocabulary, entry)
    if vocabulary == "failure_modes":
        return replace(codebook, failure_modes=(*codebook.failure_modes, entry))
    return replace(codebook, responsibility_boundaries=(*codebook.responsibility_boundaries, entry))

build_crew_definition

build_crew_definition(
    *, agents, handoffs, scenarios, rules
)

Build a validated crew Define artifact.

Raises CrewDefinitionError if the crew declares no agents, a scenario declares an empty expected agent set, a scenario references an agent not in agents, a mandated rule's consequent_agent is not a declared agent, or a handoff is not a well-formed a->b over two declared agents. No silent fall-back; an invalid Define artifact refuses loudly.

Source code in src/calibrated_agent_evaluation/attribution/definition.py
def build_crew_definition(
    *,
    agents: Iterable[str],
    handoffs: Iterable[str],
    scenarios: Mapping[str, ExpectedProcess],
    rules: Iterable[MandatedRule],
) -> CrewDefinition:
    """Build a validated crew Define artifact.

    Raises :class:`CrewDefinitionError` if the crew declares no agents, a scenario declares an empty
    expected agent set, a scenario references an agent not in ``agents``, a mandated rule's
    ``consequent_agent`` is not a declared agent, or a handoff is not a well-formed ``a->b`` over
    two declared agents. No silent fall-back; an invalid Define artifact refuses loudly.
    """
    agent_tuple = tuple(agents)
    if not agent_tuple:
        raise CrewDefinitionError("A crew Define artifact must declare at least one agent.")
    agent_set = set(agent_tuple)

    scenario_map = dict(scenarios)
    for name, proc in scenario_map.items():
        _validate_expected_process(name, proc, agent_set)

    rule_tuple = tuple(rules)
    for rule in rule_tuple:
        if rule.consequent_agent not in agent_set:
            raise CrewDefinitionError(
                f"rule {rule.rule_id!r} names consequent agent {rule.consequent_agent!r}, "
                f"which is not a declared crew agent {sorted(agent_set)}."
            )

    handoff_tuple = tuple(handoffs)
    for handoff in handoff_tuple:
        _validate_handoff(handoff, agent_set)

    return CrewDefinition(
        agents=agent_tuple,
        handoffs=handoff_tuple,
        scenarios=scenario_map,
        rules=rule_tuple,
    )

require_expected_process

require_expected_process(crew, scenario)

Return the ExpectedProcess for scenario, or refuse.

Raises CrewDefinitionError if the scenario declares no expected process; the process judge cannot diff expected minus actual without it, and an undeclared boundary is a defect.

Source code in src/calibrated_agent_evaluation/attribution/definition.py
def require_expected_process(crew: CrewDefinition, scenario: str) -> ExpectedProcess:
    """Return the ``ExpectedProcess`` for ``scenario``, or refuse.

    Raises :class:`CrewDefinitionError` if the scenario declares no expected process; the process
    judge cannot diff *expected minus actual* without it, and an undeclared boundary is a defect.
    """
    proc = crew.scenarios.get(scenario)
    if proc is None:
        raise CrewDefinitionError(
            f"scenario {scenario!r} declares no expected process; the process judge cannot "
            f"evaluate an undeclared crew route."
        )
    return proc

attributed_agent

attributed_agent(label, agents=None)

The agent label attributes its verdict to, or None for a trace-level label.

The source span is the human's culprit_span, or for a judge or scorer verdict the span it judged. Its agent is read from agents when the span is in scope, since a span id need not spell its agent; otherwise from the id convention "CLM-4471/intake" (the segment after the last /). A re-executed node's occurrence suffix ("CLM-4471/intake#2") is stripped: both executions are the same agent, and keying them apart would split one agent's rate.

Source code in src/calibrated_agent_evaluation/attribution/labels.py
def attributed_agent(label: Label, agents: SpanAgents | None = None) -> str | None:
    """The agent ``label`` attributes its verdict to, or ``None`` for a trace-level label.

    The source span is the human's ``culprit_span``, or for a judge or scorer verdict the span it
    judged. Its agent is read from ``agents`` when the span is in scope, since a span id need not
    spell its agent; otherwise from the id convention ``"CLM-4471/intake"`` (the segment after the
    last ``/``). A re-executed node's occurrence suffix (``"CLM-4471/intake#2"``) is stripped:
    both executions are the same agent, and keying them apart would split one agent's rate.
    """
    source = label.culprit_span
    if source is None and label.kind in _SCORED_KINDS:
        source = label.span_id
    if not source:
        return None
    if agents and (label.trace_id, source) in agents:
        return agents[(label.trace_id, source)]
    if "/" in source:
        return source.rsplit("/", 1)[-1].split("#", 1)[0]
    return None

span_agents

span_agents(traces)

(trace_id, span_id) -> agent for every span of traces.

Source code in src/calibrated_agent_evaluation/attribution/labels.py
def span_agents(traces: Iterable[Trace]) -> dict[tuple[str, str], str]:
    """``(trace_id, span_id) -> agent`` for every span of ``traces``."""
    return {(t.trace_id, s.span_id): s.agent for t in traces for s in t.spans}

per_agent_rates

per_agent_rates(
    agent_records,
    agent_verdicts,
    *,
    agent_judges,
    denominator,
    seed=0,
)

Return one independent AgentRate per agent.

agent_records maps each agent to its own judge's certified CalibrationRecord; agent_verdicts maps each agent to (verdicts, rate_set_id, trace_ids) -- that agent's judge verdicts (1=FAIL, 0=PASS) over a production-representative sample, the id of that sample, and the trace id of each verdict (checked against the calibration gold). agent_judges maps each agent to the judge_id that legitimately judges it.

Each agent's rate is corrected via corrected_rate (Rogan-Gladen with its interval); the resulting Interval is attached with the stated denominator. The calibration refusals propagate unchanged -- an uncertified agent judge raises UncalibratedJudgeError; a J<0.5 or non-identifiable correction yields an interval whose point is None (the widening bound is the signal), never a silently fabricated number.

Raises AgentRecordMismatchError if an agent has no declared judge in agent_judges (the binding cannot be verified, so it is refused rather than silently trusted), or if an agent's record's judge_id is not that declared judge -- a policy-checker rate must not be computed from intake's judge.

Source code in src/calibrated_agent_evaluation/attribution/rates.py
def per_agent_rates(
    agent_records: Mapping[str, CalibrationRecord],
    agent_verdicts: Mapping[str, tuple[Sequence[int], str, Sequence[str]]],
    *,
    agent_judges: Mapping[str, str],
    denominator: Denominator,
    seed: int = 0,
) -> list[AgentRate]:
    """Return one independent :class:`AgentRate` per agent.

    ``agent_records`` maps each agent to its own judge's certified :class:`CalibrationRecord`;
    ``agent_verdicts`` maps each agent to ``(verdicts, rate_set_id, trace_ids)`` -- that agent's
    judge verdicts (1=FAIL, 0=PASS) over a production-representative sample, the id of that
    sample, and the trace id of each verdict (checked against the calibration gold).
    ``agent_judges`` maps each agent to the ``judge_id`` that legitimately judges it.

    Each agent's rate is corrected via ``corrected_rate`` (Rogan-Gladen with its interval); the
    resulting :class:`~calibrated_agent_evaluation.calibrate.record.Interval` is attached with the
    stated ``denominator``. The calibration refusals propagate unchanged -- an uncertified agent
    judge raises ``UncalibratedJudgeError``; a J<0.5 or non-identifiable correction yields an
    interval whose point is ``None`` (the widening bound is the signal), never a silently fabricated
    number.

    Raises :class:`AgentRecordMismatchError` if an agent has no declared judge in ``agent_judges``
    (the binding cannot be verified, so it is refused rather than silently trusted), or if an
    agent's record's ``judge_id`` is not that declared judge -- a policy-checker rate must not be
    computed from intake's judge.
    """
    rates: list[AgentRate] = []
    for agent, record in agent_records.items():
        expected_judge = agent_judges.get(agent)
        if expected_judge is None:
            # No declared judge => the own-judge binding cannot be checked. Refuse, never wave it
            # through: an unverifiable binding is exactly the cross-agent contamination we bar.
            raise AgentRecordMismatchError(
                f"agent {agent!r} is being rated but has no declared judge in agent_judges; the "
                f"own-judge binding cannot be verified, so the rate is refused."
            )
        if record.judge_id != expected_judge:
            raise AgentRecordMismatchError(
                f"agent {agent!r} was handed calibration record for judge {record.judge_id!r}, "
                f"but its declared judge is {expected_judge!r}; a per-agent rate must be corrected "
                f"by that agent's own judge."
            )
        if agent not in agent_verdicts:
            # No silent KeyError across the public surface: name the missing rate sample.
            raise AgentRecordMismatchError(
                f"agent {agent!r} has a calibration record but no (verdicts, rate_set_id, "
                "trace_ids) entry in agent_verdicts; every agent being rated needs its own rate "
                "sample."
            )
        verdicts, rate_set_id, trace_ids = agent_verdicts[agent]
        scored = corrected_rate(record, verdicts, rate_set_id, rate_trace_ids=trace_ids, seed=seed)
        interval = scored.corrected_rate
        if interval is None:  # corrected_rate always attaches an Interval on the success path
            raise AgentRecordMismatchError(
                f"agent {agent!r}: corrected_rate returned no interval for record "
                f"{record.calib_id!r} (unexpected)."
            )
        rates.append(AgentRate(agent=agent, interval=interval, denominator=denominator))
    return rates

relocation_check

relocation_check(before, after, *, touched_agent)

Classify a crew before/after improvement, gating on the workflow outcome rate.

Both slices are read as rates over their own traces, so slices of different sizes compare fairly, and each comparison is the 95% Newcombe interval for the after-minus-before difference of two independent proportions. The verdict is:

  • regressed -- the workflow failure rate's interval lies above 0 (a relocation flag never masks this);
  • improved -- the workflow failure rate's interval lies below 0;
  • relocation -- the workflow rate's interval straddles 0 while the touched agent's failure rate's interval lies below 0: a per-agent gain with no shown workflow gain is failure relocation, not a fix;
  • flat -- neither interval excludes 0.

relocated_from names the touched agent when (and only when) the verdict is relocation. The point rates are reported beside the intervals and decide nothing.

Raises AttributionError when either slice holds zero traces. A rate over no traces is not a rate, and the ladder below reads two of them as equal -- so an empty after-slice used to come back flat, or improved against any non-zero before-rate: a verdict about an improvement that was never measured. Same refusal run/gate.py:run_ci_gate makes over zero labels rather than reporting a vacuous pass. A failure count above its slice's trace count is refused too.

Source code in src/calibrated_agent_evaluation/attribution/relocation.py
def relocation_check(
    before: CrewOutcomes,
    after: CrewOutcomes,
    *,
    touched_agent: str,
) -> RelocationResult:
    """Classify a crew before/after improvement, gating on the workflow outcome rate.

    Both slices are read as rates over their own traces, so slices of different sizes compare
    fairly, and each comparison is the 95% Newcombe interval for the after-minus-before
    difference of two independent proportions. The verdict is:

      * ``regressed`` -- the workflow failure rate's interval lies above 0 (a relocation flag never
        masks this);
      * ``improved`` -- the workflow failure rate's interval lies below 0;
      * ``relocation`` -- the workflow rate's interval straddles 0 while the touched agent's
        failure rate's interval lies below 0: a per-agent gain with no shown workflow gain is
        failure relocation, not a fix;
      * ``flat`` -- neither interval excludes 0.

    ``relocated_from`` names the touched agent when (and only when) the verdict is ``relocation``.
    The point rates are reported beside the intervals and decide nothing.

    Raises :class:`AttributionError` when either slice holds zero traces. A rate over no traces
    is not a rate, and the ladder below reads two of them as equal -- so an empty after-slice used
    to come back ``flat``, or ``improved`` against any non-zero before-rate: a verdict about an
    improvement that was never measured. Same refusal ``run/gate.py:run_ci_gate`` makes over zero
    labels rather than reporting a vacuous pass. A failure count above its slice's trace count is
    refused too.
    """
    _refuse_empty_slice(before, "before")
    _refuse_empty_slice(after, "after")
    agent_before = before.per_agent_failures.get(touched_agent, 0)
    agent_after = after.per_agent_failures.get(touched_agent, 0)
    workflow = _delta(
        after.workflow_failures,
        after.workflow_total,
        before.workflow_failures,
        before.workflow_total,
        "workflow failures",
    )
    agent = _delta(
        agent_after, after.workflow_total, agent_before, before.workflow_total, touched_agent
    )

    relocated_from = None
    if workflow.lo > 0.0:
        verdict = "regressed"
    elif workflow.hi < 0.0:
        verdict = "improved"
    elif agent.hi < 0.0:
        verdict = "relocation"
        relocated_from = touched_agent
    else:
        verdict = "flat"

    return RelocationResult(
        workflow_outcome_rate_before=_rate(before.workflow_failures, before.workflow_total),
        workflow_outcome_rate_after=_rate(after.workflow_failures, after.workflow_total),
        per_agent_failures_before=dict(before.per_agent_failures),
        per_agent_failures_after=dict(after.per_agent_failures),
        verdict=verdict,
        relocated_from=relocated_from,
        workflow_rate_change=workflow,
        touched_agent_rate_change=agent,
    )

ratify_suggestion

ratify_suggestion(suggestion, *, ratified_by)

Return suggestion iff a human ratified it; otherwise refuse.

An LLM-proposed attribution may draft the culprit, but it is not a Label until a human accepts or overrides it. Raises RatificationError when ratified_by is empty -- an unratified LLM proposal never reaches the gold path.

Source code in src/calibrated_agent_evaluation/attribution/walkback.py
def ratify_suggestion(
    suggestion: AttributionSuggestion,
    *,
    ratified_by: str | None,
) -> AttributionSuggestion:
    """Return ``suggestion`` iff a human ratified it; otherwise refuse.

    An LLM-proposed attribution may draft the culprit, but it is not a Label until a human accepts
    or overrides it. Raises :class:`RatificationError` when ``ratified_by`` is empty -- an
    unratified LLM proposal never reaches the gold path.
    """
    if not ratified_by:
        raise RatificationError(
            "This attribution was proposed by an LLM and has not been ratified by a human. "
            "A human must accept or override the suggested culprit before it is recorded."
        )
    return suggestion

validate_walkback

validate_walkback(
    trace,
    crew,
    *,
    culprit_spans,
    manifestation_span,
    handoff_implicated=None,
    span_defect=False,
)

Validate a recorded counterfactual walk-back against the real span tree.

culprit_spans must be non-empty and every id must exist in trace; manifestation_span (if given) must exist; each culprit must be at or before the manifestation in causal order; and handoff_implicated (if given) must be a seam declared in crew.handoffs. More than one culprit yields status="disputed" -- the culprits are kept, never collapsed to one.

Raises AttributionError (recording no attribution) when a named span is absent, a culprit lies strictly after the manifestation, or the handoff is undeclared.

Source code in src/calibrated_agent_evaluation/attribution/walkback.py
def validate_walkback(
    trace: Trace,
    crew: CrewDefinition,
    *,
    culprit_spans: Sequence[str],
    manifestation_span: str | None,
    handoff_implicated: str | None = None,
    span_defect: bool = False,
) -> Attribution:
    """Validate a recorded counterfactual walk-back against the real span tree.

    ``culprit_spans`` must be non-empty and every id must exist in ``trace``; ``manifestation_span``
    (if given) must exist; each culprit must be at or before the manifestation in causal order; and
    ``handoff_implicated`` (if given) must be a seam declared in ``crew.handoffs``. More than one
    culprit yields ``status="disputed"`` -- the culprits are kept, never collapsed to one.

    Raises :class:`AttributionError` (recording no attribution) when a named span is absent, a
    culprit lies strictly after the manifestation, or the handoff is undeclared.
    """
    culprits = tuple(culprit_spans)
    if not culprits:
        raise AttributionError("A walk-back must record at least one culprit span; got none.")

    by_id = _span_index(trace.spans)
    agent_order = _agent_order(trace, crew)
    for span_id in culprits:
        _require_span(span_id, by_id, trace, role="culprit")
    if manifestation_span is not None:
        _require_span(manifestation_span, by_id, trace, role="manifestation")
        for span_id in culprits:
            _require_causal_order(span_id, manifestation_span, by_id, agent_order)

    if handoff_implicated is not None and handoff_implicated not in crew.handoffs:
        raise AttributionError(
            f"handoff_implicated {handoff_implicated!r} is not a declared seam in the crew "
            f"handoff axis {list(crew.handoffs)}."
        )

    status = "disputed" if len(culprits) > 1 else "single"
    return Attribution(
        trace_id=trace.trace_id,
        culprit_spans=culprits,
        manifestation_span=manifestation_span,
        handoff_implicated=handoff_implicated,
        status=status,
        span_defect=span_defect,
    )