calibrated_agent_evaluation.attribution¶
calibrated_agent_evaluation.attribution ¶
Crew attribution layer.
Makes a multi-agent crew evaluable per-agent, not just at the outcome: - Crew Define artifact -- expected agent set / order / rules per scenario, so a skip is a diff. - Counterfactual walk-back -- validate a recorded culprit/manifestation against the span tree (plus an opt-in LLM proposer the human ratifies). - Two-vocabulary codebook -- failure modes + responsibility boundaries, versioned. - Per-agent corrected rates -- each agent corrected by its own judge's TPR/TNR (reuses the calibration engine). - Relocation primitive -- gate crew improvement on the workflow outcome rate, not one agent's.
Pure methodology plane: reads normalized Trace/Span/Label and reuses the
calibration engine; no trace-store/executor vendor type appears here. It does NOT run LLM
judges nor cluster critiques into modes.
Public API re-exports from submodules.
RelocationVerdict
module-attribute
¶
AgentRecordMismatchError ¶
Bases: CalibratedAgentEvaluationDataError
Per-agent correction was handed another agent's judge calibration record.
AttributionError ¶
Bases: CalibratedAgentEvaluationDataError
A walk-back that doesn't fit the trace.
It names a span absent from the trace, orders the culprit after the manifestation, or implicates an undeclared handoff seam.
CodebookError ¶
Bases: CalibratedAgentEvaluationDataError
A codebook entry is missing its definition or a positive/boundary example.
CrewDefinitionError ¶
Bases: CalibratedAgentEvaluationDataError
An invalid crew Define artifact, or a scenario with no declared expected process.
AgentRate
dataclass
¶
One agent's Rogan-Gladen-corrected rate.
interval is the Interval (corrected
point and its interval, or a refusal reason) produced by that agent's own judge; rows do not
share a fate. denominator states whether this is a share-of-failures or a rate-over-traffic
number, so a discovery-pile rate is never quoted as the customer SLA.
AgreementReport
dataclass
¶
AgreementReport(
outcome_kappa,
culprit_kappa,
n_dual_coded,
outcome_undefined_reason=None,
culprit_undefined_reason=None,
n_culprit_coded=0,
)
Per-axis inter-coder agreement over a dual-coded subset.
A high outcome_kappa with a low culprit_kappa is the most useful multiagent signal; a
fuzzy agent/handoff responsibility boundary (a real product bug), not labeler noise.
Each κ carries its 95% interval. An axis on which both coders used one and the same value
throughout has κ = 0/0: its field is None and its *_undefined_reason says so, rather
than the conventional 1.0 that would read as proven agreement. n_dual_coded counts the
traces both coders labeled (the outcome axis); n_culprit_coded the ones both called FAIL
(the culprit axis).
Attribution
dataclass
¶
Attribution(
trace_id,
culprit_spans,
manifestation_span,
handoff_implicated,
status,
span_defect=False,
)
A validated counterfactual walk-back over the span tree.
culprit_spans has ≥1 entry; more than one ⇒ status="disputed" (an undefined
responsibility boundary, recorded; never collapsed to a single culprit). span_defect is
the outcome-pass / span-fail flag: a defect recorded independently of the trace
outcome, supplying per-span gold.
AttributionSuggestion
dataclass
¶
An LLM proposal for a crew FAIL: NEVER a Label until a human ratifies it.
critique precedes the (human-ratified) verdict, mirroring the review-UI Suggestion.
Whatever produced it,
ratify_suggestion refuses it until a
named human ratifies it. No proposer ships: the WalkBackProposer protocol and its
NoProposer default were removed, since nothing implemented or consumed them and no planned
feature owns one.
Codebook
dataclass
¶
A crew's two versioned vocabularies: failure modes and agent or handoff boundaries.
A version bump flags labels written under the prior version for re-labeling.
CodebookEntry
dataclass
¶
One versioned vocabulary entry.
It needs a definition, at least one positive example and at least one boundary example, and is
registered through register_entry.
CrewDefinition
dataclass
¶
The crew Define artifact.
Built + validated by
build_crew_definition (a skipped
agent is a diff against scenarios[...].expected_agents; a violated boundary is a rules
clause). handoffs are declared seams like "intake->policy-checker".
CrewOutcomes
dataclass
¶
A before/after snapshot for the relocation check.
workflow_failures / workflow_total give the crew's workflow outcome rate (the gate
metric); per_agent_failures counts each agent's span-level failures over the same slice.
ExpectedProcess
dataclass
¶
ExpectedProcess(
scenario,
expected_agents,
terminal_decision,
payout_allowed,
escalation_required,
)
Per-scenario expected process: the diff target for the process judge.
expected_agents is the valid ordering (a sequence, not just a set); a skipped/mis-routed
agent is only detectable as expected minus actual, so it must exist before any trace.
MandatedRule
dataclass
¶
A named responsibility boundary: antecedent ⇒ required consequent agent.
deterministic=False marks a rule the process judge cannot decide by agent set-difference
alone; it must be routed to a calibrated sub-judge.
RelocationResult
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
The before/after crew-improvement verdict.
verdict="relocation" when the touched agent's failure rate fell (its interval below 0) but
the workflow outcome rate was not shown to improve; a per-agent gain with a flat outcome rate is
failure relocation, not a fix. workflow_rate_change and touched_agent_rate_change are
the after-minus-before intervals the verdict read; None on results made before they were
recorded.
Config:
frozen:Trueextra:forbid
Fields:
-
workflow_outcome_rate_before(float) -
workflow_outcome_rate_after(float) -
per_agent_failures_before(dict[str, int]) -
per_agent_failures_after(dict[str, int]) -
verdict(RelocationVerdict) -
relocated_from(str | None) -
workflow_rate_change(Interval | None) -
touched_agent_rate_change(Interval | None)
attribution_agreement ¶
Return per-axis (outcome, culprit) inter-coder agreement, or None if none dual-coded.
Labels are trace-level and matched on trace_id. The outcome axis compares outcome over
every trace both coders labeled. The culprit axis compares culprit_span over the traces
both coders called FAIL: a PASS has no culprit, and counting the PASS traces where both
culprits are None would carry the outcome agreement into the culprit axis (80 shared PASS
traces and 20 FAILs with culprits picked at random read as culprit kappa 0.54 instead of about
0). Returns:
- an
AgreementReport(outcome_kappa/culprit_kappa/n_dual_coded/n_culprit_coded) when at least one trace was co-labeled, each kappa with its 95% interval orNoneplus a reason when it is 0/0 or has no traces on that axis; Nonewhen the coders share no trace -- there is nothing to agree on, so no figure is fabricated from a single coder (the capability-absent path).
Raises:
| Type | Description |
|---|---|
AttributionError
|
A coder passed a span-level label or labeled a trace twice. |
Source code in src/calibrated_agent_evaluation/attribution/agreement.py
bump_version ¶
Return the re-versioned codebook and the labels to flag for re-labeling.
Every Label whose codebook_version differs from new_version (i.e. was written under a
superseded version, including None) is returned in the flag list -- a definition change is
retroactive, so labeled history must be re-checked against the new definition. Labels already at
new_version are not flagged. change_note is appended to the changelog for audit.
Source code in src/calibrated_agent_evaluation/attribution/codebook.py
register_entry ¶
Return a new codebook with entry appended to the named vocabulary.
Raises CodebookError if the entry has an empty definition, or fewer than one positive
or one boundary example -- the examples are what make a category ratifiable, so an
entry without them is refused rather than stored half-defined. Also refuses a name already
present in the target vocabulary.
Source code in src/calibrated_agent_evaluation/attribution/codebook.py
build_crew_definition ¶
Build a validated crew Define artifact.
Raises CrewDefinitionError if the crew declares no agents, a scenario declares an empty
expected agent set, a scenario references an agent not in agents, a mandated rule's
consequent_agent is not a declared agent, or a handoff is not a well-formed a->b over
two declared agents. No silent fall-back; an invalid Define artifact refuses loudly.
Source code in src/calibrated_agent_evaluation/attribution/definition.py
require_expected_process ¶
Return the ExpectedProcess for scenario, or refuse.
Raises CrewDefinitionError if the scenario declares no expected process; the process
judge cannot diff expected minus actual without it, and an undeclared boundary is a defect.
Source code in src/calibrated_agent_evaluation/attribution/definition.py
attributed_agent ¶
The agent label attributes its verdict to, or None for a trace-level label.
The source span is the human's culprit_span, or for a judge or scorer verdict the span it
judged. Its agent is read from agents when the span is in scope, since a span id need not
spell its agent; otherwise from the id convention "CLM-4471/intake" (the segment after the
last /). A re-executed node's occurrence suffix ("CLM-4471/intake#2") is stripped:
both executions are the same agent, and keying them apart would split one agent's rate.
Source code in src/calibrated_agent_evaluation/attribution/labels.py
span_agents ¶
(trace_id, span_id) -> agent for every span of traces.
per_agent_rates ¶
Return one independent AgentRate per agent.
agent_records maps each agent to its own judge's certified CalibrationRecord;
agent_verdicts maps each agent to (verdicts, rate_set_id, trace_ids) -- that agent's
judge verdicts (1=FAIL, 0=PASS) over a production-representative sample, the id of that
sample, and the trace id of each verdict (checked against the calibration gold).
agent_judges maps each agent to the judge_id that legitimately judges it.
Each agent's rate is corrected via corrected_rate (Rogan-Gladen with its interval); the
resulting Interval is attached with the
stated denominator. The calibration refusals propagate unchanged -- an uncertified agent
judge raises UncalibratedJudgeError; a J<0.5 or non-identifiable correction yields an
interval whose point is None (the widening bound is the signal), never a silently fabricated
number.
Raises AgentRecordMismatchError if an agent has no declared judge in agent_judges
(the binding cannot be verified, so it is refused rather than silently trusted), or if an
agent's record's judge_id is not that declared judge -- a policy-checker rate must not be
computed from intake's judge.
Source code in src/calibrated_agent_evaluation/attribution/rates.py
relocation_check ¶
Classify a crew before/after improvement, gating on the workflow outcome rate.
Both slices are read as rates over their own traces, so slices of different sizes compare fairly, and each comparison is the 95% Newcombe interval for the after-minus-before difference of two independent proportions. The verdict is:
regressed-- the workflow failure rate's interval lies above 0 (a relocation flag never masks this);improved-- the workflow failure rate's interval lies below 0;relocation-- the workflow rate's interval straddles 0 while the touched agent's failure rate's interval lies below 0: a per-agent gain with no shown workflow gain is failure relocation, not a fix;flat-- neither interval excludes 0.
relocated_from names the touched agent when (and only when) the verdict is relocation.
The point rates are reported beside the intervals and decide nothing.
Raises AttributionError when either slice holds zero traces. A rate over no traces
is not a rate, and the ladder below reads two of them as equal -- so an empty after-slice used
to come back flat, or improved against any non-zero before-rate: a verdict about an
improvement that was never measured. Same refusal run/gate.py:run_ci_gate makes over zero
labels rather than reporting a vacuous pass. A failure count above its slice's trace count is
refused too.
Source code in src/calibrated_agent_evaluation/attribution/relocation.py
ratify_suggestion ¶
Return suggestion iff a human ratified it; otherwise refuse.
An LLM-proposed attribution may draft the culprit, but it is not a Label until a human accepts
or overrides it. Raises RatificationError when ratified_by is empty -- an
unratified LLM proposal never reaches the gold path.
Source code in src/calibrated_agent_evaluation/attribution/walkback.py
validate_walkback ¶
validate_walkback(
trace,
crew,
*,
culprit_spans,
manifestation_span,
handoff_implicated=None,
span_defect=False,
)
Validate a recorded counterfactual walk-back against the real span tree.
culprit_spans must be non-empty and every id must exist in trace; manifestation_span
(if given) must exist; each culprit must be at or before the manifestation in causal order; and
handoff_implicated (if given) must be a seam declared in crew.handoffs. More than one
culprit yields status="disputed" -- the culprits are kept, never collapsed to one.
Raises AttributionError (recording no attribution) when a named span is absent, a
culprit lies strictly after the manifestation, or the handoff is undeclared.