Skip to content

Evaluate a multi-agent crew

In a crew, the agent that causes a failure is often not the one where it shows up. Label both, and your fix goes to the agent that caused the harm, and improve can tell when a fix only moved the failure to a later agent.

The examples on this page use ClaimsCrew, an insurance-claims crew in the sample corpus: a supervisor hands a claim to intake, a policy checker, a fraud scorer and payout.

Where the harm appears and what caused it

On claim CLM-4471 the customer writes that they have been in Florida for the last month, which matters because the policy excludes damage to a home left empty that long. The intake agent drops that fact and passes occupancy_status: null on. Every later agent does its job correctly with what it was given, and payout pays a claim it should have denied.

The harm appears at payout. It was caused at intake. Here is the trace, with the culprit drawn with a solid red border and the span where the harm appeared with a dashed one:

flowchart TD
  supervisor["supervisor<br/><small>supervisor</small>"]
  intake["intake<br/><small>culprit</small>"]
  policy_checker["policy-checker<br/><small>specialist</small>"]
  fraud_scorer["fraud-scorer<br/><small>specialist</small>"]
  payout["payout<br/><small>manifestation</small>"]
  supervisor --> intake
  supervisor --> policy_checker
  supervisor --> fraud_scorer
  supervisor --> payout
  intake -. handoff .-> policy_checker
  policy_checker -. handoff .-> fraud_scorer
  fraud_scorer -. handoff .-> payout
  classDef culprit stroke:#c0392b,stroke-width:3px
  classDef manifestation stroke:#776c59,stroke-width:3px,stroke-dasharray:4 3
  class intake culprit
  class payout manifestation

When you label a crew trace FAIL in the review app, the form asks for two spans: the one that caused the failure (the culprit) and the one where it showed up (the manifestation). If the handoff between two agents is to blame, you can name the handoff too.

Check the labels against the trace

Every culprit a reviewer names is checked against the real span tree. The label is rejected with a message if it names a span that isn't in the trace, blames a span that ran after the harm appeared, or names a handoff the crew doesn't have. This program checks three ClaimsCrew failures:

# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
# SPDX-License-Identifier: Apache-2.0

"""Who broke the claim? Attribute crew failures to the agent that caused them.

Reads three ClaimsCrew traces and their gold labels from the repo's fixtures. For each failed
claim, the walk-back starts at the span where the harm appeared (the manifestation) and checks the
culprit the reviewer recorded against the real span tree. CLM-4477 has two defensible culprits, so
it is recorded as disputed. Run from the repository root.

    uv run --extra stats python docs/site/examples/crew_attribution.py
"""

from __future__ import annotations

import json
from pathlib import Path

from calibrated_agent_evaluation.attribution import (
    CrewDefinition,
    ExpectedProcess,
    MandatedRule,
    build_crew_definition,
    validate_walkback,
)
from calibrated_agent_evaluation.schema.serde import label_from_dict, trace_from_dict

FIXTURES = Path("fixtures")


def claimscrew() -> CrewDefinition:
    """The crew's agents, handoffs and the expected process for a vacancy-exclusion claim."""
    vacancy = ExpectedProcess(
        scenario="S2_vacancy_exclusion",
        expected_agents=("supervisor", "intake", "policy-checker", "fraud-scorer", "payout"),
        terminal_decision="deny",
        payout_allowed=False,
        escalation_required=False,
    )
    return build_crew_definition(
        agents=("supervisor", "intake", "policy-checker", "fraud-scorer", "payout", "escalation"),
        handoffs=(
            "intake->policy-checker",
            "policy-checker->fraud-scorer",
            "fraud-scorer->payout",
            "fraud-scorer->escalation",
        ),
        scenarios={"S2_vacancy_exclusion": vacancy},
        rules=(
            MandatedRule("R2", "fraud flag raised", "escalation", deterministic=False),
            MandatedRule("R3", "occupancy narrative present", "intake"),
        ),
    )


crew = claimscrew()
gold = {
    (label.trace_id, label.span_id): label
    for label in map(label_from_dict, json.loads((FIXTURES / "labels" / "gold.json").read_text()))
}

for trace_id in ("CLM-4471", "CLM-4473", "CLM-4477"):
    trace = trace_from_dict(json.loads((FIXTURES / "normalized" / f"{trace_id}.json").read_text()))
    label = gold[(trace_id, None)]
    if label.culprit_span == "disputed":
        # The reviewer could not pick between the two agents, so both go forward as culprits.
        culprits: tuple[str, ...] = (f"{trace_id}/intake", f"{trace_id}/policy-checker")
    else:
        culprits = (label.culprit_span,) if label.culprit_span else ()
    attribution = validate_walkback(
        trace,
        crew,
        culprit_spans=culprits,
        manifestation_span=label.manifestation_span,
        handoff_implicated=label.handoff_implicated,
    )
    print(f"{trace_id} ({label.failure_mode}): {attribution.status}")
    print(f"  harm appeared at  {attribution.manifestation_span}")
    print(f"  caused by         {', '.join(attribution.culprit_spans)}")
    if label.handoff_implicated:
        print(f"  handoff           {label.handoff_implicated}")
CLM-4471 (dropped_material_fact): single
  harm appeared at  CLM-4471/payout
  caused by         CLM-4471/intake
  handoff           intake->policy-checker
CLM-4473 (trusted_broken_policy_tool): single
  harm appeared at  CLM-4473/payout
  caused by         CLM-4473/policy-checker
CLM-4477 (vacancy_threshold_margin_mishandled): disputed
  harm appeared at  CLM-4477/payout
  caused by         CLM-4477/intake, CLM-4477/policy-checker
  handoff           intake->policy-checker

CLM-4477 is disputed. Its reviewer could not choose between intake and the policy checker, so the label keeps both as culprits and marks it disputed.

Gate the crew

The CI gate counts each verdict under its failure mode and the agent of its culprit span (for a judge or scorer, the span it judged), so a crew's modes are gated per agent: intake/dropped_material_fact and policy-checker/trusted_broken_policy_tool pass or fail separately (the CI job page shows this output). A verdict that names no span counts under its mode alone, so to gate the crew's outcome, label or judge a workflow-level mode such as wrong_claim_decision on the whole trace. In the library, calibrated_agent_evaluation.attribution also computes a corrected failure rate per agent, for finding where to look.

A fix that moves the failure

A change to one agent can make that agent look better while the crew does no better: the failure moves to a later agent. calibrated-agent-evaluation improve checks for this. Both fixes below take a held-out set of 40 traces from 18 failures to 4. The OrderBot fix is accepted. The ClaimsCrew fix to intake is rejected, because intake fails less but the crew's outcome rate did not improve:

OrderBot fix: accepted. The failure rate dropped by at least the registered amount, more than chance would explain.

ClaimsCrew fix to intake: rejected. The agent it touched fails less, but the crew's outcome did not improve: the failures moved instead of going away.
The same verdicts, with the test behind them
OrderBot fix: accepted
  McNemar (midp) p = 6.1e-05
  change in failure rate -0.350

ClaimsCrew fix to intake: rejected
  McNemar (midp) p = 6.1e-05
  change in failure rate -0.350
  reason: failure relocation: the touched agent's failures fell but the workflow outcome rate did not improve (the failures moved from 'intake')

The program that produced this is docs/site/examples/improve_guard.py in the repository. Check that a fix worked explains what improve checks.

Evaluate a crew fix from two runs

To check a real fix, run the same claims through the crew before and after the change, into two local stores, and label one workflow failure mode in both, such as wrong_claim_decision. Each FAIL names its culprit span as usual. Then describe the crew in a file:

{
  "agents": [
    "supervisor",
    "intake",
    "policy-checker",
    "fraud-scorer",
    "payout",
    "escalation"
  ],
  "handoffs": [
    "intake->policy-checker",
    "policy-checker->fraud-scorer",
    "fraud-scorer->payout",
    "fraud-scorer->escalation"
  ]
}

Register the metric with --crew before the fix, and evaluate it with the crew file (--crew-file) and the agent you changed (--touched-agent). In this example, intake caused 14 of 40 wrong decisions before the fix. After it, intake causes 2 and the policy checker 12, on the same claims:

$ calibrated-agent-evaluation --root store improve --register wrong_claim_decision --min-reduction 0.05 --crew
Pre-registered 'wrong_claim_decision': the fix must cut its failure rate by at least 5 percentage points. The threshold is now frozen.
Next: after the fix, compare the run before it with the run after it: `calibrated-agent-evaluation improve --evaluate wrong_claim_decision --before-root A --after-root B --development FILE --crew-file CREW.json --touched-agent NAME`.
Run with --explain for the numbers behind this.
(exit 0)

$ calibrated-agent-evaluation --root store improve --evaluate wrong_claim_decision --crew-file crew.json --touched-agent intake --verdicts human --before-root before --after-root after --development dev.txt
calibrated-agent-evaluation improve: Fix not accepted: the agent you changed fails less, but the crew as a whole doesn't: the failures moved to another step instead of going away.
Next: check where the failures went before shipping the change.
Not measured by the human verdicts, so not guarded: escalation, fraud-scorer->escalation.
Not checked: whether the target was registered before the after run began (no trace in after records a time).
Run with --explain for the numbers behind this.
(exit 1)

The output also lists what the guard could not check. Escalation never runs on these claims, so the reviewer's labels say nothing about it or the handoff into it. Those two slices stay in the output and are listed as not measured. Judge verdicts leave more unmeasured: a judge writes no handoff, and an agent with no span judge has no verdicts.

The same evaluation, with the counts and the test behind it
$ calibrated-agent-evaluation --root store improve --register wrong_claim_decision --min-reduction 0.05 --crew
Pre-registered 'wrong_claim_decision': the fix must cut its failure rate by at least 5 percentage points. The threshold is now frozen.
Next: after the fix, compare the run before it with the run after it: `calibrated-agent-evaluation improve --evaluate wrong_claim_decision --before-root A --after-root B --development FILE --crew-file CREW.json --touched-agent NAME`.
Run with --explain for the numbers behind this.
(exit 0)

$ calibrated-agent-evaluation --root store improve --evaluate wrong_claim_decision --crew-file crew.json --touched-agent intake --verdicts human --before-root before --after-root after --development dev.txt --explain
calibrated-agent-evaluation improve: Fix not accepted: the agent you changed fails less, but the crew as a whole doesn't: the failures moved to another step instead of going away.
Next: check where the failures went before shipping the change.
Not measured by the human verdicts, so not guarded: escalation, fraud-scorer->escalation.
Not checked: whether the target was registered before the after run began (no trace in after records a time).

improve 'wrong_claim_decision': REJECTED
  paired fail-rate change (after - before) +0.000, 95% CI [-0.054, +0.054] (newcombe_paired_hybrid_score, n=40); McNemar undefined, 0 discordant pair(s), no p-value
  pre-registered minimum reduction 0.05: the interval's upper bound must be at or below -0.050
  guard panel: 9 slice(s), Holm; regressed: policy-checker
  zero discordant pairs (no before/after flips); McNemar's test is undefined
  reason: paired test undefined (no discordant pairs) -- no improvement shown
  reason: paired effect upper bound +0.0537 does not clear the pre-registered reduction in fail rate of 0.05 (needs <= -0.05)
  reason: guard slices regressed: policy-checker
  reason: failure relocation: the touched agent's failures fell but the workflow outcome rate did not improve (the failures moved from 'intake')
  paired 40 trace(s) by id between before (before) and after (after), human verdicts
  pre-registration frozen at <time>; not checked against the after run: no trace in after records a time
  crew crew.json (sha256 17d3611be14e), touched agent 'intake': workflow failures 14 of 40 before, 14 of 40 after
  relocation check: relocation
    agent supervisor: 0 failed trace(s) before, 0 after
    agent intake: 14 failed trace(s) before, 2 after (touched)
    agent policy-checker: 0 failed trace(s) before, 12 after
    agent fraud-scorer: 0 failed trace(s) before, 0 after
    agent payout: 0 failed trace(s) before, 0 after
    handoff intake->policy-checker: 0 failed trace(s) before, 0 after
    handoff policy-checker->fraud-scorer: 0 failed trace(s) before, 0 after
    handoff fraud-scorer->payout: 0 failed trace(s) before, 0 after
  guard slices:
    supervisor: fail rate 0.000 [0.000, 0.088] (n=40) before, 0.000 [0.000, 0.088] (n=40) after; one-sided McNemar regression p=1
    policy-checker: fail rate 0.000 [0.000, 0.088] (n=40) before, 0.300 [0.181, 0.454] (n=40) after; one-sided McNemar regression p=0.0001221
    fraud-scorer: fail rate 0.000 [0.000, 0.088] (n=40) before, 0.000 [0.000, 0.088] (n=40) after; one-sided McNemar regression p=1
    payout: fail rate 0.000 [0.000, 0.088] (n=40) before, 0.000 [0.000, 0.088] (n=40) after; one-sided McNemar regression p=1
    escalation (agent): not measured by the human verdicts; held at p=1
    intake->policy-checker: fail rate 0.000 [0.000, 0.088] (n=40) before, 0.000 [0.000, 0.088] (n=40) after; one-sided McNemar regression p=1
    policy-checker->fraud-scorer: fail rate 0.000 [0.000, 0.088] (n=40) before, 0.000 [0.000, 0.088] (n=40) after; one-sided McNemar regression p=1
    fraud-scorer->payout: fail rate 0.000 [0.000, 0.088] (n=40) before, 0.000 [0.000, 0.088] (n=40) after; one-sided McNemar regression p=1
    fraud-scorer->escalation (handoff): not measured by the human verdicts; held at p=1
recorded improve-<id> in store/records/improve
(exit 1)

Failures count per trace. A trace counts once for an agent when a FAIL on it blames that agent: a reviewer's label through its culprit span, a judge's verdict through the span it judged. This is the same rule calibrated-agent-evaluation run uses for per-agent rates. A FAIL whose culprit is disputed, like CLM-4477's, counts for no agent but still counts as a wrong decision, and the output names the trace. The program that produced this is docs/site/examples/guides/improve_crew.py.

In the library

Crew attribution is calibrated_agent_evaluation.attribution: build_crew_definition, validate_walkback, per-agent corrected rates and the relocation check. See the calibrated_agent_evaluation.attribution reference.

Next