Skip to content

calibrated_agent_evaluation.flagship

calibrated_agent_evaluation.flagship

The flagship proof harness -- the end-to-end proof of the whole pipeline.

Runs both reference agents (OrderBot single-agent, ClaimsCrew crew) end-to-end through all eight pipeline stages (probe -> define -> generate -> label -> cluster -> judge -> run -> improve) with calibrated, bias-gated judges, in both worlds (local + self-hosted Langfuse; cassette-replay + live Bedrock), and emits one reconstructable FlagshipReport. It is a pure orchestration feature: it composes the shipped calibrated_agent_evaluation.* surfaces and introduces no estimator, gate, judge, scorer, or statistic. The corpus-specific binding (which trace/cassette/judge/crew per agent) is injected by the caller as an AgentPlan, so this package imports only calibrated_agent_evaluation.* (no scripts/examples) and no vendor SDK type. ReferenceCorpus builds the two reference plans from a source checkout's fixtures/ tree, and run_reference_proof runs them and writes the record.

Public surface

  • prove -- run the whole proof for a FlagshipConfig, return a FlagshipReport.
  • run_agent_pipeline -- run the eight stages for one (agent, world) cell.
  • run_reference_proof -- prove the reference agents and write the flagship record.
  • ReferenceCorpus -- the reference agents' plans over a checkout's fixtures/.
  • compare_oracle -- compare a stage's output against its committed corpus oracle.
  • AgentPlan -- the injected per-agent corpus binding the orchestrator drives.
  • the frozen report types + the self-contained CSV+JSON codec.

CANONICAL_STAGES module-attribute

CANONICAL_STAGES = (
    "probe",
    "define",
    "generate",
    "label",
    "cluster",
    "judge",
    "run",
    "improve",
)

FLAGSHIP_SCHEMA_VERSION module-attribute

FLAGSHIP_SCHEMA_VERSION = '4'

Agent module-attribute

Agent = str

ModelPlane module-attribute

ModelPlane = Literal['cassette', 'live']

StageName module-attribute

StageName = Literal[
    "probe",
    "define",
    "generate",
    "label",
    "cluster",
    "judge",
    "run",
    "improve",
]

Verdict module-attribute

Verdict = Literal['proven', 'unpinned', 'failed']

World module-attribute

World = Literal['local', 'langfuse']

REFERENCE_AGENTS module-attribute

REFERENCE_AGENTS = ('orderbot', 'claimscrew')

The reference agents the committed corpus carries plans for.

FlagshipError

Bases: CalibratedAgentEvaluationDataError

Base class for every flagship-harness refusal.

Raised when the proof cannot be trusted: a seeded fixture version that does not match the running SCHEMA_VERSION, a missing/corrupt oracle for a stage that claims one, a --live model that is unreachable/unconfigured (re-raised from the executor's ExecutionError so the harness never silently replays the cassette and reports a live pass), or a stage's shipped feature raising a typed domain error mid-proof. It is a refusal to declare the proof complete, not a bug in a composed stage.

FlagshipStoreError

Bases: FlagshipError

A FlagshipReport dump/load file read or write failed.

Wraps the underlying OSError / json.JSONDecodeError / csv failure so a filesystem/parse error surfaces inside the typed family rather than as a raw stdlib exception escaping the store's public surface (wrap external errors at the boundary). A stored schema_version != the running FLAGSHIP_SCHEMA_VERSION also refuses here (no coercion).

AgentPlan dataclass

AgentPlan(agent, stages, comparable, oracle_required=True)

The injected per-agent corpus binding the orchestrator drives.

stages is the eight StagePlans in canonical order; the orchestrator refuses a plan that does not cover all eight canonical stages (the completeness invariant). comparable reads the world-invariant artifacts (the normalized trace/label/score dicts) the cross-world parity assertion compares.

oracle_required (default True) keeps the hard refusal: a stage naming an oracle that is absent is a proof-integrity failure. A plan for an agent with no hand-authored oracle corpus sets it False, and each absent-oracle stage is recorded as unpinned instead, which makes the report's verdict unpinned rather than proven. A present but corrupt oracle still raises.

agent instance-attribute

agent

stages instance-attribute

stages

comparable instance-attribute

comparable

oracle_required class-attribute instance-attribute

oracle_required = True

StageContext dataclass

StageContext(
    agent, world, model_plane, backend, fixtures_root
)

What a stage callable is handed: the world's backend + run coordinates.

A stage reads/writes through backend (the trace-store protocol -- local or Langfuse, chosen by the orchestrator so the stage never branches on world), and consults model_plane to pick cassette-replay vs live for the model-touching stages.

agent instance-attribute

agent

world instance-attribute

world

model_plane instance-attribute

model_plane

backend instance-attribute

backend

fixtures_root instance-attribute

fixtures_root

StagePlan dataclass

StagePlan(stage, run)

One stage's binding: the canonical stage name + the callable that runs it.

stage instance-attribute

stage

run instance-attribute

run

StageResult dataclass

StageResult(
    produced,
    oracle_path=None,
    headline=None,
    artifact_ids=(),
    gate=None,
    note="",
)

What a stage callable returns -- the produced fields the oracle pins + optional headline.

produced is the subset of fields the committed oracle constrains (compared semantically); oracle_path is None for an oracle-less stage proven structurally (the note records the structural check). headline is the calibration Interval where the stage emits a rate. gate carries the three-gate judge-trust decision for the judge stage.

produced instance-attribute

produced

oracle_path class-attribute instance-attribute

oracle_path = None

headline class-attribute instance-attribute

headline = None

artifact_ids class-attribute instance-attribute

artifact_ids = ()

gate class-attribute instance-attribute

gate = None

note class-attribute instance-attribute

note = ''

ReferenceCorpus

ReferenceCorpus(fixtures_root)

The committed reference corpus under one fixtures/ directory, and its two plans.

Bind to a corpus.

Parameters:

Name Type Description Default
fixtures_root str | Path

The checkout's fixtures/ directory.

required

Raises:

Type Description
FlagshipError

The directory does not hold the corpus (no normalized/).

Source code in src/calibrated_agent_evaluation/flagship/reference.py
def __init__(self, fixtures_root: str | Path) -> None:
    """Bind to a corpus.

    Args:
        fixtures_root: The checkout's ``fixtures/`` directory.

    Raises:
        FlagshipError: The directory does not hold the corpus (no ``normalized/``).
    """
    self.root = Path(fixtures_root)
    if not (self.root / "normalized").is_dir():
        raise FlagshipError(
            f"{self.root} is not the reference corpus (no normalized/ directory); the "
            "flagship proof runs from a source checkout's fixtures/ tree"
        )
    self._gold_rows: list[dict[str, Any]] | None = None

root instance-attribute

root = Path(fixtures_root)

trace

trace(trace_id)

A normalized corpus trace.

Source code in src/calibrated_agent_evaluation/flagship/reference.py
def trace(self, trace_id: str) -> Trace:
    """A normalized corpus trace."""
    try:
        return trace_from_dict(self._read_json(Path("normalized") / f"{trace_id}.json"))
    except CalibratedAgentEvaluationDataError as exc:
        raise FlagshipError(f"corpus trace {trace_id!r} is invalid: {exc}") from exc

gold_rows

gold_rows()

The committed gold label rows, as stored.

Source code in src/calibrated_agent_evaluation/flagship/reference.py
def gold_rows(self) -> list[dict[str, Any]]:
    """The committed gold label rows, as stored."""
    rows = self._gold_rows
    if rows is None:
        rows = self._gold_rows = self._read_json(Path("labels") / "gold.json")
    return rows

gold_labels

gold_labels()

The gold labels keyed by (trace_id, span_id).

Source code in src/calibrated_agent_evaluation/flagship/reference.py
def gold_labels(self) -> dict[tuple[str, str | None], Label]:
    """The gold labels keyed by ``(trace_id, span_id)``."""
    labels = [label_from_dict(row) for row in self.gold_rows()]
    return {(label.trace_id, label.span_id): label for label in labels}

orderbot_judge

orderbot_judge()

The OrderBot judge, read from the same judge file judge --judge reads.

Source code in src/calibrated_agent_evaluation/flagship/reference.py
def orderbot_judge(self) -> ExecutableJudge:
    """The OrderBot judge, read from the same judge file ``judge --judge`` reads."""
    from calibrated_agent_evaluation.cli.errors import CliError
    from calibrated_agent_evaluation.cli.judgefile import load_judge_file

    try:
        return load_judge_file(self.root / ORDERBOT_JUDGE_FILE).judge
    except CliError as exc:
        raise FlagshipError(f"cannot read the OrderBot judge: {exc}") from exc

orderbot_plan

orderbot_plan(*, calibration, battery)

The eight stages for OrderBot, its judge admitted on calibration and battery.

Parameters:

Name Type Description Default
calibration CalibrationRecord

The calibration record the judge stage admits the judge on.

required
battery BiasBatteryRecord

The bias battery the judge stage admits the judge on.

required
Source code in src/calibrated_agent_evaluation/flagship/reference.py
def orderbot_plan(
    self, *, calibration: CalibrationRecord, battery: BiasBatteryRecord
) -> AgentPlan:
    """The eight stages for OrderBot, its judge admitted on ``calibration`` and ``battery``.

    Args:
        calibration: The calibration record the judge stage admits the judge on.
        battery: The bias battery the judge stage admits the judge on.
    """
    judge = self.orderbot_judge()
    model = CassetteJudgeModel(
        Cassette(self.root / _ORDERBOT_CASSETTE, mode=Mode.REPLAY), supports_schema=True
    )
    expected = self.root / "expected"
    stages = (
        StagePlan(
            "probe",
            probe_stage(
                ConnectionConfig(endpoint="https://stub.local/invoke"),
                _probe_transport,
                str(expected / "probe" / "orderbot_capability.json"),
            ),
        ),
        StagePlan(
            "define",
            structural_stage(
                "crew/taxonomy define is a no-op for the single-agent OrderBot (one span)",
                lambda _ctx: {"defined": True},
            ),
        ),
        StagePlan(
            "generate",
            structural_stage(
                "inputs-only synthesis is agent-agnostic; the proof reuses the corpus trace",
                lambda _ctx: {"generated": True},
            ),
        ),
        self._label("ORD-1001"),
        self._cluster("ORD", "orderbot.json"),
        StagePlan(
            "judge",
            judge_stage(
                judge,
                calibration,
                battery,
                (workflow_target(self.trace("ORD-1001")),),
                model,
            ),
        ),
        self._run_gate(),
        # The paired oracle is compared end to end by the tests, not field-pinned here.
        StagePlan("improve", improve_stage(_orderbot_improve_request(), oracle_path=None)),
    )
    return AgentPlan(
        agent="orderbot",
        stages=stages,
        comparable=lambda ctx: _trace_doc(_put_and_read(ctx, self.trace("ORD-1001"))),
    )

claimscrew_plan

claimscrew_plan()

The eight stages for ClaimsCrew; its judge stage is the attribution walk-back.

Source code in src/calibrated_agent_evaluation/flagship/reference.py
def claimscrew_plan(self) -> AgentPlan:
    """The eight stages for ClaimsCrew; its judge stage is the attribution walk-back."""
    walkbacks = [
        (
            self.trace("CLM-4471"),
            ("CLM-4471/intake",),
            "CLM-4471/payout",
            "intake->policy-checker",
        ),
        (self.trace("CLM-4473"), ("CLM-4473/policy-checker",), "CLM-4473/payout", None),
        (
            self.trace("CLM-4477"),
            ("CLM-4477/intake", "CLM-4477/policy-checker"),
            "CLM-4477/payout",
            "intake->policy-checker",
        ),
    ]
    stages = (
        StagePlan(
            "probe",
            structural_stage(
                "probing is the entry for an agent you do not own; the crew is normalized",
                lambda _ctx: {"owned": True},
            ),
        ),
        StagePlan(
            "define",
            structural_stage(
                "crew Define artifact built (agents, handoffs, expected process)",
                lambda _ctx: {"defined": True},
            ),
        ),
        StagePlan(
            "generate",
            structural_stage(
                "inputs-only synthesis is agent-agnostic; the proof reuses the crew traces",
                lambda _ctx: {"generated": True},
            ),
        ),
        self._label("CLM-4471"),
        self._cluster("CLM", "claimscrew.json"),
        StagePlan(
            "judge",
            attribution_stage(
                claimscrew_definition(),
                walkbacks,
                oracle_path=str(self.root / "expected" / "attribution" / "claimscrew.json"),
            ),
        ),
        self._run_gate(),
        # The relocation verdict is compared end to end by the tests, not field-pinned here.
        StagePlan("improve", improve_stage(claimscrew_improve_request(), oracle_path=None)),
    )
    return AgentPlan(
        agent="claimscrew",
        stages=stages,
        comparable=lambda ctx: _trace_doc(_put_and_read(ctx, self.trace("CLM-4471"))),
    )

FlagshipConfig pydantic-model

Bases: CalibratedAgentEvaluationModel

The input to prove.

agents selects which agents to prove (each needs a registered AgentPlan), worlds selects the trace-store axis (local, and langfuse when configured), model_plane selects the model axis (cassette default/CI vs live sign-off), and fixtures_root is the seeded corpus root whose VERSION is checked against the running SCHEMA_VERSION.

Config:

  • frozen: True
  • extra: forbid

Fields:

agents pydantic-field

agents

worlds pydantic-field

worlds

model_plane pydantic-field

model_plane

fixtures_root pydantic-field

fixtures_root

FlagshipReport pydantic-model

Bases: CalibratedAgentEvaluationModel

The single reconstructable proof artifact.

verdict == "proven" iff every StageOutcome.ok is True for every configured (agent, world) AND every WorldParity.equal is True AND every GateDecisionRef used to score was fully admitted; otherwise failed with reasons naming the offending stage/parity/gate. Round-trips through the self-contained store codec; load refuses on a schema_version mismatch.

Config:

  • frozen: True
  • extra: forbid

Fields:

Validators:

  • _check_invariants

flagship_id pydantic-field

flagship_id

verdict pydantic-field

verdict

agents pydantic-field

agents

worlds pydantic-field

worlds

model_plane pydantic-field

model_plane

fixture_version pydantic-field

fixture_version

stages pydantic-field

stages

gates pydantic-field

gates

parity pydantic-field

parity

reasons pydantic-field

reasons

inputs pydantic-field

inputs

schema_version pydantic-field

schema_version = FLAGSHIP_SCHEMA_VERSION

provenance pydantic-field

provenance

GateDecisionRef pydantic-model

Bases: CalibratedAgentEvaluationModel

The three-gate judge-trust outcome for one judge.

A judge is admitted to score only when all three gates pass -- certify then enforce_bias_gate then enforce_score_gate. A refused ref carries the refusal_reason and means no Score was written from that judge.

Config:

  • frozen: True
  • extra: forbid

Fields:

judge_id pydantic-field

judge_id

certified pydantic-field

certified

bias_verdict pydantic-field

bias_verdict

score_gate_admitted pydantic-field

score_gate_admitted

refusal_reason pydantic-field

refusal_reason = None

admitted property

admitted

True iff every gate passed -- the only state in which a Score may be written.

OracleCheck pydantic-model

Bases: CalibratedAgentEvaluationModel

One stage's output compared against its committed corpus oracle.

oracle_path is None only for an oracle-less stage proven structurally (documented in the note); every stage that claims a committed oracle names it. matched=False carries the divergence -- what differed -- so a failed proof is diagnosable.

pinned=False means the stage named an oracle that does not exist and its plan opted out of oracle pinning: nothing was compared, so matched is False with no divergence. The row stays visible as unpinned rather than failing the run or reading as a match.

Config:

  • frozen: True
  • extra: forbid

Fields:

stage pydantic-field

stage

oracle_path pydantic-field

oracle_path

matched pydantic-field

matched

divergence pydantic-field

divergence = None

pinned pydantic-field

pinned = True

StageOutcome pydantic-model

Bases: CalibratedAgentEvaluationModel

One (agent, world, stage) result.

ok is True iff the stage ran and its oracle matched. headline is the calibration Interval where the stage emits a headline rate (judge/run/improve), None otherwise -- never a bare float. error_type names the typed error class when a stage's shipped feature raised, so the failure is diagnosable from the record alone.

Config:

  • frozen: True
  • extra: forbid

Fields:

agent pydantic-field

agent

world pydantic-field

world

stage pydantic-field

stage

ok pydantic-field

ok

oracle pydantic-field

oracle

headline pydantic-field

headline = None

artifact_ids pydantic-field

artifact_ids = ()

error_type pydantic-field

error_type = None

note pydantic-field

note = ''

WorldParity pydantic-model

Bases: CalibratedAgentEvaluationModel

Cross-world equality for one agent.

equal=False names the divergent field/artifact -- a divergence is a blocker that flips the proof to failed, not a footnote.

Config:

  • frozen: True
  • extra: forbid

Fields:

agent pydantic-field

agent

compared pydantic-field

compared

equal pydantic-field

equal

divergence pydantic-field

divergence = None

compare_oracle

compare_oracle(stage, produced, oracle_path)

Compare a stage's produced fields against the committed oracle.

Only the fields the oracle pins are compared (semantic comparison, not byte-exact prose). A missing/corrupt oracle raises FlagshipError. Returns an OracleCheck whose matched is True iff every pinned field is reproduced, else matched=False naming the first divergence. An oracle that pins no field raises FlagshipError: it would match any output, so it proves nothing.

Source code in src/calibrated_agent_evaluation/flagship/oracle.py
def compare_oracle(
    stage: StageName, produced: dict[str, Any], oracle_path: str | Path
) -> OracleCheck:
    """Compare a stage's ``produced`` fields against the committed oracle.

    Only the fields the oracle pins are compared (semantic comparison, not byte-exact prose). A
    missing/corrupt oracle raises :class:`FlagshipError`. Returns an :class:`OracleCheck` whose
    ``matched`` is True iff every pinned field is reproduced, else ``matched=False`` naming the
    first divergence. An oracle that pins no field raises :class:`FlagshipError`: it would match
    any output, so it proves nothing.
    """
    oracle = load_oracle(oracle_path)
    if not isinstance(oracle, dict):
        raise FlagshipError(
            f"flagship oracle at {oracle_path} must be a JSON object of pinned fields, "
            f"not {type(oracle).__name__}"
        )
    # An oracle either wraps its pinned fields under "expected", or pins them at the top level
    # alongside a human-readable "note" (the shared oracle shape). Drop "note" so a prose
    # annotation is never compared as a pinned field.
    expected = oracle.get("expected", {k: v for k, v in oracle.items() if k != "note"})
    if not isinstance(expected, dict) or not expected:
        raise FlagshipError(
            f"flagship oracle at {oracle_path} pins no fields, so any output would match it; "
            "pin the fields the stage must reproduce (at the top level or under 'expected')"
        )
    divergence = _diverging_key(expected, produced)
    return OracleCheck(
        stage=stage,
        oracle_path=str(oracle_path),
        matched=divergence is None,
        divergence=divergence,
    )

load_oracle

load_oracle(oracle_path)

Load a committed oracle document. Raises FlagshipError if absent/corrupt.

A missing oracle for a stage that claims one is a proof-integrity failure -- the proof cannot be trusted against an oracle that is not there -- so this refuses loudly rather than returning an empty dict that would let a stage "match" nothing.

Source code in src/calibrated_agent_evaluation/flagship/oracle.py
def load_oracle(oracle_path: str | Path) -> dict[str, Any]:
    """Load a committed oracle document. Raises :class:`FlagshipError` if absent/corrupt.

    A missing oracle for a stage that claims one is a proof-integrity failure -- the proof cannot be
    trusted against an oracle that is not there -- so this refuses loudly rather than returning an
    empty dict that would let a stage "match" nothing.
    """
    path = Path(oracle_path)
    if not path.exists():
        raise FlagshipError(
            f"flagship oracle missing at {path}: a stage claiming an oracle cannot be proven "
            "against a file that is not there"
        )
    try:
        return json.loads(path.read_text(encoding="utf-8"))
    except (OSError, json.JSONDecodeError) as exc:
        raise FlagshipError(f"flagship oracle at {path} is unreadable/corrupt: {exc}") from exc

assert_worlds_equal

assert_worlds_equal(agent, comparable_by_world)

Assert the normalized results are field-for-field equal across worlds.

Compares the world-invariant artifacts (normalized trace/label/score dicts) each world produced; equal=False names the first divergent field -- a divergence is a blocker that flips the proof to failed. With fewer than two worlds there is nothing to compare, so parity holds vacuously (a single-world run is not a both-worlds claim).

Source code in src/calibrated_agent_evaluation/flagship/orchestrator.py
def assert_worlds_equal(
    agent: Agent, comparable_by_world: Mapping[World, dict[str, Any]]
) -> WorldParity:
    """Assert the normalized results are field-for-field equal across worlds.

    Compares the world-invariant artifacts (normalized trace/label/score dicts) each world produced;
    ``equal=False`` names the first divergent field -- a divergence is a blocker that flips
    the proof to ``failed``. With fewer than two worlds there is nothing to compare, so
    parity holds vacuously (a single-world run is not a both-worlds claim).
    """
    worlds = tuple(comparable_by_world.keys())
    if len(worlds) < 2:
        only: World = worlds[0] if worlds else "local"
        return WorldParity(agent=agent, compared=(only, only), equal=True, divergence=None)
    a, b = worlds[0], worlds[1]
    doc_a, doc_b = comparable_by_world[a], comparable_by_world[b]
    # Canonicalize to JSON so ordering/tuple-vs-list differences do not read as divergences.
    if json.dumps(doc_a, sort_keys=True) == json.dumps(doc_b, sort_keys=True):
        return WorldParity(agent=agent, compared=(a, b), equal=True, divergence=None)
    divergence = _first_divergent_key(doc_a, doc_b)
    return WorldParity(agent=agent, compared=(a, b), equal=False, divergence=divergence)

prove

prove(
    config,
    plans,
    *,
    backend_factory,
    gates=(),
    flagship_id="flagship",
)

Run the whole flagship proof and return the reconstructable FlagshipReport.

plans supplies the per-agent corpus binding (one AgentPlan per requested agent); backend_factory returns the backend for a world (local or Langfuse) so the orchestrator never handles a vendor object. gates carries the judge-trust decisions (admit_judge outputs) so the report records whether every scoring judge was fully admitted.

The overall verdict is proven iff every StageOutcome.ok is True for every configured (agent, world) AND every WorldParity.equal is True AND every GateDecisionRef is admitted; otherwise failed with reasons naming the offending stage/parity/gate. With no failures but an unpinned stage, the verdict is unpinned. Every requested agent needs exactly one registered plan; an unregistered or doubly-registered id is refused.

Source code in src/calibrated_agent_evaluation/flagship/orchestrator.py
def prove(
    config: FlagshipConfig,
    plans: Sequence[AgentPlan],
    *,
    backend_factory: Callable[[World], TraceStoreBackend],
    gates: Sequence[GateDecisionRef] = (),
    flagship_id: str = "flagship",
) -> FlagshipReport:
    """Run the whole flagship proof and return the reconstructable :class:`FlagshipReport`.

    ``plans`` supplies the per-agent corpus binding (one :class:`AgentPlan` per requested agent);
    ``backend_factory`` returns the backend for a world (local or Langfuse) so the
    orchestrator never handles a vendor object. ``gates`` carries the judge-trust decisions
    (:func:`calibrated_agent_evaluation.enforce.admission.admit_judge` outputs) so the report
    records whether every scoring judge was fully admitted.

    The overall verdict is ``proven`` iff every ``StageOutcome.ok`` is True for every configured
    ``(agent, world)`` AND every ``WorldParity.equal`` is True AND every ``GateDecisionRef`` is
    ``admitted``; otherwise ``failed`` with ``reasons`` naming the offending stage/parity/gate.
    With no failures but an unpinned stage, the verdict is ``unpinned``. Every requested agent
    needs exactly one registered plan; an unregistered or doubly-registered id is refused.
    """
    fixtures_root = Path(config.fixtures_root)
    fixture_version = _assert_fixture_version(fixtures_root)  # before any stage runs

    plan_by_agent: dict[Agent, AgentPlan] = {}
    for p in plans:
        if p.agent in plan_by_agent:
            raise FlagshipError(
                f"more than one AgentPlan registered for agent {p.agent!r}; one would silently "
                "shadow the other"
            )
        plan_by_agent[p.agent] = p
    missing = [a for a in config.agents if a not in plan_by_agent]
    if missing:
        raise FlagshipError(
            f"no AgentPlan supplied for requested agent(s) {missing}; a proof cannot run an agent "
            "it was given no plan for"
        )

    all_stages: list[StageOutcome] = []
    all_parity: list[WorldParity] = []
    reasons: list[str] = []

    for agent in config.agents:
        plan = plan_by_agent[agent]
        comparable_by_world: dict[World, dict[str, Any]] = {}
        for world in config.worlds:
            backend = backend_factory(world)
            ctx = StageContext(
                agent=agent,
                world=world,
                model_plane=config.model_plane,
                backend=backend,
                fixtures_root=fixtures_root,
            )
            outcomes = run_agent_pipeline(plan, ctx)
            all_stages.extend(outcomes)
            for oc in outcomes:
                if not oc.ok:
                    reasons.append(
                        f"{agent}/{world}/{oc.stage} failed: "
                        f"{oc.oracle.divergence or oc.error_type or 'oracle mismatch'}"
                    )
            comparable_by_world[world] = plan.comparable(ctx)

        parity = assert_worlds_equal(agent, comparable_by_world)
        all_parity.append(parity)
        if not parity.equal:
            reasons.append(
                f"{agent} both-worlds divergence at {parity.divergence} "
                f"({parity.compared[0]} != {parity.compared[1]}); the two worlds must agree"
            )

    for g in gates:
        if not g.admitted:
            reasons.append(f"judge {g.judge_id} not admitted: {g.refusal_reason}")

    if reasons:
        verdict = "failed"
    elif any(not oc.oracle.pinned for oc in all_stages):
        verdict = "unpinned"
    else:
        verdict = "proven"
    return FlagshipReport(
        flagship_id=flagship_id,
        verdict=verdict,
        agents=tuple(config.agents),
        worlds=tuple(config.worlds),
        model_plane=config.model_plane,
        fixture_version=fixture_version,
        stages=tuple(all_stages),
        gates=tuple(gates),
        parity=tuple(all_parity),
        reasons=tuple(reasons),
        inputs={
            "agents": ",".join(config.agents),
            "worlds": ",".join(config.worlds),
            "model_plane": config.model_plane,
            "fixtures_root": str(config.fixtures_root),
        },
        provenance=capture_provenance(),
    )

run_agent_pipeline

run_agent_pipeline(plan, ctx)

Run the eight canonical stages for one (agent, world) cell.

Each stage callable composes a shipped surface (never re-implemented) and returns a StageResult; this compares the produced fields against the committed oracle and records a StageOutcome. A stage whose shipped feature raises a typed domain error is not swallowed -- the harness records the stage + the error type on the outcome and marks it failed, never a catch-and-continue partial pass.

Source code in src/calibrated_agent_evaluation/flagship/orchestrator.py
def run_agent_pipeline(plan: AgentPlan, ctx: StageContext) -> tuple[StageOutcome, ...]:
    """Run the eight canonical stages for one ``(agent, world)`` cell.

    Each stage callable composes a shipped surface (never re-implemented) and returns a
    :class:`StageResult`; this compares the produced fields against the committed oracle
    and records a :class:`StageOutcome`. A stage whose shipped feature raises a typed domain error
    is not swallowed -- the harness records the stage + the error type on the outcome and marks it
    failed, never a catch-and-continue partial pass.
    """
    _assert_plan_complete(plan)
    outcomes: list[StageOutcome] = []
    for sp in plan.stages:
        try:
            result = sp.run(ctx)
        except FlagshipError:
            raise  # a proof-integrity failure (missing oracle, unreachable --live) propagates
        except Exception as exc:  # noqa: BLE001 -- a stage's typed domain error fails the proof
            outcomes.append(
                StageOutcome(
                    agent=ctx.agent,
                    world=ctx.world,
                    stage=sp.stage,
                    ok=False,
                    oracle=OracleCheck(
                        stage=sp.stage,
                        oracle_path=None,
                        matched=False,
                        divergence=f"{type(exc).__name__}: {exc}",
                    ),
                    error_type=type(exc).__name__,
                    note=f"stage raised {type(exc).__name__} -- proof failed",
                )
            )
            continue

        if (
            result.oracle_path is not None
            and not plan.oracle_required
            and not Path(result.oracle_path).exists()
        ):
            # Nothing to compare against: record the row as unpinned, never as a match.
            oracle = OracleCheck(
                stage=sp.stage, oracle_path=result.oracle_path, matched=False, pinned=False
            )
        elif result.oracle_path is not None:
            oracle = compare_oracle(sp.stage, result.produced, result.oracle_path)
        else:
            # Oracle-less stage proven structurally; the callable's note records the check made.
            oracle = OracleCheck(stage=sp.stage, oracle_path=None, matched=True, divergence=None)

        gate_ok = result.gate is None or result.gate.admitted
        outcomes.append(
            StageOutcome(
                agent=ctx.agent,
                world=ctx.world,
                stage=sp.stage,
                ok=(oracle.matched or not oracle.pinned) and gate_ok,
                oracle=oracle,
                headline=result.headline,
                artifact_ids=result.artifact_ids,
                note=result.note,
            )
        )
    return tuple(outcomes)

format_report

format_report(report)

The per-agent, per-world, per-stage table with parity, gates and the verdict.

Source code in src/calibrated_agent_evaluation/flagship/runner.py
def format_report(report: FlagshipReport) -> str:
    """The per-agent, per-world, per-stage table with parity, gates and the verdict."""
    lines = [
        f"Calibrated Agent Evaluation flagship proof: agents={list(report.agents)} "
        f"worlds={list(report.worlds)} "
        f"model_plane={report.model_plane} fixture_v={report.fixture_version}",
        "",
    ]
    for outcome in report.stages:
        mark = ("✓" if outcome.oracle.pinned else "?") if outcome.ok else "✗"
        cell = f"{outcome.agent}/{outcome.world}/{outcome.stage}"
        lines.append(f"  {mark}  {cell:34} {outcome.note}")
    lines.append("")
    for parity in report.parity:
        mark = "✓" if parity.equal else "✗"
        detail = "field-for-field equal" if parity.equal else f"DIVERGED at {parity.divergence}"
        lines.append(
            f"  {mark}  parity[{parity.agent}] "
            f"{parity.compared[0]} vs {parity.compared[1]}: {detail}"
        )
    for gate in report.gates:
        mark = "✓" if gate.admitted else "✗"
        detail = (
            "admitted (certify+bias+score)" if gate.admitted else f"REFUSED: {gate.refusal_reason}"
        )
        lines.append(f"  {mark}  judge[{gate.judge_id}] {detail}")
    lines += ["", f"  VERDICT: {report.verdict.upper()}"]
    if report.reasons:
        lines.append("  reasons:")
        lines += [f"    - {reason}" for reason in report.reasons]
    return "\n".join(lines)

run_reference_proof

run_reference_proof(
    corpus,
    *,
    agents,
    worlds,
    model_plane,
    backend_factory,
    calibration,
    battery,
    out_dir=None,
    flagship_id="flagship-m1",
)

Prove the reference agents and, when out_dir is given, write the flagship record.

Parameters:

Name Type Description Default
corpus ReferenceCorpus

The reference corpus to prove against.

required
agents Sequence[Agent]

Which of REFERENCE_AGENTS to prove.

required
worlds Sequence[World]

The worlds to run each agent in.

required
model_plane ModelPlane

"cassette" for the hermetic replay, "live" for a live model plane.

required
backend_factory Callable[[World], TraceStoreBackend]

The trace-store backend for each world.

required
calibration CalibrationRecord

The record the OrderBot judge is admitted on.

required
battery BiasBatteryRecord

The bias battery the OrderBot judge is admitted on.

required
out_dir str | Path | None

Where to write flagship_report.{csv,json}; None writes nothing.

None
flagship_id str

The id stamped on the report.

'flagship-m1'

Returns:

Type Description
FlagshipReport

The report, whatever its verdict.

Raises:

Type Description
FlagshipError

An unknown agent, or a proof-integrity failure prove refuses.

FlagshipStoreError

The record could not be written.

Source code in src/calibrated_agent_evaluation/flagship/runner.py
def run_reference_proof(
    corpus: ReferenceCorpus,
    *,
    agents: Sequence[Agent],
    worlds: Sequence[World],
    model_plane: ModelPlane,
    backend_factory: Callable[[World], TraceStoreBackend],
    calibration: CalibrationRecord,
    battery: BiasBatteryRecord,
    out_dir: str | Path | None = None,
    flagship_id: str = "flagship-m1",
) -> FlagshipReport:
    """Prove the reference agents and, when ``out_dir`` is given, write the flagship record.

    Args:
        corpus: The reference corpus to prove against.
        agents: Which of :data:`REFERENCE_AGENTS` to prove.
        worlds: The worlds to run each agent in.
        model_plane: ``"cassette"`` for the hermetic replay, ``"live"`` for a live model plane.
        backend_factory: The trace-store backend for each world.
        calibration: The record the OrderBot judge is admitted on.
        battery: The bias battery the OrderBot judge is admitted on.
        out_dir: Where to write ``flagship_report.{csv,json}``; ``None`` writes nothing.
        flagship_id: The id stamped on the report.

    Returns:
        The report, whatever its verdict.

    Raises:
        FlagshipError: An unknown agent, or a proof-integrity failure ``prove`` refuses.
        FlagshipStoreError: The record could not be written.
    """
    unknown = [a for a in agents if a not in REFERENCE_AGENTS]
    if unknown:
        raise FlagshipError(
            f"no reference plan for agent(s) {unknown}; the corpus carries {list(REFERENCE_AGENTS)}"
        )
    plans: list[AgentPlan] = []
    for agent in agents:
        if agent == "orderbot":
            plans.append(corpus.orderbot_plan(calibration=calibration, battery=battery))
        else:
            plans.append(corpus.claimscrew_plan())
    config = FlagshipConfig(
        agents=tuple(agents),
        worlds=tuple(worlds),
        model_plane=model_plane,
        fixtures_root=str(corpus.root),
    )
    report = prove(config, plans, backend_factory=backend_factory, flagship_id=flagship_id)
    if out_dir is not None:
        dump_flagship_reports([report], out_dir, fmt="json")
    return report

attribution_stage

attribution_stage(crew, walkbacks, *, oracle_path=None)

Crew attribution: walk-back (culprit != manifestation) over each crew FAIL.

Produces a dict keyed by trace_id -- the culprit/manifestation/status the fixtures/expected/attribution/* oracle pins for each crew trace. Composes validate_walkback verbatim; each walkbacks entry is (trace, culprit_spans, manifestation, handoff).

Source code in src/calibrated_agent_evaluation/flagship/stages.py
def attribution_stage(
    crew: CrewDefinition,
    walkbacks: Sequence[WalkBackInput],
    *,
    oracle_path: str | None = None,
) -> StageCallable:
    """Crew attribution: walk-back (culprit != manifestation) over each crew FAIL.

    Produces a dict keyed by ``trace_id`` -- the culprit/manifestation/status the
    ``fixtures/expected/attribution/*`` oracle pins for each crew trace. Composes
    ``validate_walkback`` verbatim; each ``walkbacks`` entry is
    ``(trace, culprit_spans, manifestation, handoff)``.
    """

    def _run(_ctx: StageContext) -> StageResult:
        produced: dict[str, object] = {}
        for trace, culprit_spans, manifestation_span, handoff in walkbacks:
            att: Attribution = validate_walkback(
                trace,
                crew,
                culprit_spans=culprit_spans,
                manifestation_span=manifestation_span,
                handoff_implicated=handoff,
            )
            produced[trace.trace_id] = {
                "status": att.status,
                "culprit_spans": list(att.culprit_spans),
                "manifestation_span": att.manifestation_span,
                "handoff_implicated": att.handoff_implicated,
            }
        return StageResult(
            produced=produced,
            oracle_path=oracle_path,
            note=f"walk-back over {len(walkbacks)} crew FAIL(s): culprit != manifestation",
        )

    return _run

cluster_stage

cluster_stage(
    fail_modes,
    critiques,
    embed,
    *,
    threshold,
    oracle_path=None,
)

Recover the failure-mode taxonomy + check saturation; oracle = ratified modes.

The ground truth is the human-ratified failure-mode taxonomy carried on the FAIL labels (fail_modes -- the distinct failure_mode names and their counts), which the fixtures/expected/cluster/* oracle pins (the ratified codebook is the taxonomy, not an arbitrary embedding partition). The builder also runs the shipped hierarchical_cluster + saturation_check over the critiques' embeddings so the clusterer surface is genuinely exercised, but the oracle-compared value is the ratified taxonomy the clusterer must recover.

Source code in src/calibrated_agent_evaluation/flagship/stages.py
def cluster_stage(
    fail_modes: Sequence[str],
    critiques: Sequence[str],
    embed: Callable[[list[str]], list[list[float]]],
    *,
    threshold: float,
    oracle_path: str | None = None,
) -> StageCallable:
    """Recover the failure-mode taxonomy + check saturation; oracle = ratified modes.

    The ground truth is the **human-ratified** failure-mode taxonomy carried on the FAIL labels
    (``fail_modes`` -- the distinct ``failure_mode`` names and their counts), which the
    ``fixtures/expected/cluster/*`` oracle pins (the ratified codebook is the taxonomy, not
    an arbitrary embedding partition). The builder also runs the shipped ``hierarchical_cluster`` +
    ``saturation_check`` over the critiques' embeddings so the clusterer surface is genuinely
    exercised, but the oracle-compared value is the ratified taxonomy the clusterer must recover.
    """

    def _run(_ctx: StageContext) -> StageResult:
        from collections import Counter

        # Exercise the shipped clusterer + saturation estimator over the real critiques. The
        # agglomerative clusterer needs >=2 critiques; a single-agent corpus with one failure mode
        # (the OrderBot case) has one, so the embedding-cluster call is skipped there -- the
        # oracle-compared value is the ratified taxonomy, which is well-defined at any count.
        crit = list(critiques)
        saturation_note = "saturation not checked (fewer than 2 critiques to cluster)"
        if len(crit) >= 2:
            assignments = hierarchical_cluster(embed(crit), threshold=threshold)
            # The saturation defaults (30-failure floor, coverage lower bound >= 0.95, >= 2 modes):
            # a small corpus is reported unsaturated with its reason, not passed on a lowered floor.
            sat = saturation_check(list(Counter(assignments).values()))
            saturation_note = (
                f"saturated={sat.saturated} (coverage {sat.coverage.point:.3f}, lb95 "
                f"{sat.coverage.lb95:.3f}, {sat.total_failures} failures in {sat.total_modes} "
                f"cluster(s){'' if sat.reason is None else '; ' + sat.reason})"
            )
        # The oracle-compared taxonomy is the human-ratified failure-mode set (ratified codebook).
        modes = dict(sorted(Counter(fail_modes).items()))
        n = len(fail_modes)
        produced = {"n_modes": len(modes), "modes": modes, "n_fail_labels": n}
        return StageResult(
            produced=produced,
            oracle_path=oracle_path,
            note=(
                f"{produced['n_modes']} ratified failure mode(s) recovered from {n} FAIL "
                f"label(s); {saturation_note}"
            ),
        )

    return _run

improve_stage

improve_stage(request, *, oracle_path=None)

The improve guard: pre-reg + held-out paired McNemar + Holm guard panel.

Produces the verdict + the paired McNemar method/p-value the fixtures/expected/improve/* oracle pins. Composes evaluate_improvement verbatim.

Source code in src/calibrated_agent_evaluation/flagship/stages.py
def improve_stage(request: ImproveRequest, *, oracle_path: str | None = None) -> StageCallable:
    """The improve guard: pre-reg + held-out paired McNemar + Holm guard panel.

    Produces the verdict + the paired McNemar method/p-value the ``fixtures/expected/improve/*``
    oracle pins. Composes ``evaluate_improvement`` verbatim.
    """

    def _run(_ctx: StageContext) -> StageResult:
        verdict = evaluate_improvement(request)
        produced = {
            "verdict": verdict.verdict,
            "paired": {
                "method": verdict.paired.method,
                "n_discordant": verdict.paired.n_discordant,
            },
        }
        relocation = verdict.panel.relocation
        if relocation is not None:
            produced["relocation"] = relocation.verdict
        return StageResult(
            produced=produced,
            oracle_path=oracle_path,
            headline=verdict.paired.effect,
            note=f"improve verdict={verdict.verdict} ({verdict.paired.method})",
        )

    return _run

judge_stage

judge_stage(
    judge,
    calib,
    battery,
    targets,
    model_backend,
    *,
    write_back=True,
)

Run a judge behind the three-gate trust chain; write Scores back if admitted.

Runs admit_judge first (certify, bias, score); only an admitted judge reaches run_judge (which re-checks the score gate at write-back). A refused judge produces a GateDecisionRef carrying the refusal and writes no Score, so the stage's ok is driven off gate.admitted.

Source code in src/calibrated_agent_evaluation/flagship/stages.py
def judge_stage(
    judge: ExecutableJudge,
    calib: CalibrationRecord,
    battery: BiasBatteryRecord,
    targets: Sequence[JudgeTarget],
    model_backend: JudgeModelBackend,
    *,
    write_back: bool = True,
) -> StageCallable:
    """Run a judge behind the three-gate trust chain; write Scores back if admitted.

    Runs :func:`~calibrated_agent_evaluation.enforce.admission.admit_judge` first (certify, bias,
    score); only an admitted judge reaches ``run_judge`` (which re-checks the score gate at
    write-back). A refused judge produces a :class:`GateDecisionRef` carrying the refusal and writes
    no Score, so the stage's ``ok`` is driven off ``gate.admitted``.
    """

    def _run(ctx: StageContext) -> StageResult:
        gate = admit_judge(judge, calib, battery)
        if not gate.admitted:
            return StageResult(
                produced={"admitted": False, "judge_id": gate.judge_id},
                gate=gate,
                note=f"judge refused: {gate.refusal_reason}",
            )
        store = ctx.backend if write_back else None
        result = run_judge(
            judge, list(targets), calib_record=calib, model_backend=model_backend, store=store
        )
        return StageResult(
            produced={
                "admitted": True,
                "judge_id": gate.judge_id,
                "n_labels": len(result.labels),
            },
            gate=gate,
            artifact_ids=tuple(lb.trace_id for lb in result.labels),
            note=f"admitted judge wrote {len(result.labels)} verdict(s) back as Scores",
        )

    return _run

probe_stage

probe_stage(config, transport, oracle_path)

Stage 0 -- run probe.probe over an (offline) transport; oracle = the capability map.

Produces the oracle-comparable subset: the observed tools (a lower bound), the lower-bound flag, the probed-trace shape, and the minimum failure-hypothesis count -- matching fixtures/expected/probe/orderbot_capability.json.

Source code in src/calibrated_agent_evaluation/flagship/stages.py
def probe_stage(config: ConnectionConfig, transport: Transport, oracle_path: str) -> StageCallable:
    """Stage 0 -- run ``probe.probe`` over an (offline) transport; oracle = the capability map.

    Produces the oracle-comparable subset: the observed tools (a lower bound), the lower-bound flag,
    the probed-trace shape, and the minimum failure-hypothesis count -- matching
    ``fixtures/expected/probe/orderbot_capability.json``.
    """

    def _run(_ctx: StageContext) -> StageResult:
        report: ProbeReport = probe(config, authorized=True, transport=transport)
        trace = report.traces[0]
        framing = report.capability_map.framing
        produced = {
            "observed_tools": list(report.capability_map.observed_tools),
            "observed_tools_are_lower_bound": bool(framing.tool_inventory_is_lower_bound),
            "probed_trace": {
                "ownership": trace.ownership,
                "spans": list(trace.spans),
                "endpoint_fingerprint_present": trace.endpoint_fingerprint is not None,
            },
            "min_failure_hypotheses": len(report.hypotheses),
        }
        return StageResult(
            produced=produced,
            oracle_path=oracle_path,
            artifact_ids=tuple(t.trace_id for t in report.traces),
            note=f"probed {len(report.traces)} trace(s); tools={produced['observed_tools']}",
        )

    return _run

run_gate_stage

run_gate_stage(labels, gold, *, threshold, oracle_path)

The run/CI gate: aggregate verdicts -> pass/fail on the lower bound.

Produces the gate-decision subset the fixtures/expected/run/gate_decision.json oracle pins: passed, the failing modes (failure_type + agent), and n_modes, the modes gated. Composes run_ci_gate verbatim.

Source code in src/calibrated_agent_evaluation/flagship/stages.py
def run_gate_stage(
    labels: Sequence[Label],
    gold: dict[tuple[str, str | None], object],
    *,
    threshold: float,
    oracle_path: str,
) -> StageCallable:
    """The run/CI gate: aggregate verdicts -> pass/fail on the lower bound.

    Produces the gate-decision subset the ``fixtures/expected/run/gate_decision.json`` oracle pins:
    ``passed``, the failing modes (``failure_type`` + ``agent``), and ``n_modes``, the modes gated.
    Composes ``run_ci_gate`` verbatim.
    """

    def _run(_ctx: StageContext) -> StageResult:
        decision = run_ci_gate(
            CiGateRequest(labels=list(labels), gold=gold, threshold=threshold)  # type: ignore[arg-type]
        )
        produced = {
            "passed": decision.passed,
            "failing_modes": [
                {"failure_type": m.failure_type, "agent": m.agent} for m in decision.failing_modes
            ],
            "n_modes": decision.n_modes,
        }
        return StageResult(
            produced=produced,
            oracle_path=oracle_path,
            note=f"gate passed={decision.passed}, failing={len(decision.failing_modes)}",
        )

    return _run

structural_stage

structural_stage(note, produce)

A stage with no committed oracle, proven structurally (define/generate/label).

produce runs the shipped surface and returns a summary dict; the note records the structural check made (e.g. "one input per frame row, every input frame-tagged"). No oracle_path -> the orchestrator marks it matched and relies on the structural check.

Source code in src/calibrated_agent_evaluation/flagship/stages.py
def structural_stage(
    note: str, produce: Callable[[StageContext], dict[str, object]]
) -> StageCallable:
    """A stage with no committed oracle, proven structurally (define/generate/label).

    ``produce`` runs the shipped surface and returns a summary dict; the ``note`` records the
    structural check made (e.g. "one input per frame row, every input frame-tagged"). No
    ``oracle_path`` -> the orchestrator marks it matched and relies on the structural check.
    """

    def _run(ctx: StageContext) -> StageResult:
        fields = produce(ctx)
        return StageResult(produced=fields, oracle_path=None, note=note)

    return _run

dump_flagship_reports

dump_flagship_reports(reports, path, fmt='csv')

Write reports to flagship_report.{csv,json} under path.

Parameters:

Name Type Description Default
reports Sequence[FlagshipReport]

The flagship reports to persist.

required
path str | Path

The directory to write into.

required
fmt str

Which file a later load_flagship_reports reads back (both are always written).

'csv'

Raises:

Type Description
FlagshipStoreError

A bad fmt or any filesystem failure.

Source code in src/calibrated_agent_evaluation/flagship/store.py
def dump_flagship_reports(
    reports: Sequence[FlagshipReport], path: str | Path, fmt: str = "csv"
) -> None:
    """Write ``reports`` to ``flagship_report.{csv,json}`` under ``path``.

    Args:
        reports: The flagship reports to persist.
        path: The directory to write into.
        fmt: Which file a later :func:`load_flagship_reports` reads back (both are always written).

    Raises:
        FlagshipStoreError: A bad ``fmt`` or any filesystem failure.
    """
    _STORE.dump(reports, path, fmt)

load_flagship_reports

load_flagship_reports(path, fmt='csv')

Read flagship reports back from path; a missing file is an empty store.

Parameters:

Name Type Description Default
path str | Path

The directory dump_flagship_reports wrote.

required
fmt str

"csv" or "json".

'csv'

Returns:

Type Description
list[FlagshipReport]

The stored reports.

Raises:

Type Description
FlagshipStoreError

A schema-version mismatch, or any read, parse or validation failure.

Source code in src/calibrated_agent_evaluation/flagship/store.py
def load_flagship_reports(path: str | Path, fmt: str = "csv") -> list[FlagshipReport]:
    """Read flagship reports back from ``path``; a missing file is an empty store.

    Args:
        path: The directory :func:`dump_flagship_reports` wrote.
        fmt: ``"csv"`` or ``"json"``.

    Returns:
        The stored reports.

    Raises:
        FlagshipStoreError: A schema-version mismatch, or any read, parse or validation failure.
    """
    return _STORE.load(path, fmt)