calibrated_agent_evaluation.flagship¶
calibrated_agent_evaluation.flagship ¶
The flagship proof harness -- the end-to-end proof of the whole pipeline.
Runs both reference agents (OrderBot single-agent, ClaimsCrew crew) end-to-end through all eight
pipeline stages (probe -> define -> generate -> label -> cluster -> judge -> run -> improve)
with calibrated, bias-gated judges, in both worlds (local + self-hosted Langfuse; cassette-replay +
live Bedrock), and emits one reconstructable FlagshipReport. It is a pure orchestration
feature: it composes the shipped calibrated_agent_evaluation.* surfaces and introduces no
estimator, gate, judge, scorer, or statistic. The corpus-specific binding (which
trace/cassette/judge/crew per agent) is injected by the caller as an AgentPlan, so this
package imports only calibrated_agent_evaluation.* (no scripts/examples) and no vendor
SDK type. ReferenceCorpus builds the two reference plans from a source checkout's
fixtures/ tree, and run_reference_proof runs them and writes the record.
Public surface
prove-- run the whole proof for aFlagshipConfig, return aFlagshipReport.run_agent_pipeline-- run the eight stages for one(agent, world)cell.run_reference_proof-- prove the reference agents and write the flagship record.ReferenceCorpus-- the reference agents' plans over a checkout'sfixtures/.compare_oracle-- compare a stage's output against its committed corpus oracle.AgentPlan-- the injected per-agent corpus binding the orchestrator drives.- the frozen report types + the self-contained CSV+JSON codec.
CANONICAL_STAGES
module-attribute
¶
CANONICAL_STAGES = (
"probe",
"define",
"generate",
"label",
"cluster",
"judge",
"run",
"improve",
)
StageName
module-attribute
¶
StageName = Literal[
"probe",
"define",
"generate",
"label",
"cluster",
"judge",
"run",
"improve",
]
REFERENCE_AGENTS
module-attribute
¶
The reference agents the committed corpus carries plans for.
FlagshipError ¶
Bases: CalibratedAgentEvaluationDataError
Base class for every flagship-harness refusal.
Raised when the proof cannot be trusted: a seeded fixture version that does not match the
running SCHEMA_VERSION, a missing/corrupt oracle for a stage that claims one,
a --live model that is unreachable/unconfigured (re-raised from the
executor's ExecutionError so the harness never silently replays the cassette and reports a
live pass), or a stage's shipped feature raising a typed domain error mid-proof. It is
a refusal to declare the proof complete, not a bug in a composed stage.
FlagshipStoreError ¶
Bases: FlagshipError
A FlagshipReport dump/load file read or write failed.
Wraps the underlying OSError / json.JSONDecodeError / csv failure so a
filesystem/parse error surfaces inside the typed family rather than as a raw stdlib exception
escaping the store's public surface (wrap external errors at the boundary). A stored
schema_version != the running FLAGSHIP_SCHEMA_VERSION also refuses here (no coercion).
AgentPlan
dataclass
¶
The injected per-agent corpus binding the orchestrator drives.
stages is the eight StagePlans in canonical order; the orchestrator refuses a plan that
does not cover all eight canonical stages (the completeness invariant). comparable
reads the world-invariant artifacts (the normalized trace/label/score dicts) the cross-world
parity assertion compares.
oracle_required (default True) keeps the hard refusal: a stage naming an oracle that is
absent is a proof-integrity failure. A plan for an agent with no hand-authored oracle corpus
sets it False, and each absent-oracle stage is recorded as unpinned instead, which makes the
report's verdict unpinned rather than proven. A present but corrupt oracle still raises.
StageContext
dataclass
¶
What a stage callable is handed: the world's backend + run coordinates.
A stage reads/writes through backend (the trace-store protocol -- local or Langfuse, chosen
by the orchestrator so the stage never branches on world), and consults model_plane to pick
cassette-replay vs live for the model-touching stages.
StagePlan
dataclass
¶
StageResult
dataclass
¶
What a stage callable returns -- the produced fields the oracle pins + optional headline.
produced is the subset of fields the committed oracle constrains (compared semantically);
oracle_path is None for an oracle-less stage proven structurally (the note records
the structural check). headline is the calibration Interval where the stage emits a
rate. gate carries the three-gate judge-trust decision for the judge stage.
ReferenceCorpus ¶
The committed reference corpus under one fixtures/ directory, and its two plans.
Bind to a corpus.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
fixtures_root
|
str | Path
|
The checkout's |
required |
Raises:
| Type | Description |
|---|---|
FlagshipError
|
The directory does not hold the corpus (no |
Source code in src/calibrated_agent_evaluation/flagship/reference.py
trace ¶
A normalized corpus trace.
Source code in src/calibrated_agent_evaluation/flagship/reference.py
gold_rows ¶
The committed gold label rows, as stored.
Source code in src/calibrated_agent_evaluation/flagship/reference.py
gold_labels ¶
The gold labels keyed by (trace_id, span_id).
Source code in src/calibrated_agent_evaluation/flagship/reference.py
orderbot_judge ¶
The OrderBot judge, read from the same judge file judge --judge reads.
Source code in src/calibrated_agent_evaluation/flagship/reference.py
orderbot_plan ¶
The eight stages for OrderBot, its judge admitted on calibration and battery.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
calibration
|
CalibrationRecord
|
The calibration record the judge stage admits the judge on. |
required |
battery
|
BiasBatteryRecord
|
The bias battery the judge stage admits the judge on. |
required |
Source code in src/calibrated_agent_evaluation/flagship/reference.py
claimscrew_plan ¶
The eight stages for ClaimsCrew; its judge stage is the attribution walk-back.
Source code in src/calibrated_agent_evaluation/flagship/reference.py
FlagshipConfig
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
The input to prove.
agents selects which agents to prove (each needs a registered AgentPlan), worlds
selects the trace-store axis (local, and langfuse when configured), model_plane selects
the model axis (cassette default/CI vs live sign-off), and fixtures_root is the seeded
corpus root whose VERSION is checked against the running SCHEMA_VERSION.
Config:
frozen:Trueextra:forbid
Fields:
-
agents(tuple[Agent, ...]) -
worlds(tuple[World, ...]) -
model_plane(ModelPlane) -
fixtures_root(str)
FlagshipReport
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
The single reconstructable proof artifact.
verdict == "proven" iff every StageOutcome.ok is True for every configured
(agent, world) AND every WorldParity.equal is True AND every GateDecisionRef used to
score was fully admitted; otherwise failed with reasons naming the offending
stage/parity/gate. Round-trips through the self-contained
store codec; load refuses on a
schema_version mismatch.
Config:
frozen:Trueextra:forbid
Fields:
-
flagship_id(str) -
verdict(Verdict) -
agents(tuple[Agent, ...]) -
worlds(tuple[World, ...]) -
model_plane(ModelPlane) -
fixture_version(str) -
stages(tuple[StageOutcome, ...]) -
gates(tuple[GateDecisionRef, ...]) -
parity(tuple[WorldParity, ...]) -
reasons(tuple[str, ...]) -
inputs(dict[str, str]) -
schema_version(str) -
provenance(ProvenanceStamp)
Validators:
-
_check_invariants
GateDecisionRef
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
The three-gate judge-trust outcome for one judge.
A judge is admitted to score only when all three gates pass -- certify then
enforce_bias_gate then enforce_score_gate. A refused ref carries the
refusal_reason and means no Score was written from that judge.
Config:
frozen:Trueextra:forbid
Fields:
-
judge_id(str) -
certified(bool) -
bias_verdict(BiasVerdict) -
score_gate_admitted(bool) -
refusal_reason(str | None)
admitted
property
¶
True iff every gate passed -- the only state in which a Score may be written.
OracleCheck
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
One stage's output compared against its committed corpus oracle.
oracle_path is None only for an oracle-less stage proven structurally (documented in the
note); every stage that claims a committed oracle names it. matched=False carries the
divergence -- what differed -- so a failed proof is diagnosable.
pinned=False means the stage named an oracle that does not exist and its plan opted out of
oracle pinning: nothing was compared, so matched is False with no divergence. The row
stays visible as unpinned rather than failing the run or reading as a match.
Config:
frozen:Trueextra:forbid
Fields:
-
stage(StageName) -
oracle_path(str | None) -
matched(bool) -
divergence(str | None) -
pinned(bool)
StageOutcome
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
One (agent, world, stage) result.
ok is True iff the stage ran and its oracle matched. headline is the calibration
Interval where the stage emits a headline rate (judge/run/improve), None otherwise --
never a bare float. error_type names the typed error class when a stage's shipped feature
raised, so the failure is diagnosable from the record alone.
Config:
frozen:Trueextra:forbid
Fields:
-
agent(Agent) -
world(World) -
stage(StageName) -
ok(bool) -
oracle(OracleCheck) -
headline(Interval | None) -
artifact_ids(tuple[str, ...]) -
error_type(str | None) -
note(str)
WorldParity
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
Cross-world equality for one agent.
equal=False names the divergent field/artifact -- a divergence is a blocker that
flips the proof to failed, not a footnote.
Config:
frozen:Trueextra:forbid
Fields:
compare_oracle ¶
Compare a stage's produced fields against the committed oracle.
Only the fields the oracle pins are compared (semantic comparison, not byte-exact prose). A
missing/corrupt oracle raises FlagshipError. Returns an OracleCheck whose
matched is True iff every pinned field is reproduced, else matched=False naming the
first divergence. An oracle that pins no field raises FlagshipError: it would match
any output, so it proves nothing.
Source code in src/calibrated_agent_evaluation/flagship/oracle.py
load_oracle ¶
Load a committed oracle document. Raises FlagshipError if absent/corrupt.
A missing oracle for a stage that claims one is a proof-integrity failure -- the proof cannot be trusted against an oracle that is not there -- so this refuses loudly rather than returning an empty dict that would let a stage "match" nothing.
Source code in src/calibrated_agent_evaluation/flagship/oracle.py
assert_worlds_equal ¶
Assert the normalized results are field-for-field equal across worlds.
Compares the world-invariant artifacts (normalized trace/label/score dicts) each world produced;
equal=False names the first divergent field -- a divergence is a blocker that flips
the proof to failed. With fewer than two worlds there is nothing to compare, so
parity holds vacuously (a single-world run is not a both-worlds claim).
Source code in src/calibrated_agent_evaluation/flagship/orchestrator.py
prove ¶
Run the whole flagship proof and return the reconstructable FlagshipReport.
plans supplies the per-agent corpus binding (one AgentPlan per requested agent);
backend_factory returns the backend for a world (local or Langfuse) so the
orchestrator never handles a vendor object. gates carries the judge-trust decisions
(admit_judge outputs) so the report
records whether every scoring judge was fully admitted.
The overall verdict is proven iff every StageOutcome.ok is True for every configured
(agent, world) AND every WorldParity.equal is True AND every GateDecisionRef is
admitted; otherwise failed with reasons naming the offending stage/parity/gate.
With no failures but an unpinned stage, the verdict is unpinned. Every requested agent
needs exactly one registered plan; an unregistered or doubly-registered id is refused.
Source code in src/calibrated_agent_evaluation/flagship/orchestrator.py
260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 | |
run_agent_pipeline ¶
Run the eight canonical stages for one (agent, world) cell.
Each stage callable composes a shipped surface (never re-implemented) and returns a
StageResult; this compares the produced fields against the committed oracle
and records a StageOutcome. A stage whose shipped feature raises a typed domain error
is not swallowed -- the harness records the stage + the error type on the outcome and marks it
failed, never a catch-and-continue partial pass.
Source code in src/calibrated_agent_evaluation/flagship/orchestrator.py
format_report ¶
The per-agent, per-world, per-stage table with parity, gates and the verdict.
Source code in src/calibrated_agent_evaluation/flagship/runner.py
run_reference_proof ¶
run_reference_proof(
corpus,
*,
agents,
worlds,
model_plane,
backend_factory,
calibration,
battery,
out_dir=None,
flagship_id="flagship-m1",
)
Prove the reference agents and, when out_dir is given, write the flagship record.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
corpus
|
ReferenceCorpus
|
The reference corpus to prove against. |
required |
agents
|
Sequence[Agent]
|
Which of |
required |
worlds
|
Sequence[World]
|
The worlds to run each agent in. |
required |
model_plane
|
ModelPlane
|
|
required |
backend_factory
|
Callable[[World], TraceStoreBackend]
|
The trace-store backend for each world. |
required |
calibration
|
CalibrationRecord
|
The record the OrderBot judge is admitted on. |
required |
battery
|
BiasBatteryRecord
|
The bias battery the OrderBot judge is admitted on. |
required |
out_dir
|
str | Path | None
|
Where to write |
None
|
flagship_id
|
str
|
The id stamped on the report. |
'flagship-m1'
|
Returns:
| Type | Description |
|---|---|
FlagshipReport
|
The report, whatever its verdict. |
Raises:
| Type | Description |
|---|---|
FlagshipError
|
An unknown agent, or a proof-integrity failure |
FlagshipStoreError
|
The record could not be written. |
Source code in src/calibrated_agent_evaluation/flagship/runner.py
attribution_stage ¶
Crew attribution: walk-back (culprit != manifestation) over each crew FAIL.
Produces a dict keyed by trace_id -- the culprit/manifestation/status the
fixtures/expected/attribution/* oracle pins for each crew trace. Composes
validate_walkback verbatim; each walkbacks entry is
(trace, culprit_spans, manifestation, handoff).
Source code in src/calibrated_agent_evaluation/flagship/stages.py
cluster_stage ¶
Recover the failure-mode taxonomy + check saturation; oracle = ratified modes.
The ground truth is the human-ratified failure-mode taxonomy carried on the FAIL labels
(fail_modes -- the distinct failure_mode names and their counts), which the
fixtures/expected/cluster/* oracle pins (the ratified codebook is the taxonomy, not
an arbitrary embedding partition). The builder also runs the shipped hierarchical_cluster +
saturation_check over the critiques' embeddings so the clusterer surface is genuinely
exercised, but the oracle-compared value is the ratified taxonomy the clusterer must recover.
Source code in src/calibrated_agent_evaluation/flagship/stages.py
improve_stage ¶
The improve guard: pre-reg + held-out paired McNemar + Holm guard panel.
Produces the verdict + the paired McNemar method/p-value the fixtures/expected/improve/*
oracle pins. Composes evaluate_improvement verbatim.
Source code in src/calibrated_agent_evaluation/flagship/stages.py
judge_stage ¶
Run a judge behind the three-gate trust chain; write Scores back if admitted.
Runs admit_judge first (certify, bias,
score); only an admitted judge reaches run_judge (which re-checks the score gate at
write-back). A refused judge produces a GateDecisionRef carrying the refusal and writes
no Score, so the stage's ok is driven off gate.admitted.
Source code in src/calibrated_agent_evaluation/flagship/stages.py
probe_stage ¶
Stage 0 -- run probe.probe over an (offline) transport; oracle = the capability map.
Produces the oracle-comparable subset: the observed tools (a lower bound), the lower-bound flag,
the probed-trace shape, and the minimum failure-hypothesis count -- matching
fixtures/expected/probe/orderbot_capability.json.
Source code in src/calibrated_agent_evaluation/flagship/stages.py
run_gate_stage ¶
The run/CI gate: aggregate verdicts -> pass/fail on the lower bound.
Produces the gate-decision subset the fixtures/expected/run/gate_decision.json oracle pins:
passed, the failing modes (failure_type + agent), and n_modes, the modes gated.
Composes run_ci_gate verbatim.
Source code in src/calibrated_agent_evaluation/flagship/stages.py
structural_stage ¶
A stage with no committed oracle, proven structurally (define/generate/label).
produce runs the shipped surface and returns a summary dict; the note records the
structural check made (e.g. "one input per frame row, every input frame-tagged"). No
oracle_path -> the orchestrator marks it matched and relies on the structural check.
Source code in src/calibrated_agent_evaluation/flagship/stages.py
dump_flagship_reports ¶
Write reports to flagship_report.{csv,json} under path.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
reports
|
Sequence[FlagshipReport]
|
The flagship reports to persist. |
required |
path
|
str | Path
|
The directory to write into. |
required |
fmt
|
str
|
Which file a later |
'csv'
|
Raises:
| Type | Description |
|---|---|
FlagshipStoreError
|
A bad |
Source code in src/calibrated_agent_evaluation/flagship/store.py
load_flagship_reports ¶
Read flagship reports back from path; a missing file is an empty store.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str | Path
|
The directory |
required |
fmt
|
str
|
|
'csv'
|
Returns:
| Type | Description |
|---|---|
list[FlagshipReport]
|
The stored reports. |
Raises:
| Type | Description |
|---|---|
FlagshipStoreError
|
A schema-version mismatch, or any read, parse or validation failure. |