Skip to content

calibrated_agent_evaluation.improve

calibrated_agent_evaluation.improve

Check whether a proposed fix really worked.

The last stage of the loop. Four guards combine into one ImproveVerdict:

  • a pre-registration ledger, which freezes the metric and threshold before the fix;
  • a held-out confirmation slice, fresh and paired, that the fix was not tuned on;
  • a paired McNemar test on before and after verdicts (with a mid-p variant, and an explicit undefined result when no verdict changed);
  • a Holm-corrected panel of guard slices that catches a fix that only moves failures elsewhere, which for a crew means gating on the workflow outcome rate.

No trace-store or observability SDK and no backend writes; the only external library is statsmodels, used in mcnemar.

EMPTY_LEDGER_HEAD module-attribute

EMPTY_LEDGER_HEAD = 'empty'

LEDGER_HEAD_FIELD module-attribute

LEDGER_HEAD_FIELD = 'ledger_head'

IMPROVE_SCHEMA_VERSION module-attribute

IMPROVE_SCHEMA_VERSION = '6'

CrewComparison pydantic-model

Bases: CalibratedAgentEvaluationModel

Both sides' crew failures, paired per trace for the guard slices.

agent_pairs and handoff_pairs hold, for each agent and handoff measured on both sides, every paired trace's (failed before, failed after). not_measured_agents and not_measured_handoffs were measured on neither side.

Config:

  • frozen: True
  • extra: forbid

Fields:

before pydantic-field

before

after pydantic-field

after

agent_pairs pydantic-field

agent_pairs

handoff_pairs pydantic-field

handoff_pairs

not_measured_agents pydantic-field

not_measured_agents

not_measured_handoffs pydantic-field

not_measured_handoffs

CrewFile pydantic-model

Bases: _Strict

The crew file calibrated-agent-evaluation improve --crew-file reads.

agents and handoffs drive the guard panel. scenarios and rules are optional, validated by build_crew_definition when present, and not read by the counting.

Fields:

agents pydantic-field

agents

handoffs pydantic-field

handoffs

scenarios pydantic-field

scenarios

rules pydantic-field

rules

CrewSide pydantic-model

Bases: CalibratedAgentEvaluationModel

One side's crew failures over the paired traces.

agent_failed and handoff_failed map every declared agent and handoff to the paired traces that failed on it, each trace once. agents_measured and handoffs_measured are the ones the source spoke to on this side. disputed are the traces with a FAIL whose culprit is disputed.

Config:

  • frozen: True
  • extra: forbid

Fields:

agent_failed pydantic-field

agent_failed

handoff_failed pydantic-field

handoff_failed

agents_measured pydantic-field

agents_measured

handoffs_measured pydantic-field

handoffs_measured

disputed pydantic-field

disputed

AdjudicationRequiredError

Bases: ImproveError

A judge-labeled discordant pair lacks a human-ratified adjudication.

McNemar assumes ground-truth labels; a judge's own run-to-run flip-flops land in exactly the discordant cells it counts. So when an LLM judge produced the before/after labels, the improve guard refuses to run the test until every discordant pair carries a kind="human" adjudicating label -- the human-gold boundary (the "refuse to calibrate from gold" analogue). The improve stage writes no label itself; it requires the human-provenance one the review surface produces.

ConfirmationContaminationError

Bases: ImproveError

The confirmation slice overlaps the development / tuning set.

Measuring a headline before/after on traces the fix was tuned against is regression-to-the-mean contamination -- the dev set scores better by construction. Raised (naming the overlapping trace_id) rather than measuring a contaminated headline.

ImproveError

Bases: CalibratedAgentEvaluationDataError

Base class for every improve-stage refusal.

ImproveStoreError

Bases: ImproveError

An ImproveRecord file read or write failed.

Wraps the underlying OSError / json.JSONDecodeError / csv failure so a filesystem/parse error surfaces inside the typed family rather than as a raw stdlib exception escaping the store's public surface ("wrap external errors at the boundary"). A stored schema_version != the running IMPROVE_SCHEMA_VERSION also refuses here (no coercion).

LedgerIntegrityError

Bases: ImproveError

The pre-registration ledger was edited, replaced or frozen after the run it gates.

Raised when a write would change or drop an entry already on disk, when the ledger no longer holds an entry an earlier evaluation recorded, and when a pre-registration was frozen after the earliest time recorded in the after run. Each is how a threshold chosen after seeing the result would get in, so the gate refuses rather than measure against it.

GuardPanelReport pydantic-model

Bases: CalibratedAgentEvaluationModel

The Holm-corrected guard-slice panel.

method is the literal "holm" only: one surviving slice rejects the fix, so the panel controls the family-wise error rate, which Holm does under any dependence between the overlapping slices (fdr_bh and fdr_by are not accepted). m is the comparison count, stated per run. regressed names the slices whose regression survived the correction. relocation carries the crew verdict for a crew improvement, None for a single-agent one.

Config:

  • frozen: True
  • extra: forbid

Fields:

slices pydantic-field

slices

method pydantic-field

method

m pydantic-field

m

regressed pydantic-field

regressed

relocation pydantic-field

relocation = None

GuardSlice pydantic-model

Bases: CalibratedAgentEvaluationModel

One non-targeted slice watched for regression, with its confidence-treated rates.

regression_p is the p-value that this slice regressed (worsened) between before and after; the panel corrects these across all slices with Holm. rate_before / rate_after are Interval\ s.

Config:

  • frozen: True
  • extra: forbid

Fields:

name pydantic-field

name

kind pydantic-field

kind

regression_p pydantic-field

regression_p

rate_before pydantic-field

rate_before

rate_after pydantic-field

rate_after

ImproveRecord pydantic-model

Bases: CalibratedAgentEvaluationModel

The persisted, version-pinned audit artifact for one improvement evaluation.

Carries the referenced pre-registration, the paired result, the guard panel (with its multiplicity m and any crew relocation), the confirmation/development slice ids it was proven disjoint against, the final verdict, and the inputs that produced it. Round-trips losslessly through store.

Config:

  • frozen: True
  • extra: forbid

Fields:

Validators:

  • _check_invariants

improve_id pydantic-field

improve_id

verdict pydantic-field

verdict

prereg pydantic-field

prereg

paired pydantic-field

paired

panel pydantic-field

panel

confirmation_slice_id pydantic-field

confirmation_slice_id

development_set_id pydantic-field

development_set_id

inputs pydantic-field

inputs

reasons pydantic-field

reasons = ()

schema_version pydantic-field

schema_version = IMPROVE_SCHEMA_VERSION

provenance pydantic-field

provenance

ImproveRequest dataclass

ImproveRequest(
    ledger,
    target_metric,
    pairs,
    confirmation_ids,
    development_ids,
    slices=(),
    adjudications=dict(),
    crew=None,
    crew_before=None,
    crew_after=None,
    touched_agent=None,
    success_threshold=None,
    anchors=(),
    after_run_started_at=None,
)

The full input to evaluate_improvement.

ledger + target_metric select the frozen pre-registration to gate on; pairs are the paired before/after observations on the confirmation slice; confirmation_ids / development_ids prove disjointness; slices are the non-targeted guard panel. adjudications maps a discordant trace_id to the kind="human" Label that adjudicates it -- the KEY presence + human provenance IS the ratification record (there is no separate Adjudication type; the review surface records it via Label.kind="human", not a label field). The crew / crew_before / crew_after / touched_agent fields are populated only for a crew improvement. success_threshold is optional and never used as the threshold: the gate reads the pre-registered one, and a caller who states a threshold that differs from it is refused. anchors are the pre-registration entries earlier evaluations recorded (each ImproveRecord carries its own); the ledger must still hold every one unchanged. after_run_started_at is the earliest time recorded in the after run, when known; the pre-registration must have been frozen before it.

ledger instance-attribute

ledger

target_metric instance-attribute

target_metric

pairs instance-attribute

pairs

confirmation_ids instance-attribute

confirmation_ids

development_ids instance-attribute

development_ids

slices class-attribute instance-attribute

slices = ()

adjudications class-attribute instance-attribute

adjudications = field(default_factory=dict)

crew class-attribute instance-attribute

crew = None

crew_before class-attribute instance-attribute

crew_before = None

crew_after class-attribute instance-attribute

crew_after = None

touched_agent class-attribute instance-attribute

touched_agent = None

success_threshold class-attribute instance-attribute

success_threshold = None

anchors class-attribute instance-attribute

anchors = ()

after_run_started_at class-attribute instance-attribute

after_run_started_at = None

ImproveVerdict pydantic-model

Bases: CalibratedAgentEvaluationModel

The composed improve decision.

verdict is rejected if the paired test did not show an improvement, OR any guard slice regressed, OR a crew improvement relocated its failure; accepted only when all guards pass. reasons records why a rejection happened (named slices / relocation / not-significant).

Config:

  • frozen: True
  • extra: forbid

Fields:

verdict pydantic-field

verdict

prereg pydantic-field

prereg

paired pydantic-field

paired

panel pydantic-field

panel

reasons pydantic-field

reasons = ()

McNemarResult pydantic-model

Bases: CalibratedAgentEvaluationModel

The paired before/after significance.

method records how it was computed -- asymptotic (>= 25 discordant), exact / midp (< 25 discordant), or undefined (zero discordant pairs: the test has no info, so p_value is None and undefined_reason is populated -- never a fabricated p-value). effect is the change in fail-rate (after - before) as an Interval.

Config:

  • frozen: True
  • extra: forbid

Fields:

Validators:

  • _check_invariants

method pydantic-field

method

n_discordant pydantic-field

n_discordant

p_value pydantic-field

p_value

effect pydantic-field

effect

undefined_reason pydantic-field

undefined_reason = None

PairedObs dataclass

PairedObs(
    trace_id,
    before_fail,
    after_fail,
    labels_are_judge_produced=False,
)

One paired before/after observation on a single confirmation-slice input.

before_fail / after_fail are the old/new agent's binary FAIL verdict on the SAME input (fresh-and-paired); together across the slice they populate the 2x2 McNemar table. labels_are_judge_produced drives the adjudication gate.

trace_id instance-attribute

trace_id

before_fail instance-attribute

before_fail

after_fail instance-attribute

after_fail

labels_are_judge_produced class-attribute instance-attribute

labels_are_judge_produced = False

PreRegEntry pydantic-model

Bases: CalibratedAgentEvaluationModel

One frozen pre-registration: a target metric + success threshold committed BEFORE a fix.

freeze (in prereg) stamps content_hash over frozen_fields; require_prereg re-verifies it on read, so a metric or threshold edited after the entry was frozen fails the check and is refused. is_crew marks target_metric as the crew's workflow outcome rate.

success_threshold is the minimum absolute reduction in the target's fail rate the fix must show, in (0, 1): 0.10 means the fail rate must fall by at least ten percentage points. The improve gate accepts only when the upper bound of the paired effect interval (after minus before) is at most -success_threshold, so the whole interval shows that reduction, not just its point. freeze refuses a value outside (0, 1).

The temporal anchor: frozen_at is the UTC time freeze stamped, prev_hash the content_hash of the ledger entry before this one (None for the first), and code_ref the commit of the agent under test the entry was frozen against, supplied by the caller. All three are inside the hash, so an entry edited and re-frozen gets a new frozen_at and breaks the chain to every later entry.

Config:

  • frozen: True
  • extra: forbid

Fields:

prereg_id pydantic-field

prereg_id

target_metric pydantic-field

target_metric

success_threshold pydantic-field

success_threshold

is_crew pydantic-field

is_crew = False

content_hash pydantic-field

content_hash = ''

frozen_at pydantic-field

frozen_at = ''

prev_hash pydantic-field

prev_hash = None

code_ref pydantic-field

code_ref = None

frozen_fields

frozen_fields()

The tuple the content_hash is computed over (everything except the hash itself).

Source code in src/calibrated_agent_evaluation/improve/report.py
def frozen_fields(self) -> tuple[object, ...]:
    """The tuple the ``content_hash`` is computed over (everything except the hash itself)."""
    return (
        self.prereg_id,
        self.target_metric,
        self.success_threshold,
        # nosemgrep: is-function-without-parentheses  -- bool field, not a method
        self.is_crew,
        self.frozen_at,
        self.prev_hash,
        self.code_ref,
    )

compute_hash

compute_hash()

Deterministic content hash over frozen_fields (JSON-canonical, sha256).

Source code in src/calibrated_agent_evaluation/improve/report.py
def compute_hash(self) -> str:
    """Deterministic content hash over :meth:`frozen_fields` (JSON-canonical, sha256)."""
    payload = json.dumps(self.frozen_fields(), sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(payload.encode("utf-8")).hexdigest()

PreRegLedger pydantic-model

Bases: CalibratedAgentEvaluationModel

The append-only ledger of frozen pre-registrations.

register_prereg returns a NEW ledger with the frozen entry appended; there is no update or delete path -- immutability is what makes "the metric was chosen before the fix" a guarantee rather than a convention.

Config:

  • frozen: True
  • extra: forbid

Fields:

entries pydantic-field

entries = ()

assert_held_out

assert_held_out(confirmation_ids, development_ids)

Refuse if the confirmation slice overlaps the development set.

Raises ConfirmationContaminationError naming the shared trace_id\ s. An empty confirmation slice is refused too -- there is nothing to measure a headline on (a vacuous "held out" is not held out).

Source code in src/calibrated_agent_evaluation/improve/confirm.py
def assert_held_out(confirmation_ids: Sequence[str], development_ids: Sequence[str]) -> None:
    r"""Refuse if the confirmation slice overlaps the development set.

    Raises :class:`ConfirmationContaminationError` naming the shared ``trace_id``\ s. An empty
    confirmation slice is refused too -- there is nothing to measure a headline on (a vacuous
    "held out" is not held out).
    """
    if not confirmation_ids:
        raise ImproveError(
            "assert_held_out: the confirmation slice is empty; there are no held-out traces to "
            "measure a before/after headline on"
        )
    overlap = sorted(set(confirmation_ids) & set(development_ids))
    if overlap:
        raise ConfirmationContaminationError(
            f"assert_held_out: the confirmation slice shares {len(overlap)} trace(s) with the "
            f"development set ({overlap[:5]}{'...' if len(overlap) > 5 else ''}); a headline "
            "measured on traces the fix was tuned against is regression-to-the-mean, not a "
            "fresh measurement"
        )

pair_observations

pair_observations(before, after, *, judge_produced=False)

Pair before/after FAIL verdicts on the same inputs into PairedObs.

before / after map trace_id -> before/after FAIL (True = the agent failed on that input). The two must cover exactly the same inputs -- an input present on only one side means the slice was not run through both agents on the same inputs, breaking the paired assumption, so it is refused (naming the offending ids) rather than silently dropped. judge_produced marks every observation as judge-labeled (drives the adjudication gate downstream).

Source code in src/calibrated_agent_evaluation/improve/confirm.py
def pair_observations(
    before: Mapping[str, bool], after: Mapping[str, bool], *, judge_produced: bool = False
) -> list[PairedObs]:
    """Pair before/after FAIL verdicts on the same inputs into :class:`PairedObs`.

    ``before`` / ``after`` map ``trace_id -> before/after FAIL`` (True = the agent failed on that
    input). The two must cover exactly the same inputs -- an input present on only one side means
    the slice was not run through both agents on the same inputs, breaking the paired assumption, so
    it is refused (naming the offending ids) rather than silently dropped. ``judge_produced`` marks
    every observation as judge-labeled (drives the adjudication gate downstream).
    """
    if not before or not after:
        raise ImproveError(
            "pair_observations: both the before and after label sets must be non-empty to form a "
            "paired confirmation slice"
        )
    before_ids, after_ids = set(before), set(after)
    if before_ids != after_ids:
        only_before = sorted(before_ids - after_ids)
        only_after = sorted(after_ids - before_ids)
        raise ImproveError(
            "pair_observations: the before and after label sets cover different inputs "
            f"(only-before={only_before[:5]}, only-after={only_after[:5]}); a paired before/after "
            "requires the SAME inputs run through both agents"
        )
    return [
        PairedObs(
            trace_id=tid,
            before_fail=bool(before[tid]),
            after_fail=bool(after[tid]),
            labels_are_judge_produced=judge_produced,
        )
        for tid in sorted(before_ids)
    ]

slice_id

slice_id(prefix, trace_ids)

The id of a set of traces: prefix, then a digest of its sorted trace ids.

An improve record names its confirmation slice and development set this way, so another stage can tell whether a slice it holds is one a record already decided.

Source code in src/calibrated_agent_evaluation/improve/confirm.py
def slice_id(prefix: str, trace_ids: Sequence[str]) -> str:
    """The id of a set of traces: ``prefix``, then a digest of its sorted trace ids.

    An improve record names its confirmation slice and development set this way, so another
    stage can tell whether a slice it holds is one a record already decided.
    """
    digest = hashlib.sha256("\n".join(sorted(trace_ids)).encode()).hexdigest()[:12]
    return f"{prefix}-{digest}"

compare_crew_sides

compare_crew_sides(crew, before, after, paired, *, source)

Pair before and after per trace over paired, for every agent and handoff.

Raises ImproveError naming each agent or handoff measured on one side only: its other side has no evidence, and reading that as no failures would hide a regression. source names the verdicts in that message.

Source code in src/calibrated_agent_evaluation/improve/crew.py
def compare_crew_sides(
    crew: CrewDefinition,
    before: CrewSide,
    after: CrewSide,
    paired: Collection[str],
    *,
    source: str,
) -> CrewComparison:
    """Pair ``before`` and ``after`` per trace over ``paired``, for every agent and handoff.

    Raises :class:`ImproveError` naming each agent or handoff measured on one side only: its
    other side has no evidence, and reading that as no failures would hide a regression.
    ``source`` names the verdicts in that message.
    """
    one_sided = [
        f"{what} {name!r} ({'before' if name in b else 'after'} only)"
        for what, names, b, a in (
            ("agent", crew.agents, before.agents_measured, after.agents_measured),
            ("handoff", crew.handoffs, before.handoffs_measured, after.handoffs_measured),
        )
        for name in names
        if (name in b) != (name in a)
    ]
    if one_sided:
        raise ImproveError(
            f"the {source} measure {', '.join(one_sided)} on one side only, so a change there "
            "cannot be paired; give it verdicts in both stores or fix the crew file"
        )
    ids = sorted(paired)

    def table(b: tuple[str, ...], a: tuple[str, ...]) -> dict[str, tuple[bool, bool]]:
        sb, sa = set(b), set(a)
        return {t: (t in sb, t in sa) for t in ids}

    return CrewComparison(
        before=before,
        after=after,
        agent_pairs={
            g: table(before.agent_failed[g], after.agent_failed[g])
            for g in crew.agents
            if g in before.agents_measured
        },
        handoff_pairs={
            h: table(before.handoff_failed[h], after.handoff_failed[h])
            for h in crew.handoffs
            if h in before.handoffs_measured
        },
        not_measured_agents=tuple(g for g in crew.agents if g not in before.agents_measured),
        not_measured_handoffs=tuple(h for h in crew.handoffs if h not in before.handoffs_measured),
    )

count_crew_side

count_crew_side(
    crew,
    labels,
    paired,
    *,
    is_verdict,
    attribute,
    ran=None,
    side,
)

Count one side's agent and handoff failures over the paired traces.

is_verdict says which labels are the chosen source's verdicts; attribute names the agent a verdict blames (None for none). A trace counts once for an agent when at least one FAIL verdict on it is attributed to the agent, and once for a handoff when one names it in handoff_implicated. A FAIL whose culprit_span is "disputed" blames no agent and puts its trace on disputed.

An agent is measured when any verdict on a paired trace is attributed to it. ran (trace id -> the agents with a span on it) is given when the source's verdicts speak for every agent that ran, as a human's trace-level verdict does: an agent is then also measured when it ran on a paired trace, and a handoff when both its agents ran on one. Otherwise a handoff is measured only when a verdict names it.

Raises ImproveError when a verdict is attributed to an agent, or names a handoff, that crew does not declare: the crew file and the store describe different crews.

Source code in src/calibrated_agent_evaluation/improve/crew.py
def count_crew_side(
    crew: CrewDefinition,
    labels: Mapping[str, Sequence[Label]],
    paired: Collection[str],
    *,
    is_verdict: Callable[[Label], bool],
    attribute: Callable[[Label], str | None],
    ran: Mapping[str, Collection[str]] | None = None,
    side: str,
) -> CrewSide:
    """Count one side's agent and handoff failures over the ``paired`` traces.

    ``is_verdict`` says which labels are the chosen source's verdicts; ``attribute`` names the
    agent a verdict blames (``None`` for none). A trace counts once for an agent when at least one
    FAIL verdict on it is attributed to the agent, and once for a handoff when one names it in
    ``handoff_implicated``. A FAIL whose ``culprit_span`` is ``"disputed"`` blames no agent and
    puts its trace on ``disputed``.

    An agent is measured when any verdict on a paired trace is attributed to it. ``ran`` (trace id
    -> the agents with a span on it) is given when the source's verdicts speak for every agent that
    ran, as a human's trace-level verdict does: an agent is then also measured when it ran on a
    paired trace, and a handoff when both its agents ran on one. Otherwise a handoff is measured
    only when a verdict names it.

    Raises :class:`ImproveError` when a verdict is attributed to an agent, or names a handoff, that
    ``crew`` does not declare: the crew file and the store describe different crews.
    """
    agent_failed: dict[str, set[str]] = {a: set() for a in crew.agents}
    handoff_failed: dict[str, set[str]] = {h: set() for h in crew.handoffs}
    agents_seen: set[str] = set()
    handoffs_seen: set[str] = set()
    disputed: set[str] = set()
    undeclared: dict[str, set[str]] = {}
    for trace_id in sorted(paired):
        for label in labels.get(trace_id, ()):
            if not is_verdict(label):
                continue
            failed = label.outcome == "FAIL"
            agent = attribute(label)
            if agent is not None:
                if agent not in agent_failed:
                    undeclared.setdefault(f"agent {agent!r}", set()).add(trace_id)
                    continue
                agents_seen.add(agent)
                if failed:
                    agent_failed[agent].add(trace_id)
            elif failed and label.culprit_span == DISPUTED_CULPRIT:
                disputed.add(trace_id)
            seam = label.handoff_implicated
            if seam is not None:
                if seam not in handoff_failed:
                    undeclared.setdefault(f"handoff {seam!r}", set()).add(trace_id)
                    continue
                handoffs_seen.add(seam)
                if failed:
                    handoff_failed[seam].add(trace_id)
        if ran is not None:
            here = set(ran.get(trace_id, ())) & set(agent_failed)
            agents_seen |= here
            handoffs_seen |= {h for h in crew.handoffs if set(h.split("->")) <= here}
    if undeclared:
        named = "; ".join(f"{what} on {_names(ids)}" for what, ids in sorted(undeclared.items()))
        raise ImproveError(
            f"the {side} store blames {named}, which the crew file does not declare (agents "
            f"{list(crew.agents)}, handoffs {list(crew.handoffs)}); the crew file and the store "
            "describe different crews"
        )
    return CrewSide(
        agent_failed={a: tuple(sorted(ids)) for a, ids in agent_failed.items()},
        handoff_failed={h: tuple(sorted(ids)) for h, ids in handoff_failed.items()},
        agents_measured=tuple(a for a in crew.agents if a in agents_seen),
        handoffs_measured=tuple(h for h in crew.handoffs if h in handoffs_seen),
        disputed=tuple(sorted(disputed)),
    )

crew_definition_from_file

crew_definition_from_file(doc)

Validate a parsed crew file and build its CrewDefinition.

Raises ImproveError for an unknown field, a value of the wrong type or a repeated agent or handoff, and CrewDefinitionError for whatever build_crew_definition refuses (no agents, a malformed or undeclared handoff, a scenario or rule naming an undeclared agent).

Source code in src/calibrated_agent_evaluation/improve/crew.py
def crew_definition_from_file(doc: Mapping[str, Any]) -> CrewDefinition:
    """Validate a parsed crew file and build its :class:`CrewDefinition`.

    Raises :class:`ImproveError` for an unknown field, a value of the wrong type or a repeated
    agent or handoff, and :class:`~calibrated_agent_evaluation.attribution.CrewDefinitionError` for
    whatever ``build_crew_definition`` refuses (no agents, a malformed or undeclared handoff, a
    scenario or rule naming an undeclared agent).
    """
    crew_file = CrewFile.model_validate(dict(doc))
    for what, values in (("agent", crew_file.agents), ("handoff", crew_file.handoffs)):
        repeated = sorted({v for v in values if values.count(v) > 1})
        if repeated:
            raise ImproveError(f"the crew file lists {what}(s) more than once: {repeated}")
    return build_crew_definition(
        agents=crew_file.agents,
        handoffs=crew_file.handoffs,
        scenarios={
            name: ExpectedProcess(
                scenario=name,
                expected_agents=tuple(s.expected_agents),
                terminal_decision=s.terminal_decision,
                payout_allowed=s.payout_allowed,
                escalation_required=s.escalation_required,
            )
            for name, s in crew_file.scenarios.items()
        },
        rules=[
            MandatedRule(
                rule_id=r.rule_id,
                antecedent=r.antecedent,
                consequent_agent=r.consequent_agent,
                deterministic=r.deterministic,
            )
            for r in crew_file.rules
        ],
    )

crew_outcomes

crew_outcomes(comparison, workflow)

The before and after CrewOutcomes for relocation_check.

The workflow failures are the FAILs in the (adjudicated) workflow pairs, over all of them; each agent's failures are its counted traces on that side.

Source code in src/calibrated_agent_evaluation/improve/crew.py
def crew_outcomes(
    comparison: CrewComparison, workflow: Sequence[PairedObs]
) -> tuple[CrewOutcomes, CrewOutcomes]:
    """The before and after :class:`CrewOutcomes` for ``relocation_check``.

    The workflow failures are the FAILs in the (adjudicated) workflow pairs, over all of them;
    each agent's failures are its counted traces on that side.
    """
    total = len(workflow)
    return (
        CrewOutcomes(
            workflow_failures=sum(p.before_fail for p in workflow),
            workflow_total=total,
            per_agent_failures={a: len(t) for a, t in comparison.before.agent_failed.items()},
        ),
        CrewOutcomes(
            workflow_failures=sum(p.after_fail for p in workflow),
            workflow_total=total,
            per_agent_failures={a: len(t) for a, t in comparison.after.agent_failed.items()},
        ),
    )

not_measured_rate

not_measured_rate(source)

The rate of a slice the source did not measure: no point, and the whole [0, 1] range.

Source code in src/calibrated_agent_evaluation/improve/crew.py
def not_measured_rate(source: str) -> Interval:
    """The rate of a slice the source did not measure: no point, and the whole [0, 1] range."""
    return Interval(
        point=None,
        lo=0.0,
        hi=1.0,
        refusal_reason=f"not measured by the {source}",
        method=METHOD_NOT_MEASURED,
        level=0.95,
        sided="two-sided",
        n=0,
    )

evaluate_improvement

evaluate_improvement(request)

Evaluate one proposed improvement against all four causal guards.

Runs the guards in order (pre-registration -> held-out -> adjudication -> McNemar -> panel) and returns the composed ImproveVerdict. Raises before returning a verdict when a guard refuses: ImproveError (unregistered/mutated prereg, a ledger that disagrees with the entries in anchors, a pre-registration frozen after a time in after_run_at, a trace paired twice, a success_threshold outside (0, 1) or differing from the caller's, empty input), ConfirmationContaminationError, AdjudicationRequiredError, or StatisticsInputError (a forbidden multiplicity method). For a crew request (crew/crew_before/crew_after/touched_agent set), the panel is the crew panel with the relocation fold-in.

Parameters:

Name Type Description Default
request ImproveRequest

The pre-registration, the paired before/after observations and the guard slices.

required

Returns:

Type Description
ImproveVerdict

Whether the improvement is accepted, with each guard's result.

Source code in src/calibrated_agent_evaluation/improve/guard.py
def evaluate_improvement(request: ImproveRequest) -> ImproveVerdict:
    """Evaluate one proposed improvement against all four causal guards.

    Runs the guards in order (pre-registration -> held-out -> adjudication -> McNemar -> panel) and
    returns the composed :class:`ImproveVerdict`. Raises before returning a verdict when a guard
    refuses: :class:`~calibrated_agent_evaluation.improve.ImproveError` (unregistered/mutated
    prereg, a ledger that disagrees with the entries in ``anchors``, a pre-registration frozen after
    a time in ``after_run_at``, a trace paired twice, a ``success_threshold`` outside (0, 1) or
    differing from the caller's, empty input),
    :class:`~calibrated_agent_evaluation.improve.ConfirmationContaminationError`,
    :class:`~calibrated_agent_evaluation.improve.AdjudicationRequiredError`, or
    :class:`~calibrated_agent_evaluation.calibrate.errors.StatisticsInputError` (a forbidden
    multiplicity method). For a crew request
    (``crew``/``crew_before``/``crew_after``/``touched_agent`` set), the panel is the crew panel
    with the relocation fold-in.

    Args:
        request: The pre-registration, the paired before/after observations and the guard slices.

    Returns:
        Whether the improvement is accepted, with each guard's result.
    """
    verify_anchors(request.ledger, request.anchors)
    prereg = require_prereg(request.ledger, request.target_metric)
    require_frozen_before(prereg, request.after_run_started_at)
    success_threshold = _success_threshold(request, prereg)
    assert_held_out(request.confirmation_ids, request.development_ids)

    judge_produced = any(p.labels_are_judge_produced for p in request.pairs)
    discordant = discordant_pairs(request.pairs)
    require_adjudication(
        discordant,
        request.adjudications,
        labels_are_judge_produced=judge_produced,
        target_metric=request.target_metric,
    )
    pairs = adjudicate_pairs(
        request.pairs, request.adjudications, target_metric=request.target_metric
    )

    paired = paired_mcnemar(pairs)

    if request.crew is not None:
        if (
            request.crew_before is None
            or request.crew_after is None
            or request.touched_agent is None
        ):
            raise _crew_incomplete(prereg)
        panel = crew_panel(
            request.crew,
            request.crew_before,
            request.crew_after,
            touched_agent=request.touched_agent,
            slices=request.slices,
        )
    else:
        panel = guard_panel(request.slices)

    verdict, reasons = _verdict(paired, panel, success_threshold)
    return ImproveVerdict(
        verdict=verdict,  # type: ignore[arg-type]  # "accepted"|"rejected" from _verdict
        prereg=prereg,
        paired=paired,
        panel=panel,
        reasons=reasons,
    )

record_of

record_of(
    verdict,
    *,
    improve_id,
    confirmation_slice_id,
    development_set_id,
    inputs,
)

Assemble the persisted ImproveRecord from a verdict.

A thin constructor so the caller (or the CLI) can persist the audit artifact via dump_improve_records. inputs must be non-empty (the record refuses otherwise).

Source code in src/calibrated_agent_evaluation/improve/guard.py
def record_of(
    verdict: ImproveVerdict,
    *,
    improve_id: str,
    confirmation_slice_id: str,
    development_set_id: str,
    inputs: dict[str, str],
) -> ImproveRecord:
    """Assemble the persisted :class:`ImproveRecord` from a verdict.

    A thin constructor so the caller (or the CLI) can persist the audit artifact via
    :func:`calibrated_agent_evaluation.improve.dump_improve_records`. ``inputs`` must be non-empty
    (the record refuses otherwise).
    """
    thresholds = {
        "alpha": _ALPHA,
        "alpha_one_sided": ALPHA_ONE_SIDED,
        # The key names the method: the guard panel is Holm at this family-wise alpha.
        "guard_holm_alpha": GUARD_FWER_ALPHA,
        "success_threshold": verdict.prereg.success_threshold,
    }
    return ImproveRecord(
        improve_id=improve_id,
        verdict=verdict.verdict,
        prereg=verdict.prereg,
        paired=verdict.paired,
        panel=verdict.panel,
        confirmation_slice_id=confirmation_slice_id,
        development_set_id=development_set_id,
        inputs=inputs,
        reasons=verdict.reasons,
        # McNemar's exact test and the Holm panel draw no random numbers: no seed to record.
        provenance=capture_provenance(thresholds=thresholds),
    )

adjudicate_pairs

adjudicate_pairs(pairs, adjudications, *, target_metric)

Return pairs with each judge-labeled discordant pair's after outcome set by its human.

The human's verdict replaces the judge's after-run verdict, so McNemar and the effect interval count what the human saw: a flip the human says did not happen becomes a concordant pair. Call require_adjudication first; a pair this cannot adjudicate raises AdjudicationRequiredError. Concordant pairs and human-gold pairs are kept as given.

Source code in src/calibrated_agent_evaluation/improve/paired.py
def adjudicate_pairs(
    pairs: Sequence[PairedObs], adjudications: Mapping[str, Label], *, target_metric: str
) -> tuple[PairedObs, ...]:
    """Return ``pairs`` with each judge-labeled discordant pair's after outcome set by its human.

    The human's verdict replaces the judge's after-run verdict, so McNemar and the effect interval
    count what the human saw: a flip the human says did not happen becomes a concordant pair. Call
    :func:`require_adjudication` first; a pair this cannot adjudicate raises
    :class:`AdjudicationRequiredError`. Concordant pairs and human-gold pairs are kept as given.
    """
    out: list[PairedObs] = []
    for pair in pairs:
        if not pair.labels_are_judge_produced or pair.before_fail == pair.after_fail:
            out.append(pair)
            continue
        label = adjudications.get(pair.trace_id)
        reason = adjudication_problem(pair.trace_id, label, target_metric)
        if label is None or reason is not None:
            raise AdjudicationRequiredError(
                f"adjudicate_pairs: discordant pair {pair.trace_id!r} has no usable human verdict "
                f"({reason}); run require_adjudication first"
            )
        out.append(replace(pair, after_fail=label.outcome == "FAIL"))
    return tuple(out)

adjudication_problem

adjudication_problem(trace_id, label, target_metric)

Why label cannot adjudicate trace_id for target_metric, or None if it can.

A usable adjudication is a human's trace-level label on that trace that speaks to the target: a FAIL for target_metric, or a PASS for it or for no mode.

Source code in src/calibrated_agent_evaluation/improve/paired.py
def adjudication_problem(trace_id: str, label: Label | None, target_metric: str) -> str | None:
    """Why ``label`` cannot adjudicate ``trace_id`` for ``target_metric``, or ``None`` if it can.

    A usable adjudication is a human's trace-level label on that trace that speaks to the target:
    a FAIL for ``target_metric``, or a PASS for it or for no mode.
    """
    if label is None:
        return "no human label"
    if label.kind != "human":
        return f"a {label.kind!r} label, not a human one"
    if label.trace_id != trace_id:
        return f"a label for another trace ({label.trace_id!r})"
    if label.span_id is not None:
        return "a span-level label, not a verdict on the whole trace"
    if label.outcome == "FAIL" and label.failure_mode == target_metric:
        return None
    if label.outcome == "PASS" and label.failure_mode in (None, target_metric):
        return None
    return f"a human {label.outcome} about another failure mode ({label.failure_mode!r})"

discordant_pairs

discordant_pairs(pairs)

Return the discordant (flipped) pairs -- the b + c cells McNemar counts.

Source code in src/calibrated_agent_evaluation/improve/paired.py
def discordant_pairs(pairs: Sequence[PairedObs]) -> tuple[PairedObs, ...]:
    """Return the discordant (flipped) pairs -- the ``b`` + ``c`` cells McNemar counts."""
    return tuple(p for p in pairs if p.before_fail != p.after_fail)

discovery_enqueue

discovery_enqueue(discordant)

Route the discordant pairs into the labeling queue tagged frame="discovery".

Reuses DiscoveryEnqueue: a discordant pair a judge disagreed with itself on is non-random augmentation (it grows the discovery set) and must never enter a production-frame rate (the assert_production_frame guard enforces that downstream).

Source code in src/calibrated_agent_evaluation/improve/paired.py
def discovery_enqueue(discordant: Sequence[PairedObs]) -> tuple[DiscoveryEnqueue, ...]:
    """Route the discordant pairs into the labeling queue tagged ``frame="discovery"``.

    Reuses :class:`~calibrated_agent_evaluation.run.DiscoveryEnqueue`: a discordant pair a judge
    disagreed with itself on is non-random augmentation (it grows the discovery set) and must never
    enter a production-frame rate (the ``assert_production_frame`` guard enforces that downstream).
    """
    return tuple(DiscoveryEnqueue(trace_id=p.trace_id) for p in discordant)

paired_mcnemar

paired_mcnemar(pairs)

Run the paired McNemar test over pairs.

Auto-selects the exact/mid-p variant when the discordant count is < 25 and the asymptotic one otherwise (via mcnemar_paired). A zero-discordant slice is surfaced as McNemarResult(method="undefined", p_value=None, ...) with the effect still reported (a genuine 0.0 change) -- never a fabricated p-value. Raises ImproveError on empty input or a trace id that appears in more than one pair.

Source code in src/calibrated_agent_evaluation/improve/paired.py
def paired_mcnemar(pairs: Sequence[PairedObs]) -> McNemarResult:
    """Run the paired McNemar test over ``pairs``.

    Auto-selects the exact/mid-p variant when the discordant count is < 25 and the asymptotic one
    otherwise (via :func:`calibrated_agent_evaluation.stats.mcnemar_paired`). A **zero-discordant**
    slice is surfaced as ``McNemarResult(method="undefined", p_value=None, ...)`` with the effect
    still reported (a genuine 0.0 change) -- never a fabricated p-value. Raises
    :class:`ImproveError` on empty input or a trace id that appears in more than one pair.
    """
    if not pairs:
        raise ImproveError(
            "paired_mcnemar: no paired observations; refusing to run a before/after test over an "
            "empty confirmation slice"
        )
    a, b, c, d = _counts(pairs)
    effect = _paired_effect_interval(a, b, c, d)
    n_discordant = b + c
    if n_discordant == 0:
        # No flipped pairs: McNemar has no information. Report an honest undefined result rather
        # than fabricate a p-value; the effect is a real 0.0 change and is still carried.
        return McNemarResult(
            method="undefined",
            n_discordant=0,
            p_value=None,
            effect=effect,
            undefined_reason=(
                "zero discordant pairs (no before/after flips); McNemar's test is undefined"
            ),
        )
    # midp is meaningful only on the small-sample exact path; mcnemar_paired ignores it above 25.
    use_midp = n_discordant < 25
    stat = mcnemar_paired([[a, b], [c, d]], midp=use_midp)
    return McNemarResult(
        method=stat.method,
        n_discordant=stat.n_discordant,
        p_value=stat.p_value,
        effect=effect,
        undefined_reason=None,
    )

require_adjudication

require_adjudication(
    discordant,
    adjudications,
    *,
    labels_are_judge_produced,
    target_metric,
)

Refuse McNemar until every judge-labeled discordant pair has a usable human verdict.

When labels_are_judge_produced is False the before/after labels are human gold and no adjudication is required (returns immediately). When True, every discordant trace_id must map to a Label that is a human's trace-level verdict on that trace's after-run output for target_metric: a FAIL for target_metric, or a PASS for it or for no mode. Anything else raises AdjudicationRequiredError, naming the traces: a missing entry, a non-human (unratified) label, a span-level label, a label for another trace, or one about another failure mode, which says nothing about whether the target was fixed.

Source code in src/calibrated_agent_evaluation/improve/paired.py
def require_adjudication(
    discordant: Sequence[PairedObs],
    adjudications: Mapping[str, Label],
    *,
    labels_are_judge_produced: bool,
    target_metric: str,
) -> None:
    """Refuse McNemar until every judge-labeled discordant pair has a usable human verdict.

    When ``labels_are_judge_produced`` is False the before/after labels are human gold and no
    adjudication is required (returns immediately). When True, every discordant ``trace_id`` must
    map to a :class:`Label` that is a human's trace-level verdict on that trace's after-run output
    for ``target_metric``: a FAIL for ``target_metric``, or a PASS for it or for no mode. Anything
    else raises :class:`AdjudicationRequiredError`, naming the traces: a missing entry, a non-human
    (unratified) label, a span-level label, a label for another trace, or one about another failure
    mode, which says nothing about whether the target was fixed.
    """
    if not labels_are_judge_produced:
        return
    problems: dict[str, list[str]] = {}
    for pair in discordant:
        reason = adjudication_problem(
            pair.trace_id, adjudications.get(pair.trace_id), target_metric
        )
        if reason is not None:
            problems.setdefault(reason, []).append(pair.trace_id)
    if problems:
        found = "; ".join(
            f"{reason} for {len(ids)} trace(s) ({', '.join(sorted(ids)[:5])})"
            for reason, ids in problems.items()
        )
        raise AdjudicationRequiredError(
            "require_adjudication: the before/after labels are judge-produced, so every discordant "
            "pair needs a human verdict on the after run before McNemar runs (a judge's run-to-run "
            f"flips land in exactly the discordant cells the test counts). Found {found}. Next: "
            f"route them to review (frame='discovery') and label each trace PASS or FAIL for "
            f"{target_metric!r} at trace level."
        )

build_crew_slices

build_crew_slices(
    crew,
    *,
    touched_agent,
    agent_regression_p,
    agent_rates,
    handoff_regression_p,
    handoff_rates,
)

Derive the per-other-agent + per-declared-handoff guard slices for a crew.

Includes a slice for every crew agent EXCEPT touched_agent (the guard panel watches the agents the fix did NOT target), and a slice for every declared handoff in crew.handoffs. Each slice carries its before/after Interval rates and the p-value that it regressed. A handoff_regression_p key naming a seam not in crew.handoffs, or an agent_regression_p key naming an undeclared agent, is refused (the crew seam guard).

Source code in src/calibrated_agent_evaluation/improve/panel.py
def build_crew_slices(
    crew: CrewDefinition,
    *,
    touched_agent: str,
    agent_regression_p: dict[str, float],
    agent_rates: dict[str, tuple[Interval, Interval]],
    handoff_regression_p: dict[str, float],
    handoff_rates: dict[str, tuple[Interval, Interval]],
) -> tuple[GuardSlice, ...]:
    """Derive the per-other-agent + per-declared-handoff guard slices for a crew.

    Includes a slice for every crew agent EXCEPT ``touched_agent`` (the guard panel watches the
    agents the fix did NOT target), and a slice for every declared handoff in ``crew.handoffs``.
    Each slice carries its before/after :class:`Interval` rates and the p-value that it regressed. A
    ``handoff_regression_p`` key naming a seam not in ``crew.handoffs``, or an
    ``agent_regression_p`` key naming an undeclared agent, is refused (the crew seam guard).
    """
    for agent in agent_regression_p:
        if agent not in crew.agents:
            raise ImproveError(
                f"build_crew_slices: agent {agent!r} is not a declared crew agent {crew.agents} "
            )
    for seam in handoff_regression_p:
        if seam not in crew.handoffs:
            raise ImproveError(
                f"build_crew_slices: handoff {seam!r} is not a declared crew handoff "
                f"{crew.handoffs}; refusing to guard an undeclared seam"
            )
    # Completeness (the reverse direction): every non-touched agent + every declared handoff must
    # be supplied in BOTH its regression-p and its rates map. A missing key would otherwise surface
    # as a raw KeyError off this public surface -- refuse with a typed error instead (no silent
    # failure; the caller cannot catch a family it can't name).
    expected_agents = [a for a in crew.agents if a != touched_agent]
    for agent in expected_agents:
        missing = [
            n
            for n, d in (("agent_regression_p", agent_regression_p), ("agent_rates", agent_rates))
            if agent not in d
        ]
        if missing:
            raise ImproveError(
                f"build_crew_slices: guarded agent {agent!r} is missing from {missing} "
                "(every non-touched crew agent needs a regression p-value and before/after rates)"
            )
    for seam in crew.handoffs:
        missing = [
            n
            for n, d in (
                ("handoff_regression_p", handoff_regression_p),
                ("handoff_rates", handoff_rates),
            )
            if seam not in d
        ]
        if missing:
            raise ImproveError(
                f"build_crew_slices: declared handoff {seam!r} is missing from {missing} "
                "(every declared handoff needs a regression p-value and before/after rates)"
            )
    out: list[GuardSlice] = []
    for agent in crew.agents:
        if agent == touched_agent:
            continue
        rate_before, rate_after = agent_rates[agent]
        out.append(
            GuardSlice(
                name=agent,
                kind="agent",
                regression_p=agent_regression_p[agent],
                rate_before=rate_before,
                rate_after=rate_after,
            )
        )
    for seam in crew.handoffs:
        rate_before, rate_after = handoff_rates[seam]
        out.append(
            GuardSlice(
                name=seam,
                kind="handoff",
                regression_p=handoff_regression_p[seam],
                rate_before=rate_before,
                rate_after=rate_after,
            )
        )
    return tuple(out)

crew_panel

crew_panel(crew, before, after, *, touched_agent, slices)

Build the crew guard panel: agent+handoff slices + the relocation fold-in.

slices are the caller-derived per-agent and per-handoff guard slices (each kind="agent" or "handoff", its name an agent in crew.agents or a handoff in crew.handoffs -- see build_crew_slices). This runs the Holm panel over them AND folds in relocation_check over before/after at touched_agent so the workflow-outcome-rate verdict + the failure-relocation flag are attached to the report. Refuses a touched_agent absent from crew.agents.

Source code in src/calibrated_agent_evaluation/improve/panel.py
def crew_panel(
    crew: CrewDefinition,
    before: CrewOutcomes,
    after: CrewOutcomes,
    *,
    touched_agent: str,
    slices: Sequence[GuardSlice],
) -> GuardPanelReport:
    """Build the crew guard panel: agent+handoff slices + the relocation fold-in.

    ``slices`` are the caller-derived per-agent and per-handoff guard slices (each ``kind="agent"``
    or ``"handoff"``, its ``name`` an agent in ``crew.agents`` or a handoff in ``crew.handoffs`` --
    see :func:`build_crew_slices`). This runs the Holm panel over them AND folds in
    :func:`relocation_check` over ``before``/``after`` at ``touched_agent`` so the
    workflow-outcome-rate verdict + the failure-relocation flag are attached to the report. Refuses
    a ``touched_agent`` absent from ``crew.agents``.
    """
    if touched_agent not in crew.agents:
        raise ImproveError(
            f"crew_panel: touched_agent {touched_agent!r} is not a declared crew agent "
            f"{crew.agents}; cannot gate a relocation check on an unknown agent"
        )
    base = guard_panel(slices)
    relocation = relocation_check(before, after, touched_agent=touched_agent)
    return GuardPanelReport(
        slices=base.slices,
        method=base.method,
        m=base.m,
        regressed=base.regressed,
        relocation=relocation,
    )

guard_panel

guard_panel(slices, *, alpha=GUARD_FWER_ALPHA)

Correct the guard-slice regression p-values with Holm and name any survivor.

slices each carry a regression_p (the p-value that the slice worsened). The panel corrects them with holm at family-wise level alpha (via apply_multiplicity -> multiplicity; fdr_bh/unknown methods are refused by the primitive as StatisticsInputError), states m = the slice count, and lists in regressed the slices whose regression survived correction. Raises ImproveError on an empty panel -- a declared-but-empty guard panel is a caller bug, not a vacuous "no regression".

Source code in src/calibrated_agent_evaluation/improve/panel.py
def guard_panel(
    slices: Sequence[GuardSlice], *, alpha: float = GUARD_FWER_ALPHA
) -> GuardPanelReport:
    """Correct the guard-slice regression p-values with Holm and name any survivor.

    ``slices`` each carry a ``regression_p`` (the p-value that the slice worsened). The panel
    corrects them with ``holm`` at family-wise level ``alpha`` (via
    :func:`calibrated_agent_evaluation.run.apply_multiplicity` -> ``multiplicity``;
    ``fdr_bh``/unknown methods are refused by the primitive as :class:`StatisticsInputError`),
    states ``m`` = the slice count, and lists in
    ``regressed`` the slices whose regression survived correction. Raises :class:`ImproveError` on
    an empty panel -- a declared-but-empty guard panel is a caller bug, not a vacuous "no
    regression".
    """
    if not slices:
        raise ImproveError(
            "guard_panel: the guard-slice panel is empty; refusing to return a vacuous "
            "'no regression' for a declared-but-empty panel"
        )
    result = apply_multiplicity(
        [s.regression_p for s in slices], family="improve_guard", method="holm", alpha=alpha
    )
    regressed = tuple(s.name for s, rej in zip(slices, result.reject, strict=True) if rej)
    return GuardPanelReport(
        slices=tuple(slices),
        method="holm",
        m=result.m,
        regressed=regressed,
        relocation=None,
    )

freeze

freeze(entry, *, prev_hash=None, now=None)

Return entry stamped with frozen_at, prev_hash and its content_hash.

frozen_at is the current UTC time (now exists for tests and must be timezone-aware); prev_hash is the hash of the ledger entry this one follows, which register_prereg supplies. code_ref is kept as given. The stamped entry is what register_prereg stores; require_prereg recomputes the hash on read and refuses a mismatch. Refuses a success_threshold outside (0, 1): it is a minimum absolute reduction in fail rate, so 0 would accept any significant drop and 1 or more can never be met.

Source code in src/calibrated_agent_evaluation/improve/prereg.py
def freeze(
    entry: PreRegEntry, *, prev_hash: str | None = None, now: datetime | None = None
) -> PreRegEntry:
    """Return ``entry`` stamped with ``frozen_at``, ``prev_hash`` and its ``content_hash``.

    ``frozen_at`` is the current UTC time (``now`` exists for tests and must be timezone-aware);
    ``prev_hash`` is the hash of the ledger entry this one follows, which :func:`register_prereg`
    supplies. ``code_ref`` is kept as given. The stamped entry is what :func:`register_prereg`
    stores; :func:`require_prereg` recomputes the hash on read and refuses a mismatch. Refuses a
    ``success_threshold`` outside (0, 1): it is a minimum absolute reduction in fail rate, so 0
    would accept any significant drop and 1 or more can never be met.
    """
    if entry.content_hash or entry.frozen_at:
        raise LedgerIntegrityError(
            f"freeze: the pre-registration for {entry.target_metric!r} is already frozen (at "
            f"{entry.frozen_at or 'an unrecorded time'}); a frozen entry is never re-frozen, since "
            "that would give an edited threshold a fresh, consistent hash. Register a new metric."
        )
    if not 0.0 < entry.success_threshold < 1.0:
        raise ImproveError(
            f"freeze: success_threshold {entry.success_threshold!r} for {entry.target_metric!r} is "
            "not in (0, 1); it is the minimum absolute reduction in fail rate the fix must show "
            ""
        )
    stamped = PreRegEntry(
        prereg_id=entry.prereg_id,
        target_metric=entry.target_metric,
        success_threshold=entry.success_threshold,
        # nosemgrep: is-function-without-parentheses  -- bool dataclass field, not a method
        is_crew=entry.is_crew,
        frozen_at=_stamp(now),
        prev_hash=prev_hash,
        code_ref=entry.code_ref,
    )
    return PreRegEntry(
        prereg_id=stamped.prereg_id,
        target_metric=stamped.target_metric,
        success_threshold=stamped.success_threshold,
        # nosemgrep: is-function-without-parentheses  -- bool dataclass field, not a method
        is_crew=stamped.is_crew,
        content_hash=stamped.compute_hash(),
        frozen_at=stamped.frozen_at,
        prev_hash=stamped.prev_hash,
        code_ref=stamped.code_ref,
    )

ledger_head

ledger_head(ledger)

The ledger's head: its last entry's content_hash, or "empty" for no entries.

The hash chain makes the head name the whole ledger up to it, so a record stamped with the head when it was made can later tell whether the ledger it was made against still exists.

Source code in src/calibrated_agent_evaluation/improve/prereg.py
def ledger_head(ledger: PreRegLedger) -> str:
    """The ledger's head: its last entry's ``content_hash``, or ``"empty"`` for no entries.

    The hash chain makes the head name the whole ledger up to it, so a record stamped with the
    head when it was made can later tell whether the ledger it was made against still exists.
    """
    return ledger.entries[-1].content_hash if ledger.entries else EMPTY_LEDGER_HEAD

register_prereg

register_prereg(ledger, entry, *, now=None)

Append entry (freezing it, chained to the last entry) to ledger; return a NEW ledger.

Refuses a duplicate target_metric -- a second pre-registration for the same metric would let a later, result-fitted threshold shadow the one committed before the fix (the exact evasion the ledger exists to stop). The returned ledger is a new object; the input is unchanged.

Source code in src/calibrated_agent_evaluation/improve/prereg.py
def register_prereg(
    ledger: PreRegLedger, entry: PreRegEntry, *, now: datetime | None = None
) -> PreRegLedger:
    """Append ``entry`` (freezing it, chained to the last entry) to ``ledger``; return a NEW ledger.

    Refuses a duplicate ``target_metric`` -- a second pre-registration for the same metric would let
    a later, result-fitted threshold shadow the one committed before the fix (the exact evasion the
    ledger exists to stop). The returned ledger is a new object; the input is unchanged.
    """
    prev = ledger.entries[-1].content_hash if ledger.entries else None
    frozen = freeze(entry, prev_hash=prev, now=now)
    if any(e.target_metric == frozen.target_metric for e in ledger.entries):
        raise ImproveError(
            f"register_prereg: metric {frozen.target_metric!r} is already pre-registered; the "
            "ledger is append-only and a metric is registered exactly once, before its fix "
            "(re-registering would let a result-fitted threshold shadow the committed one)"
        )
    return PreRegLedger(entries=(*ledger.entries, frozen))

require_frozen_before

require_frozen_before(entry, after_run_started_at)

Refuse entry if it was frozen at or after after_run_started_at.

A pre-registration exists to fix the bar before the result is known, so it must predate the after run it gates. None means the after run recorded no time, and nothing is checked; the caller says so in its output. Raises LedgerIntegrityError when the order is wrong and ImproveError for a naive after_run_started_at.

Source code in src/calibrated_agent_evaluation/improve/prereg.py
def require_frozen_before(entry: PreRegEntry, after_run_started_at: datetime | None) -> None:
    """Refuse ``entry`` if it was frozen at or after ``after_run_started_at``.

    A pre-registration exists to fix the bar before the result is known, so it must predate the
    after run it gates. ``None`` means the after run recorded no time, and nothing is checked; the
    caller says so in its output. Raises :class:`LedgerIntegrityError` when the order is wrong and
    :class:`ImproveError` for a naive ``after_run_started_at``.
    """
    if after_run_started_at is None:
        return
    if after_run_started_at.utcoffset() is None:
        raise ImproveError("require_frozen_before: after_run_started_at must be timezone-aware")
    frozen = datetime.fromisoformat(entry.frozen_at)
    if frozen >= after_run_started_at:
        raise LedgerIntegrityError(
            f"require_frozen_before: the pre-registration for {entry.target_metric!r} was frozen "
            f"at {entry.frozen_at}, after the after run began "
            f"({after_run_started_at.astimezone(UTC).isoformat()}); a threshold set once the "
            "result could be seen does not gate it. Register before the fix and measure the after "
            "run again."
        )

require_prereg

require_prereg(ledger, metric)

Return the frozen pre-registration for metric; refuse if absent, mutated, or unchained.

Verifies the whole ledger first (verify_ledger), so an entry whose metric, threshold, or crew flag was edited after freeze, or a ledger whose chain was broken by a re-freeze, is refused before any headline is measured. An unregistered metric is likewise refused rather than silently measured.

Source code in src/calibrated_agent_evaluation/improve/prereg.py
def require_prereg(ledger: PreRegLedger, metric: str) -> PreRegEntry:
    """Return the frozen pre-registration for ``metric``; refuse if absent, mutated, or unchained.

    Verifies the whole ledger first (:func:`verify_ledger`), so an entry whose metric, threshold,
    or crew flag was edited after :func:`freeze`, or a ledger whose chain was broken by a
    re-freeze, is refused before any headline is measured. An unregistered metric is likewise
    refused rather than silently measured.
    """
    for entry in ledger.entries:
        if entry.target_metric == metric and entry.content_hash != entry.compute_hash():
            raise ImproveError(
                f"require_prereg: pre-registration for {metric!r} was mutated after it was "
                "frozen (content hash mismatch); refusing to measure a headline against a "
                "pre-registration edited after the fix"
            )
    verify_ledger(ledger)
    for entry in ledger.entries:
        if entry.target_metric == metric:
            return entry
    raise ImproveError(
        f"require_prereg: metric {metric!r} was never pre-registered; refusing to compute a "
        "before/after headline for a metric chosen after the fix"
    )

verify_anchors

verify_anchors(ledger, anchors)

Refuse a ledger that no longer holds, unchanged, every entry in anchors.

anchors are the pre-registrations earlier evaluations recorded (each improve record keeps the entry it gated on). An anchor missing from the ledger, or present with a different hash, means the ledger was edited, replaced or deleted after that evaluation, so a threshold it now holds may have been chosen after a result was seen. Raises LedgerIntegrityError naming the first such entry.

Source code in src/calibrated_agent_evaluation/improve/prereg.py
def verify_anchors(ledger: PreRegLedger, anchors: Sequence[PreRegEntry]) -> None:
    """Refuse a ledger that no longer holds, unchanged, every entry in ``anchors``.

    ``anchors`` are the pre-registrations earlier evaluations recorded (each improve record keeps
    the entry it gated on). An anchor missing from the ledger, or present with a different hash,
    means the ledger was edited, replaced or deleted after that evaluation, so a threshold it now
    holds may have been chosen after a result was seen. Raises :class:`LedgerIntegrityError`
    naming the first such entry.
    """
    held = {entry.content_hash: entry for entry in ledger.entries}
    by_id = {entry.prereg_id: entry for entry in ledger.entries}
    for anchor in anchors:
        if held.get(anchor.content_hash) == anchor:
            continue
        now = by_id.get(anchor.prereg_id)
        state = (
            "is missing from the ledger"
            if now is None
            else f"now reads threshold {now.success_threshold} frozen at {now.frozen_at}"
        )
        raise LedgerIntegrityError(
            f"verify_anchors: an earlier evaluation recorded pre-registration {anchor.prereg_id!r} "
            f"({anchor.target_metric!r}, threshold {anchor.success_threshold}, frozen at "
            f"{anchor.frozen_at}, hash {anchor.content_hash[:12]}), and it {state}; the ledger was "
            "edited, replaced or deleted after that evaluation. Restore the ledger it was recorded "
            "against (from version control or a `calibrated-agent-evaluation dump`) before "
            "evaluating."
        )

verify_heads

verify_heads(ledger, stamped)

Refuse a ledger that no longer holds a head a stored record was stamped with.

stamped pairs a description of each record ("run record 'run-1'") with the head it carries. A head is held when it is "empty" or the content_hash of an entry still in the ledger. Raises LedgerIntegrityError naming the first record whose head is gone.

Source code in src/calibrated_agent_evaluation/improve/prereg.py
def verify_heads(ledger: PreRegLedger, stamped: Iterable[tuple[str, str]]) -> None:
    """Refuse a ledger that no longer holds a head a stored record was stamped with.

    ``stamped`` pairs a description of each record (``"run record 'run-1'"``) with the head it
    carries. A head is held when it is ``"empty"`` or the ``content_hash`` of an entry still in
    the ledger. Raises :class:`LedgerIntegrityError` naming the first record whose head is gone.
    """
    held = {entry.content_hash for entry in ledger.entries} | {EMPTY_LEDGER_HEAD}
    for what, head in stamped:
        if head in held:
            continue
        raise LedgerIntegrityError(
            f"the {what} was made when the pre-registration ledger ended at entry {head[:12]}, "
            "and the ledger no longer holds that entry, so it was edited, replaced or deleted "
            "after that record was made. Restore the ledger the record was made against (from "
            "version control or a `calibrated-agent-evaluation dump`) before registering or "
            "evaluating."
        )

verify_ledger

verify_ledger(ledger)

Refuse a ledger whose hashes, chain, or freeze times do not hold.

Every entry must hash to its content_hash; the first must have no prev_hash and each later one must name its predecessor's hash; frozen_at must be a UTC timestamp that never goes backwards. Raises ImproveError naming the first broken entry.

Source code in src/calibrated_agent_evaluation/improve/prereg.py
def verify_ledger(ledger: PreRegLedger) -> None:
    """Refuse a ledger whose hashes, chain, or freeze times do not hold.

    Every entry must hash to its ``content_hash``; the first must have no ``prev_hash`` and each
    later one must name its predecessor's hash; ``frozen_at`` must be a UTC timestamp that never
    goes backwards. Raises :class:`ImproveError` naming the first broken entry.
    """
    prev_hash: str | None = None
    prev_time: datetime | None = None
    for i, entry in enumerate(ledger.entries):
        if entry.content_hash != entry.compute_hash():
            raise ImproveError(
                f"verify_ledger: entry {i} ({entry.target_metric!r}) does not hash to its "
                "content_hash; it was edited after it was frozen"
            )
        if entry.prev_hash != prev_hash:
            raise ImproveError(
                f"verify_ledger: entry {i} ({entry.target_metric!r}) does not chain to the entry "
                "before it; an earlier entry was edited and re-frozen, or entries were reordered"
            )
        try:
            when = datetime.fromisoformat(entry.frozen_at)
        except ValueError as exc:
            raise ImproveError(
                f"verify_ledger: entry {i} has no valid frozen_at ({entry.frozen_at!r})"
            ) from exc
        if when.utcoffset() is None:
            raise ImproveError(f"verify_ledger: entry {i} frozen_at {entry.frozen_at!r} is naive")
        if prev_time is not None and when < prev_time:
            raise ImproveError(
                f"verify_ledger: entry {i} was frozen at {entry.frozen_at}, before the entry it "
                "follows"
            )
        prev_hash, prev_time = entry.content_hash, when

dump_improve_records

dump_improve_records(records, path, fmt='csv')

Write records to improve_record.{csv,json} under path.

Parameters:

Name Type Description Default
records Sequence[ImproveRecord]

The improve records to persist.

required
path str | Path

The directory to write into.

required
fmt str

Which file a later load_improve_records reads back (both are always written).

'csv'

Raises:

Type Description
ImproveStoreError

A bad fmt or any filesystem failure.

Source code in src/calibrated_agent_evaluation/improve/store.py
def dump_improve_records(
    records: Sequence[ImproveRecord], path: str | Path, fmt: str = "csv"
) -> None:
    """Write ``records`` to ``improve_record.{csv,json}`` under ``path``.

    Args:
        records: The improve records to persist.
        path: The directory to write into.
        fmt: Which file a later :func:`load_improve_records` reads back (both are always written).

    Raises:
        ImproveStoreError: A bad ``fmt`` or any filesystem failure.
    """
    _STORE.dump(records, path, fmt)

dump_prereg_ledger

dump_prereg_ledger(ledger, path)

Write ledger to prereg_ledger.json under path, entries in ledger order.

Every entry keeps its content_hash, frozen_at and prev_hash as stamped, so load_prereg_ledger returns a ledger verify_ledger accepts iff the original did. The ledger is append-only on disk too: a ledger that changes or drops an entry already in the file is refused, which is how an edited and re-frozen last entry would otherwise be saved.

Parameters:

Name Type Description Default
ledger PreRegLedger

The ledger to persist.

required
path str | Path

The directory to write into.

required

Raises:

Type Description
LedgerIntegrityError

ledger does not extend the ledger already in path.

ImproveStoreError

Any filesystem failure, or a stored ledger that cannot be read.

Source code in src/calibrated_agent_evaluation/improve/store.py
def dump_prereg_ledger(ledger: PreRegLedger, path: str | Path) -> None:
    """Write ``ledger`` to ``prereg_ledger.json`` under ``path``, entries in ledger order.

    Every entry keeps its ``content_hash``, ``frozen_at`` and ``prev_hash`` as stamped, so
    :func:`load_prereg_ledger` returns a ledger
    :func:`~calibrated_agent_evaluation.improve.verify_ledger` accepts iff the original did. The
    ledger is append-only on disk too: a ledger that changes or drops an entry already in the file
    is refused, which is how an edited and re-frozen last entry would otherwise be saved.

    Args:
        ledger: The ledger to persist.
        path: The directory to write into.

    Raises:
        LedgerIntegrityError: ``ledger`` does not extend the ledger already in ``path``.
        ImproveStoreError: Any filesystem failure, or a stored ledger that cannot be read.
    """
    out = Path(path)
    stored = load_prereg_ledger(out).entries
    if ledger.entries[: len(stored)] != stored:
        changed = next(
            (i for i, e in enumerate(stored) if i >= len(ledger.entries) or ledger.entries[i] != e),
            len(stored),
        )
        raise LedgerIntegrityError(
            f"refusing to write the pre-registration ledger to {out}: entry {changed} "
            f"({stored[changed].target_metric!r}) is already frozen there and this ledger "
            f"{'drops' if changed >= len(ledger.entries) else 'changes'} it. The ledger is "
            "append-only; register a new metric instead of editing a frozen one."
        )
    try:
        out.mkdir(parents=True, exist_ok=True)
        docs = _ENTRIES.dump_python(ledger.entries, mode="json")
        atomic_write_text(out / _LEDGER_NAME, json.dumps(docs, indent=2, sort_keys=True))
    except OSError as exc:
        raise ImproveStoreError(
            f"failed to write the pre-registration ledger to {out}: {exc}"
        ) from exc

load_improve_records

load_improve_records(path, fmt='csv')

Read improve records back from path; a missing file is an empty store.

Parameters:

Name Type Description Default
path str | Path

The directory dump_improve_records wrote.

required
fmt str

"csv" or "json".

'csv'

Returns:

Type Description
list[ImproveRecord]

The stored records.

Raises:

Type Description
ImproveStoreError

A schema-version mismatch, or any read, parse or validation failure.

Source code in src/calibrated_agent_evaluation/improve/store.py
def load_improve_records(path: str | Path, fmt: str = "csv") -> list[ImproveRecord]:
    """Read improve records back from ``path``; a missing file is an empty store.

    Args:
        path: The directory :func:`dump_improve_records` wrote.
        fmt: ``"csv"`` or ``"json"``.

    Returns:
        The stored records.

    Raises:
        ImproveStoreError: A schema-version mismatch, or any read, parse or validation failure.
    """
    return _STORE.load(path, fmt)

load_prereg_ledger

load_prereg_ledger(path)

Read the ledger written by dump_prereg_ledger; an absent file is an empty ledger.

Returns the entries exactly as stored and does not verify them: the improve gate's require_prereg does, so an edited file is refused there with the entry it broke.

Parameters:

Name Type Description Default
path str | Path

The directory dump_prereg_ledger wrote.

required

Returns:

Type Description
PreRegLedger

The stored ledger.

Raises:

Type Description
ImproveStoreError

Any read, parse or validation failure.

Source code in src/calibrated_agent_evaluation/improve/store.py
def load_prereg_ledger(path: str | Path) -> PreRegLedger:
    """Read the ledger written by :func:`dump_prereg_ledger`; an absent file is an empty ledger.

    Returns the entries exactly as stored and does not verify them: the improve gate's
    :func:`~calibrated_agent_evaluation.improve.require_prereg` does, so an edited file is refused
    there with the entry it broke.

    Args:
        path: The directory :func:`dump_prereg_ledger` wrote.

    Returns:
        The stored ledger.

    Raises:
        ImproveStoreError: Any read, parse or validation failure.
    """
    file = Path(path) / _LEDGER_NAME
    if not file.exists():
        return PreRegLedger()
    try:
        return PreRegLedger(entries=_ENTRIES.validate_json(file.read_bytes()))
    except OSError as exc:
        raise ImproveStoreError(
            f"failed to read the pre-registration ledger {file}: {exc}"
        ) from exc
    except ValidationError as exc:
        kind = "is not valid JSON" if _is_json_error(exc) else "is malformed"
        raise ImproveStoreError(
            f"pre-registration ledger {file} {kind}: {plain_error(exc)}"
        ) from exc
    except ImproveError as exc:
        raise ImproveStoreError(f"pre-registration ledger {file} is malformed: {exc}") from exc