Skip to content

calibrated_agent_evaluation.dumpload

calibrated_agent_evaluation.dumpload

Extensible dump/load registry for Calibrated Agent Evaluation artifacts.

This layer owns the round-trip mechanism and the core Trace / Span / Label codecs; other packages call register(artifact_type, to_row, from_row) to make their own artifact types take part in dump / load. Every artifact round-trips losslessly through CSV and JSON, so nothing is locked into one store.

The results table (Estimate, results_table, write_results_csv) is served lazily: it imports the record types of calibrate/run/improve/bias/flagship, whose stats chain pulls scipy/pyarrow, and calibrated-agent-evaluation dump/load must stay importable on a bare offline install.

MANIFEST_NAME module-attribute

MANIFEST_NAME = 'dump-manifest.json'

RESULTS_COLUMNS module-attribute

RESULTS_COLUMNS = (
    "estimand",
    "slice",
    "point",
    "lo",
    "hi",
    "level",
    "sided",
    "method",
    "n",
    "n_resamples",
    "seed",
    "refusal_reason",
    "source",
    "run_at",
    "calibrated_agent_evaluation_version",
    "git_sha",
    "python_version",
    "dep_versions",
    "thresholds",
)

CorruptDumpError

Bases: CalibratedAgentEvaluationDataError

The load path is not a loadable dump.

A file is not valid JSON, the path is missing, is not a directory, or holds no dump file, or the directory does not match its manifest (the subclasses below name which way).

Raised instead of letting a raw json.JSONDecodeError escape, or a wrong path load nothing: a corrupt dump is a data-plane failure and must surface inside the CalibratedAgentEvaluationDataError family so callers can catch the whole family at the plane boundary ("no silent failure").

DumpFileEntry pydantic-model

Bases: CalibratedAgentEvaluationModel

One file of a dump: how many records it holds and its sha256.

Config:

  • frozen: True
  • extra: forbid

Fields:

records pydantic-field

records

sha256 pydantic-field

sha256

DumpManifest pydantic-model

Bases: CalibratedAgentEvaluationModel

The manifest that seals a dump, keyed by each file's POSIX path relative to the dump.

Config:

  • frozen: True
  • extra: forbid

Fields:

format_version pydantic-field

format_version = DUMP_FORMAT_VERSION

files pydantic-field

files

DumpMismatchError

DumpMismatchError(message, *, path)

Bases: CorruptDumpError

A file the manifest lists is missing, or its contents differ from what the manifest records.

A dump overwritten in part by another, or a file edited after the dump, reads this way. path is the file, relative to the dump directory.

Source code in src/calibrated_agent_evaluation/dumpload/io.py
def __init__(self, message: str, *, path: str) -> None:
    super().__init__(message)
    self.path = path

path instance-attribute

path = path

IncompleteDumpError

Bases: CorruptDumpError

The directory holds dump files but no manifest.

dump writes the manifest after every data file, so a dump that was interrupted, or is still being written, reads this way. So does a dump written before dumps carried a manifest.

UnlistedDumpFileError

UnlistedDumpFileError(message, *, paths)

Bases: CorruptDumpError

The directory holds files the manifest does not list; paths names them.

Source code in src/calibrated_agent_evaluation/dumpload/io.py
def __init__(self, message: str, *, paths: tuple[str, ...]) -> None:
    super().__init__(message)
    self.paths = paths

paths instance-attribute

paths = paths

Estimate dataclass

Estimate(
    estimand,
    slice,
    point,
    lo,
    hi,
    level,
    sided,
    method,
    n,
    n_resamples,
    seed,
    refusal_reason,
    provenance,
    source,
)

One reported estimate with everything needed to read it correctly.

estimand names the quantity ("tpr", "corrected_rate", "bias_magnitude", ...); slice names what it is about (a judge, a failure mode, a bias, a stage). point is None exactly when the estimate was refused, and then refusal_reason is set. lo / hi are the interval's edges; a one-sided bound puts the open side at the support's edge (hi=1.0 for a lower bound, lo=0.0 for an upper one) and says so in sided. seed / n_resamples are set only for a bootstrapped estimate. source is the record kind and id the row came from.

estimand instance-attribute

estimand

slice instance-attribute

slice

point instance-attribute

point

lo instance-attribute

lo

hi instance-attribute

hi

level instance-attribute

level

sided instance-attribute

sided

method instance-attribute

method

n instance-attribute

n

n_resamples instance-attribute

n_resamples

seed instance-attribute

seed

refusal_reason instance-attribute

refusal_reason

provenance instance-attribute

provenance

source instance-attribute

source

ResultsTableError

Bases: CalibratedAgentEvaluationDataError

A results table could not be built (unknown record type) or written (filesystem failure).

dump

dump(backend, path, fmt='csv', *, extra=None)

Write every registered artifact type to <name>.csv and <name>.json under path.

Both formats are always written (the JSON is the lossless source on load; the CSV is the human-inspectable mirror). fmt selects which one load reads back. extra, when given, is called with the dump directory after the artifact files, to write more files into the dump (the CLI writes the stage records); the manifest seals those too.

Every file is written atomically, and the manifest last, so a dump that does not finish has no manifest and load refuses it. An earlier dump in path is removed first: its manifest and the files it lists, and nothing else.

Returns:

Type Description
DumpManifest

The manifest written to dump-manifest.json.

Source code in src/calibrated_agent_evaluation/dumpload/io.py
def dump(
    backend: _RoundTripBackend,
    path: str | Path,
    fmt: str = "csv",
    *,
    extra: Callable[[Path], object] | None = None,
) -> DumpManifest:
    """Write every registered artifact type to ``<name>.csv`` and ``<name>.json`` under ``path``.

    Both formats are always written (the JSON is the lossless source on load; the CSV is the
    human-inspectable mirror). ``fmt`` selects which one ``load`` reads back. ``extra``, when
    given, is called with the dump directory after the artifact files, to write more files into
    the dump (the CLI writes the stage records); the manifest seals those too.

    Every file is written atomically, and the manifest last, so a dump that does not finish has
    no manifest and ``load`` refuses it. An earlier dump in ``path`` is removed first: its
    manifest and the files it lists, and nothing else.

    Returns:
        The manifest written to ``dump-manifest.json``.
    """
    if fmt not in {"csv", "json"}:
        raise ValueError(f"fmt must be 'csv' or 'json', got {fmt!r}")
    register_core_codecs()
    out = Path(path)
    out.mkdir(parents=True, exist_ok=True)
    _remove_previous_dump(out)
    # Each writer replaces its file by rename, so a file this dump wrote has a new stamp. A data
    # file it did not write stays out of the manifest, and load refuses the directory over it.
    untouched = {rel: _stamp(out / rel) for rel in _dump_files(out)}
    for artifact_type in registered_types():
        codec = codec_for(artifact_type)  # raises UnregisteredArtifactError if missing
        rows = [codec.to_row(a) for a in _collect(backend, artifact_type)]
        _write_json(out / f"{codec.name}.json", rows)
        _write_csv(out / f"{codec.name}.csv", rows, codec)
    if extra is not None:
        extra(out)
    written = [rel for rel in _dump_files(out) if untouched.get(rel) != _stamp(out / rel)]
    manifest = DumpManifest(files={rel: _entry(rel, (out / rel).read_bytes()) for rel in written})
    _write_manifest(out, manifest)
    return manifest

load

load(backend, path, fmt='csv')

Read a dumped workspace back and re-persist it into backend.

The directory is first checked against its manifest (verify_dump), then every file is parsed, version-checked, membership-checked and validated before any put, so a refusal for any of those reasons writes nothing. Traces (with their spans) and labels are re-put; the separately-dumped spans are verified to belong to a loaded trace but not re-put independently (they ride in via their trace, avoiding orphans). A label whose trace is not in the dump is loaded as it was dumped, since a store accepts labels for traces it does not hold.

Source code in src/calibrated_agent_evaluation/dumpload/io.py
def load(backend: _RoundTripBackend, path: str | Path, fmt: str = "csv") -> None:
    """Read a dumped workspace back and re-persist it into ``backend``.

    The directory is first checked against its manifest (:func:`verify_dump`), then every file is
    parsed, version-checked, membership-checked and validated *before* any ``put``, so a refusal
    for any of those reasons writes nothing. Traces (with their spans) and labels are re-``put``;
    the separately-dumped spans are verified to belong to a loaded trace but not re-``put``
    independently (they ride in via their trace, avoiding orphans). A label whose trace is not in
    the dump is loaded as it was dumped, since a store accepts labels for traces it does not hold.
    """
    register_core_codecs()
    src = Path(path)
    if fmt not in {"csv", "json"}:
        raise CorruptDumpError(f"fmt must be 'csv' or 'json', got {fmt!r}")
    manifest = verify_dump(src)
    parsed: dict[type, list[Any]] = {}
    for artifact_type in registered_types():
        codec = codec_for(artifact_type)
        rows = _read_rows(src, codec.name, fmt)
        _check_count(src, f"{codec.name}.{fmt}", len(rows), manifest)
        _check_versions(rows, artifact_type)  # before any put
        parsed[artifact_type] = [codec.from_row(row) for row in rows]

    loaded_span_ids = {(s.trace_id, s.span_id) for t in parsed.get(Trace, []) for s in t.spans}
    for span in parsed.get(Span, []):
        if (span.trace_id, span.span_id) not in loaded_span_ids:
            raise UnregisteredArtifactError(
                f"dumped span {span.span_id!r} has no parent trace in the dump (orphan span)"
            )
    # Validated up front so a bad artifact late in the dump cannot leave the store half-loaded:
    # each put validates too, but by then the artifacts before it are already written.
    for artifact in (*parsed.get(Trace, []), *parsed.get(Label, [])):
        validate(artifact)

    # Persist traces first (spans ride in), then labels.
    for trace in parsed.get(Trace, []):
        backend.put_trace(trace)
    for label in parsed.get(Label, []):
        backend.put_score(label)

register_core_codecs

register_core_codecs()

Register the built-in Trace / Span / Label codecs (idempotent).

Source code in src/calibrated_agent_evaluation/dumpload/io.py
def register_core_codecs() -> None:
    """Register the built-in ``Trace`` / ``Span`` / ``Label`` codecs (idempotent)."""
    if not is_registered(Trace):
        register(Trace, _trace_to_row, _trace_from_row, name="trace")
    if not is_registered(Span):
        register(Span, _span_to_row, _span_from_row, name="span")
    if not is_registered(Label):
        register(Label, _label_to_row, _label_from_row, name="label")

verify_dump

verify_dump(path)

Check that path is one whole dump, as dump sealed it, and return its manifest.

Raises:

Type Description
CorruptDumpError

path is missing, is not a directory, holds no dump, or has a manifest that is not valid or is of another dump format version.

IncompleteDumpError

path holds dump files but no manifest.

DumpMismatchError

A file the manifest lists is missing or differs from its entry.

UnlistedDumpFileError

path holds files the manifest does not list.

Source code in src/calibrated_agent_evaluation/dumpload/io.py
def verify_dump(path: str | Path) -> DumpManifest:
    """Check that ``path`` is one whole dump, as ``dump`` sealed it, and return its manifest.

    Raises:
        CorruptDumpError: ``path`` is missing, is not a directory, holds no dump, or has a manifest
            that is not valid or is of another dump format version.
        IncompleteDumpError: ``path`` holds dump files but no manifest.
        DumpMismatchError: A file the manifest lists is missing or differs from its entry.
        UnlistedDumpFileError: ``path`` holds files the manifest does not list.
    """
    src = Path(path)
    if not src.exists():
        raise CorruptDumpError(f"dump directory {src} does not exist")
    if not src.is_dir():
        raise CorruptDumpError(f"dump path {src} is not a directory")
    manifest = _read_manifest(src)
    present = _dump_files(src)
    if manifest is None:
        if not present:
            raise CorruptDumpError(
                f"{src} holds no dump file; it is not a calibrated-agent-evaluation dump directory"
            )
        raise IncompleteDumpError(
            f"{src} holds dump files but no {MANIFEST_NAME}, so nothing was loaded. dump writes "
            "the manifest after every other file, so this dump was interrupted or is still being "
            "written (or was written before dumps carried a manifest); dump the source store "
            "again."
        )
    if manifest.format_version != DUMP_FORMAT_VERSION:
        raise CorruptDumpError(
            f"{src / MANIFEST_NAME} is dump format {manifest.format_version!r}; this version reads "
            f"format {DUMP_FORMAT_VERSION!r}. Load it with the release that wrote it."
        )
    for rel, expected in sorted(manifest.files.items()):
        _check_entry(src, rel, expected)
    unlisted = tuple(rel for rel in present if rel not in manifest.files)
    if unlisted:
        more = f" and {len(unlisted) - 5} more" if len(unlisted) > 5 else ""
        shown = ", ".join(unlisted[:5]) + more
        raise UnlistedDumpFileError(
            f"{src} holds {len(unlisted)} file(s) its {MANIFEST_NAME} does not list ({shown}), so "
            "nothing was loaded. They were not written by the dump the manifest seals; remove "
            "them, or dump the source store into an empty directory.",
            paths=unlisted,
        )
    return manifest

register

register(artifact_type, to_row, from_row, *, name=None)

Register a codec for artifact_type.

name is the stable identifier used for the dump filename (<name>.csv / <name>.json); it defaults to the type's lowercased class name. Re-registering an already-registered artifact_type raises ValueError (not silent last-wins). Registering a distinct type under an already-used name is likewise rejected: two codecs sharing a filename would clobber each other's <name>.json/.csv on dump (one silently overwrites the other), breaking the lossless round-trip.

Source code in src/calibrated_agent_evaluation/dumpload/registry.py
def register(
    artifact_type: type,
    to_row: ToRow,
    from_row: FromRow,
    *,
    name: str | None = None,
) -> None:
    """Register a codec for ``artifact_type``.

    ``name`` is the stable identifier used for the dump filename (``<name>.csv`` /
    ``<name>.json``); it defaults to the type's lowercased class name. Re-registering an
    already-registered ``artifact_type`` raises :class:`ValueError` (not
    silent last-wins). Registering a *distinct* type under an already-used ``name`` is likewise
    rejected: two codecs sharing a filename would clobber each other's ``<name>.json``/``.csv`` on
    dump (one silently overwrites the other), breaking the lossless round-trip.
    """
    if artifact_type in _REGISTRY:
        raise ValueError(
            f"artifact type {artifact_type.__name__!r} already has a registered codec; "
            "re-registration is rejected (use a distinct type)"
        )
    resolved_name = name or artifact_type.__name__.lower()
    clashing = next((c for c in _REGISTRY.values() if c.name == resolved_name), None)
    if clashing is not None:
        raise ValueError(
            f"codec name {resolved_name!r} is already registered for another type; "
            "two codecs cannot share a dump filename (they would clobber each other); "
            "use a distinct name"
        )
    _REGISTRY[artifact_type] = Codec(resolved_name, to_row, from_row)

results_table

results_table(records)

Flatten persisted records into one Estimate per reported interval or bound.

Raises ResultsTableError on a record type it does not know, rather than silently leaving its estimates out.

Source code in src/calibrated_agent_evaluation/dumpload/results.py
def results_table(records: Iterable[Record]) -> list[Estimate]:
    """Flatten persisted records into one :class:`Estimate` per reported interval or bound.

    Raises :class:`ResultsTableError` on a record type it does not know, rather than silently
    leaving its estimates out.
    """
    rows: list[Estimate] = []
    for record in records:
        if isinstance(record, CalibrationRecord):
            rows.extend(_calibration_rows(record))
        elif isinstance(record, RunRecord):
            rows.extend(_run_rows(record))
        elif isinstance(record, ImproveRecord):
            rows.extend(_improve_rows(record))
        elif isinstance(record, BiasBatteryRecord):
            rows.extend(_bias_rows(record))
        elif isinstance(record, FlagshipReport):
            rows.extend(_flagship_rows(record))
        elif isinstance(record, PreferenceRecord):
            # One preference is an observation, not an estimate; ratings come from all of them.
            continue
        elif isinstance(record, JudgeRunRecord):
            # A judge run records which verdicts were made and by which judge; it reports no
            # estimate, so it adds no row.
            continue
        elif isinstance(record, AgreementRecord):
            rows.extend(_agreement_rows(record))
        elif isinstance(record, CodebookRecord | ProposedModeRecord):
            continue  # a vocabulary or a proposed mode, with no estimate to report
        elif isinstance(record, JudgeCheckRecord):
            rows.extend(_judge_check_rows(record))
        elif isinstance(record, SampleRecord):
            rows.extend(_sample_rows(record))
        elif isinstance(record, ProbeBatteryRecord):
            rows.extend(_probe_rows(record))
        elif isinstance(record, AbTestClaimRecord | RecommendationClaimRecord):
            # AgentCore's A/B and recommendation figures are a vendor claim, never a Calibrated
            # Agent Evaluation estimate, so they add no row.
            continue
        elif isinstance(record, ArmAssignment | RunProvenance | AuditRecord):
            # Bookkeeping: an arm, a replay's bundle, or what a re-test rested on. The re-test's
            # estimate is the ImproveRecord it names, which adds its own rows.
            continue
        else:
            raise ResultsTableError(
                f"results_table: no estimate extractor for {type(record).__name__!r}"
            )
    return rows

write_results_csv

write_results_csv(rows, path)

Write the results table to path and return the number of rows written.

Accepts either Estimate rows or records (flattened with results_table). Raises ResultsTableError on a filesystem failure.

Source code in src/calibrated_agent_evaluation/dumpload/results.py
def write_results_csv(rows: Sequence[Estimate] | Iterable[Record], path: str | Path) -> int:
    """Write the results table to ``path`` and return the number of rows written.

    Accepts either :class:`Estimate` rows or records (flattened with :func:`results_table`).
    Raises :class:`ResultsTableError` on a filesystem failure.
    """
    items = list(rows)
    n_estimates = sum(isinstance(r, Estimate) for r in items)
    if n_estimates == len(items):
        estimates = [r for r in items if isinstance(r, Estimate)]
    elif n_estimates == 0:
        estimates = results_table(r for r in items if not isinstance(r, Estimate))
    else:
        raise ResultsTableError("write_results_csv: pass Estimate rows or records, not a mix")
    out = Path(path)
    try:
        out.parent.mkdir(parents=True, exist_ok=True)
        buffer = io.StringIO(newline="")
        writer = csv.DictWriter(buffer, fieldnames=list(RESULTS_COLUMNS))
        writer.writeheader()
        writer.writerows(_row(e) for e in estimates)
        atomic_write_text(out, buffer.getvalue())
    except OSError as exc:
        raise ResultsTableError(f"failed to write the results table to {out}: {exc}") from exc
    return len(estimates)