Extensible dump/load registry for Calibrated Agent Evaluation artifacts.
This layer owns the round-trip mechanism and the core Trace / Span / Label
codecs; other packages call register(artifact_type, to_row, from_row) to make their own
artifact types take part in dump / load. Every artifact round-trips losslessly through
CSV and JSON, so nothing is locked into one store.
The results table (Estimate, results_table, write_results_csv) is served lazily: it
imports the record types of calibrate/run/improve/bias/flagship, whose stats
chain pulls scipy/pyarrow, and calibrated-agent-evaluation dump/load must stay
importable on a bare offline install.
MANIFEST_NAME
module-attribute
MANIFEST_NAME = 'dump-manifest.json'
RESULTS_COLUMNS
module-attribute
RESULTS_COLUMNS = (
"estimand",
"slice",
"point",
"lo",
"hi",
"level",
"sided",
"method",
"n",
"n_resamples",
"seed",
"refusal_reason",
"source",
"run_at",
"calibrated_agent_evaluation_version",
"git_sha",
"python_version",
"dep_versions",
"thresholds",
)
CorruptDumpError
Bases: CalibratedAgentEvaluationDataError
The load path is not a loadable dump.
A file is not valid JSON, the path is missing, is not a directory, or holds no dump file, or
the directory does not match its manifest (the subclasses below name which way).
Raised instead of letting a raw json.JSONDecodeError escape, or a wrong path load nothing:
a corrupt dump is a
data-plane failure and must surface inside the CalibratedAgentEvaluationDataError
family so callers can catch the whole family at the plane boundary ("no silent failure").
DumpFileEntry
pydantic-model
Bases: CalibratedAgentEvaluationModel
One file of a dump: how many records it holds and its sha256.
Config:
frozen: True
extra: forbid
Fields:
DumpManifest
pydantic-model
Bases: CalibratedAgentEvaluationModel
The manifest that seals a dump, keyed by each file's POSIX path relative to the dump.
Config:
frozen: True
extra: forbid
Fields:
format_version = DUMP_FORMAT_VERSION
DumpMismatchError
DumpMismatchError(message, *, path)
Bases: CorruptDumpError
A file the manifest lists is missing, or its contents differ from what the manifest records.
A dump overwritten in part by another, or a file edited after the dump, reads this way.
path is the file, relative to the dump directory.
Source code in src/calibrated_agent_evaluation/dumpload/io.py
| def __init__(self, message: str, *, path: str) -> None:
super().__init__(message)
self.path = path
|
IncompleteDumpError
Bases: CorruptDumpError
The directory holds dump files but no manifest.
dump writes the manifest after every data file, so a dump that was interrupted, or is still
being written, reads this way. So does a dump written before dumps carried a manifest.
UnlistedDumpFileError
UnlistedDumpFileError(message, *, paths)
Bases: CorruptDumpError
The directory holds files the manifest does not list; paths names them.
Source code in src/calibrated_agent_evaluation/dumpload/io.py
| def __init__(self, message: str, *, paths: tuple[str, ...]) -> None:
super().__init__(message)
self.paths = paths
|
Estimate
dataclass
Estimate(
estimand,
slice,
point,
lo,
hi,
level,
sided,
method,
n,
n_resamples,
seed,
refusal_reason,
provenance,
source,
)
One reported estimate with everything needed to read it correctly.
estimand names the quantity ("tpr", "corrected_rate", "bias_magnitude", ...);
slice names what it is about (a judge, a failure mode, a bias, a stage). point is
None exactly when the estimate was refused, and then refusal_reason is set. lo /
hi are the interval's edges; a one-sided bound puts the open side at the support's edge
(hi=1.0 for a lower bound, lo=0.0 for an upper one) and says so in sided.
seed / n_resamples are set only for a bootstrapped estimate. source is the record
kind and id the row came from.
estimand
instance-attribute
method
instance-attribute
n_resamples
instance-attribute
refusal_reason
instance-attribute
provenance
instance-attribute
source
instance-attribute
dump
dump(backend, path, fmt='csv', *, extra=None)
Write every registered artifact type to <name>.csv and <name>.json under path.
Both formats are always written (the JSON is the lossless source on load; the CSV is the
human-inspectable mirror). fmt selects which one load reads back. extra, when
given, is called with the dump directory after the artifact files, to write more files into
the dump (the CLI writes the stage records); the manifest seals those too.
Every file is written atomically, and the manifest last, so a dump that does not finish has
no manifest and load refuses it. An earlier dump in path is removed first: its
manifest and the files it lists, and nothing else.
Returns:
| Type |
Description |
DumpManifest
|
The manifest written to dump-manifest.json.
|
Source code in src/calibrated_agent_evaluation/dumpload/io.py
| def dump(
backend: _RoundTripBackend,
path: str | Path,
fmt: str = "csv",
*,
extra: Callable[[Path], object] | None = None,
) -> DumpManifest:
"""Write every registered artifact type to ``<name>.csv`` and ``<name>.json`` under ``path``.
Both formats are always written (the JSON is the lossless source on load; the CSV is the
human-inspectable mirror). ``fmt`` selects which one ``load`` reads back. ``extra``, when
given, is called with the dump directory after the artifact files, to write more files into
the dump (the CLI writes the stage records); the manifest seals those too.
Every file is written atomically, and the manifest last, so a dump that does not finish has
no manifest and ``load`` refuses it. An earlier dump in ``path`` is removed first: its
manifest and the files it lists, and nothing else.
Returns:
The manifest written to ``dump-manifest.json``.
"""
if fmt not in {"csv", "json"}:
raise ValueError(f"fmt must be 'csv' or 'json', got {fmt!r}")
register_core_codecs()
out = Path(path)
out.mkdir(parents=True, exist_ok=True)
_remove_previous_dump(out)
# Each writer replaces its file by rename, so a file this dump wrote has a new stamp. A data
# file it did not write stays out of the manifest, and load refuses the directory over it.
untouched = {rel: _stamp(out / rel) for rel in _dump_files(out)}
for artifact_type in registered_types():
codec = codec_for(artifact_type) # raises UnregisteredArtifactError if missing
rows = [codec.to_row(a) for a in _collect(backend, artifact_type)]
_write_json(out / f"{codec.name}.json", rows)
_write_csv(out / f"{codec.name}.csv", rows, codec)
if extra is not None:
extra(out)
written = [rel for rel in _dump_files(out) if untouched.get(rel) != _stamp(out / rel)]
manifest = DumpManifest(files={rel: _entry(rel, (out / rel).read_bytes()) for rel in written})
_write_manifest(out, manifest)
return manifest
|
load
load(backend, path, fmt='csv')
Read a dumped workspace back and re-persist it into backend.
The directory is first checked against its manifest (verify_dump), then every file is
parsed, version-checked, membership-checked and validated before any put, so a refusal
for any of those reasons writes nothing. Traces (with their spans) and labels are re-put;
the separately-dumped spans are verified to belong to a loaded trace but not re-put
independently (they ride in via their trace, avoiding orphans). A label whose trace is not in
the dump is loaded as it was dumped, since a store accepts labels for traces it does not hold.
Source code in src/calibrated_agent_evaluation/dumpload/io.py
| def load(backend: _RoundTripBackend, path: str | Path, fmt: str = "csv") -> None:
"""Read a dumped workspace back and re-persist it into ``backend``.
The directory is first checked against its manifest (:func:`verify_dump`), then every file is
parsed, version-checked, membership-checked and validated *before* any ``put``, so a refusal
for any of those reasons writes nothing. Traces (with their spans) and labels are re-``put``;
the separately-dumped spans are verified to belong to a loaded trace but not re-``put``
independently (they ride in via their trace, avoiding orphans). A label whose trace is not in
the dump is loaded as it was dumped, since a store accepts labels for traces it does not hold.
"""
register_core_codecs()
src = Path(path)
if fmt not in {"csv", "json"}:
raise CorruptDumpError(f"fmt must be 'csv' or 'json', got {fmt!r}")
manifest = verify_dump(src)
parsed: dict[type, list[Any]] = {}
for artifact_type in registered_types():
codec = codec_for(artifact_type)
rows = _read_rows(src, codec.name, fmt)
_check_count(src, f"{codec.name}.{fmt}", len(rows), manifest)
_check_versions(rows, artifact_type) # before any put
parsed[artifact_type] = [codec.from_row(row) for row in rows]
loaded_span_ids = {(s.trace_id, s.span_id) for t in parsed.get(Trace, []) for s in t.spans}
for span in parsed.get(Span, []):
if (span.trace_id, span.span_id) not in loaded_span_ids:
raise UnregisteredArtifactError(
f"dumped span {span.span_id!r} has no parent trace in the dump (orphan span)"
)
# Validated up front so a bad artifact late in the dump cannot leave the store half-loaded:
# each put validates too, but by then the artifacts before it are already written.
for artifact in (*parsed.get(Trace, []), *parsed.get(Label, [])):
validate(artifact)
# Persist traces first (spans ride in), then labels.
for trace in parsed.get(Trace, []):
backend.put_trace(trace)
for label in parsed.get(Label, []):
backend.put_score(label)
|
register_core_codecs
Register the built-in Trace / Span / Label codecs (idempotent).
Source code in src/calibrated_agent_evaluation/dumpload/io.py
| def register_core_codecs() -> None:
"""Register the built-in ``Trace`` / ``Span`` / ``Label`` codecs (idempotent)."""
if not is_registered(Trace):
register(Trace, _trace_to_row, _trace_from_row, name="trace")
if not is_registered(Span):
register(Span, _span_to_row, _span_from_row, name="span")
if not is_registered(Label):
register(Label, _label_to_row, _label_from_row, name="label")
|
verify_dump
Check that path is one whole dump, as dump sealed it, and return its manifest.
Raises:
| Type |
Description |
CorruptDumpError
|
path is missing, is not a directory, holds no dump, or has a manifest
that is not valid or is of another dump format version.
|
IncompleteDumpError
|
path holds dump files but no manifest.
|
DumpMismatchError
|
A file the manifest lists is missing or differs from its entry.
|
UnlistedDumpFileError
|
path holds files the manifest does not list.
|
Source code in src/calibrated_agent_evaluation/dumpload/io.py
| def verify_dump(path: str | Path) -> DumpManifest:
"""Check that ``path`` is one whole dump, as ``dump`` sealed it, and return its manifest.
Raises:
CorruptDumpError: ``path`` is missing, is not a directory, holds no dump, or has a manifest
that is not valid or is of another dump format version.
IncompleteDumpError: ``path`` holds dump files but no manifest.
DumpMismatchError: A file the manifest lists is missing or differs from its entry.
UnlistedDumpFileError: ``path`` holds files the manifest does not list.
"""
src = Path(path)
if not src.exists():
raise CorruptDumpError(f"dump directory {src} does not exist")
if not src.is_dir():
raise CorruptDumpError(f"dump path {src} is not a directory")
manifest = _read_manifest(src)
present = _dump_files(src)
if manifest is None:
if not present:
raise CorruptDumpError(
f"{src} holds no dump file; it is not a calibrated-agent-evaluation dump directory"
)
raise IncompleteDumpError(
f"{src} holds dump files but no {MANIFEST_NAME}, so nothing was loaded. dump writes "
"the manifest after every other file, so this dump was interrupted or is still being "
"written (or was written before dumps carried a manifest); dump the source store "
"again."
)
if manifest.format_version != DUMP_FORMAT_VERSION:
raise CorruptDumpError(
f"{src / MANIFEST_NAME} is dump format {manifest.format_version!r}; this version reads "
f"format {DUMP_FORMAT_VERSION!r}. Load it with the release that wrote it."
)
for rel, expected in sorted(manifest.files.items()):
_check_entry(src, rel, expected)
unlisted = tuple(rel for rel in present if rel not in manifest.files)
if unlisted:
more = f" and {len(unlisted) - 5} more" if len(unlisted) > 5 else ""
shown = ", ".join(unlisted[:5]) + more
raise UnlistedDumpFileError(
f"{src} holds {len(unlisted)} file(s) its {MANIFEST_NAME} does not list ({shown}), so "
"nothing was loaded. They were not written by the dump the manifest seals; remove "
"them, or dump the source store into an empty directory.",
paths=unlisted,
)
return manifest
|
register
register(artifact_type, to_row, from_row, *, name=None)
Register a codec for artifact_type.
name is the stable identifier used for the dump filename (<name>.csv /
<name>.json); it defaults to the type's lowercased class name. Re-registering an
already-registered artifact_type raises ValueError (not
silent last-wins). Registering a distinct type under an already-used name is likewise
rejected: two codecs sharing a filename would clobber each other's <name>.json/.csv on
dump (one silently overwrites the other), breaking the lossless round-trip.
Source code in src/calibrated_agent_evaluation/dumpload/registry.py
| def register(
artifact_type: type,
to_row: ToRow,
from_row: FromRow,
*,
name: str | None = None,
) -> None:
"""Register a codec for ``artifact_type``.
``name`` is the stable identifier used for the dump filename (``<name>.csv`` /
``<name>.json``); it defaults to the type's lowercased class name. Re-registering an
already-registered ``artifact_type`` raises :class:`ValueError` (not
silent last-wins). Registering a *distinct* type under an already-used ``name`` is likewise
rejected: two codecs sharing a filename would clobber each other's ``<name>.json``/``.csv`` on
dump (one silently overwrites the other), breaking the lossless round-trip.
"""
if artifact_type in _REGISTRY:
raise ValueError(
f"artifact type {artifact_type.__name__!r} already has a registered codec; "
"re-registration is rejected (use a distinct type)"
)
resolved_name = name or artifact_type.__name__.lower()
clashing = next((c for c in _REGISTRY.values() if c.name == resolved_name), None)
if clashing is not None:
raise ValueError(
f"codec name {resolved_name!r} is already registered for another type; "
"two codecs cannot share a dump filename (they would clobber each other); "
"use a distinct name"
)
_REGISTRY[artifact_type] = Codec(resolved_name, to_row, from_row)
|
results_table
Flatten persisted records into one Estimate per reported interval or bound.
Raises ResultsTableError on a record type it does not know, rather than silently
leaving its estimates out.
Source code in src/calibrated_agent_evaluation/dumpload/results.py
| def results_table(records: Iterable[Record]) -> list[Estimate]:
"""Flatten persisted records into one :class:`Estimate` per reported interval or bound.
Raises :class:`ResultsTableError` on a record type it does not know, rather than silently
leaving its estimates out.
"""
rows: list[Estimate] = []
for record in records:
if isinstance(record, CalibrationRecord):
rows.extend(_calibration_rows(record))
elif isinstance(record, RunRecord):
rows.extend(_run_rows(record))
elif isinstance(record, ImproveRecord):
rows.extend(_improve_rows(record))
elif isinstance(record, BiasBatteryRecord):
rows.extend(_bias_rows(record))
elif isinstance(record, FlagshipReport):
rows.extend(_flagship_rows(record))
elif isinstance(record, PreferenceRecord):
# One preference is an observation, not an estimate; ratings come from all of them.
continue
elif isinstance(record, JudgeRunRecord):
# A judge run records which verdicts were made and by which judge; it reports no
# estimate, so it adds no row.
continue
elif isinstance(record, AgreementRecord):
rows.extend(_agreement_rows(record))
elif isinstance(record, CodebookRecord | ProposedModeRecord):
continue # a vocabulary or a proposed mode, with no estimate to report
elif isinstance(record, JudgeCheckRecord):
rows.extend(_judge_check_rows(record))
elif isinstance(record, SampleRecord):
rows.extend(_sample_rows(record))
elif isinstance(record, ProbeBatteryRecord):
rows.extend(_probe_rows(record))
elif isinstance(record, AbTestClaimRecord | RecommendationClaimRecord):
# AgentCore's A/B and recommendation figures are a vendor claim, never a Calibrated
# Agent Evaluation estimate, so they add no row.
continue
elif isinstance(record, ArmAssignment | RunProvenance | AuditRecord):
# Bookkeeping: an arm, a replay's bundle, or what a re-test rested on. The re-test's
# estimate is the ImproveRecord it names, which adds its own rows.
continue
else:
raise ResultsTableError(
f"results_table: no estimate extractor for {type(record).__name__!r}"
)
return rows
|
write_results_csv
write_results_csv(rows, path)
Write the results table to path and return the number of rows written.
Accepts either Estimate rows or records (flattened with results_table).
Raises ResultsTableError on a filesystem failure.
Source code in src/calibrated_agent_evaluation/dumpload/results.py
| def write_results_csv(rows: Sequence[Estimate] | Iterable[Record], path: str | Path) -> int:
"""Write the results table to ``path`` and return the number of rows written.
Accepts either :class:`Estimate` rows or records (flattened with :func:`results_table`).
Raises :class:`ResultsTableError` on a filesystem failure.
"""
items = list(rows)
n_estimates = sum(isinstance(r, Estimate) for r in items)
if n_estimates == len(items):
estimates = [r for r in items if isinstance(r, Estimate)]
elif n_estimates == 0:
estimates = results_table(r for r in items if not isinstance(r, Estimate))
else:
raise ResultsTableError("write_results_csv: pass Estimate rows or records, not a mix")
out = Path(path)
try:
out.parent.mkdir(parents=True, exist_ok=True)
buffer = io.StringIO(newline="")
writer = csv.DictWriter(buffer, fieldnames=list(RESULTS_COLUMNS))
writer.writeheader()
writer.writerows(_row(e) for e in estimates)
atomic_write_text(out, buffer.getvalue())
except OSError as exc:
raise ResultsTableError(f"failed to write the results table to {out}: {exc}") from exc
return len(estimates)
|