calibrated_agent_evaluation.normalize.datasets¶
calibrated_agent_evaluation.normalize.datasets ¶
Dataset ingestion: one normalized Trace per session/record.
load_traces(source, *, limit=None, mapping=None, frame="discovery") opens the reader for a
dataset source
(streaming parquet, JSONL, or an opt-in HuggingFace-hub resolver),
turns each record into a vendor-free
SessionSpans, and yields
normalize(SessionSpans(...)); so any captured-trace dataset flows through the same convention
layer + tree builder + schema validation as a single OTLP file.
This is the only package here permitted to import a loader SDK (pyarrow for parquet,
huggingface_hub for remote resolve); imported lazily inside the reader modules, so importing
calibrated_agent_evaluation.normalize.datasets needs no extra and a missing extra surfaces as a
typed NormalizationError naming the
installable string, never a bare ImportError. JSONL uses the stdlib json (zero dependency).
The public surface returns only normalized-schema Trace objects and vendor-free descriptors.
DatasetMapping
dataclass
¶
DatasetMapping(
spans_column="spans",
session_id_column="session_id",
taxonomy_columns=("harness", "benchmark"),
)
Which columns a dataset keeps its session id, span list, and taxonomy dims in.
Defaults match the Exgentic/agent-llm-traces shape. A differently-shaped dataset supplies
its own column names so the loader reads it without a code change. taxonomy_columns
are copied into each Trace's taxonomy_cell as queryable dims (a benchmark's own
success/score may be listed here, but are NEVER turned into a gold Label).
HFDataset
dataclass
¶
A HuggingFace-hub dataset resolved by repo id. Needs the [hf] extra.
Resolves the repo's parquet shards to local files, then normalizes through the same path as
ParquetDataset. split selects the config split (default "train").
JSONLDataset
dataclass
¶
ParquetDataset
dataclass
¶
SessionSpans
dataclass
¶
SessionSpans(
spans,
semconv="auto",
trace_id=None,
frame="production-representative",
taxonomy_cell=dict(),
)
One session's raw OTLP-JSON spans, in memory, to normalize via the convention layer.
spans is the list of span dicts (canonical OTLP-JSON shape) for a single session/record.
semconv="auto" runs detection; a named Convention forces one adapter.
The header fields resolve in the layered order (convention attribute → this override →
wire/default).
This is what a dataset row becomes before the convention adapter turns it into a Trace.
Attributes:
| Name | Type | Description |
|---|---|---|
spans |
list[dict[str, Any]]
|
The session's spans as OTLP-JSON dicts. |
semconv |
Convention
|
The convention to read with, or |
trace_id |
str | None
|
The trace id, when the spans don't carry one. |
frame |
Frame
|
The sampling frame. |
taxonomy_cell |
dict[str, str]
|
The taxonomy cell, when the spans don't carry one. |
load_traces ¶
Yield one normalized Trace per record of a captured-trace dataset.
Streams records from the source (parquet / JSONL / HuggingFace-hub resolve), turns each into
a SessionSpans (via mapping; the source's own mapping is the default), and
yields normalize(SessionSpans(...)). Honors limit by stopping after N records
(bounded working set). Raises a typed NormalizationError on a
missing extra, an unreadable source / bad mapping, or an unrecognized
source type; never a partial or silently-dropped result.
frame defaults to "discovery", deliberately NOT to SessionSpans.frame's own default.
frame is an analyst's claim about where data came from, and
"production-representative" is the one value that authorizes a headline rate
(sampling.assert_production_frame); so inheriting it by omission meant a public benchmark
slice arrived pre-authorized to make a production claim nobody had made. A caller who has not
said where the data came from gets the frame that unlocks nothing.
Source code in src/calibrated_agent_evaluation/normalize/datasets/__init__.py
iter_sessions ¶
Yield each record of a dataset as its SessionSpans, or the error mapping it.
For a caller that needs each record before normalization, as
calibrated-agent-evaluation ingest does to measure the source side of its no-silent-loss
gates, and that reports a record which cannot be mapped on its own line instead of losing the
rest of the dataset to it: a record the mapping cannot read is yielded as its
NormalizationError, not raised. A failure of the reader itself (an unreadable file, a
malformed JSONL line) is raised, because the stream cannot continue. Same reader, mapping and
frame default as load_traces.