Skip to content

calibrated_agent_evaluation.normalize.datasets

calibrated_agent_evaluation.normalize.datasets

Dataset ingestion: one normalized Trace per session/record.

load_traces(source, *, limit=None, mapping=None, frame="discovery") opens the reader for a dataset source (streaming parquet, JSONL, or an opt-in HuggingFace-hub resolver), turns each record into a vendor-free SessionSpans, and yields normalize(SessionSpans(...)); so any captured-trace dataset flows through the same convention layer + tree builder + schema validation as a single OTLP file.

This is the only package here permitted to import a loader SDK (pyarrow for parquet, huggingface_hub for remote resolve); imported lazily inside the reader modules, so importing calibrated_agent_evaluation.normalize.datasets needs no extra and a missing extra surfaces as a typed NormalizationError naming the installable string, never a bare ImportError. JSONL uses the stdlib json (zero dependency). The public surface returns only normalized-schema Trace objects and vendor-free descriptors.

DatasetSource module-attribute

DatasetSource = ParquetDataset | JSONLDataset | HFDataset

DatasetMapping dataclass

DatasetMapping(
    spans_column="spans",
    session_id_column="session_id",
    taxonomy_columns=("harness", "benchmark"),
)

Which columns a dataset keeps its session id, span list, and taxonomy dims in.

Defaults match the Exgentic/agent-llm-traces shape. A differently-shaped dataset supplies its own column names so the loader reads it without a code change. taxonomy_columns are copied into each Trace's taxonomy_cell as queryable dims (a benchmark's own success/score may be listed here, but are NEVER turned into a gold Label).

spans_column class-attribute instance-attribute

spans_column = 'spans'

session_id_column class-attribute instance-attribute

session_id_column = 'session_id'

taxonomy_columns class-attribute instance-attribute

taxonomy_columns = ('harness', 'benchmark')

HFDataset dataclass

HFDataset(
    repo_id,
    split="train",
    semconv="auto",
    mapping=DatasetMapping(),
)

A HuggingFace-hub dataset resolved by repo id. Needs the [hf] extra.

Resolves the repo's parquet shards to local files, then normalizes through the same path as ParquetDataset. split selects the config split (default "train").

repo_id instance-attribute

repo_id

split class-attribute instance-attribute

split = 'train'

semconv class-attribute instance-attribute

semconv = 'auto'

mapping class-attribute instance-attribute

mapping = field(default_factory=DatasetMapping)

JSONLDataset dataclass

JSONLDataset(
    path, semconv="auto", mapping=DatasetMapping()
)

A local JSON-lines dataset: one Trace per line. Read with the stdlib json.

path instance-attribute

path

semconv class-attribute instance-attribute

semconv = 'auto'

mapping class-attribute instance-attribute

mapping = field(default_factory=DatasetMapping)

ParquetDataset dataclass

ParquetDataset(
    path, semconv="auto", mapping=DatasetMapping()
)

A local parquet dataset: one Trace per row. Needs the [datasets] extra.

path instance-attribute

path

semconv class-attribute instance-attribute

semconv = 'auto'

mapping class-attribute instance-attribute

mapping = field(default_factory=DatasetMapping)

SessionSpans dataclass

SessionSpans(
    spans,
    semconv="auto",
    trace_id=None,
    frame="production-representative",
    taxonomy_cell=dict(),
)

One session's raw OTLP-JSON spans, in memory, to normalize via the convention layer.

spans is the list of span dicts (canonical OTLP-JSON shape) for a single session/record. semconv="auto" runs detection; a named Convention forces one adapter. The header fields resolve in the layered order (convention attribute → this override → wire/default). This is what a dataset row becomes before the convention adapter turns it into a Trace.

Attributes:

Name Type Description
spans list[dict[str, Any]]

The session's spans as OTLP-JSON dicts.

semconv Convention

The convention to read with, or "auto".

trace_id str | None

The trace id, when the spans don't carry one.

frame Frame

The sampling frame.

taxonomy_cell dict[str, str]

The taxonomy cell, when the spans don't carry one.

spans instance-attribute

spans

semconv class-attribute instance-attribute

semconv = 'auto'

trace_id class-attribute instance-attribute

trace_id = None

frame class-attribute instance-attribute

frame = 'production-representative'

taxonomy_cell class-attribute instance-attribute

taxonomy_cell = field(default_factory=dict)

load_traces

load_traces(
    source, *, limit=None, mapping=None, frame="discovery"
)

Yield one normalized Trace per record of a captured-trace dataset.

Streams records from the source (parquet / JSONL / HuggingFace-hub resolve), turns each into a SessionSpans (via mapping; the source's own mapping is the default), and yields normalize(SessionSpans(...)). Honors limit by stopping after N records (bounded working set). Raises a typed NormalizationError on a missing extra, an unreadable source / bad mapping, or an unrecognized source type; never a partial or silently-dropped result.

frame defaults to "discovery", deliberately NOT to SessionSpans.frame's own default. frame is an analyst's claim about where data came from, and "production-representative" is the one value that authorizes a headline rate (sampling.assert_production_frame); so inheriting it by omission meant a public benchmark slice arrived pre-authorized to make a production claim nobody had made. A caller who has not said where the data came from gets the frame that unlocks nothing.

Source code in src/calibrated_agent_evaluation/normalize/datasets/__init__.py
def load_traces(
    source: DatasetSource,
    *,
    limit: int | None = None,
    mapping: DatasetMapping | None = None,
    frame: Frame = "discovery",
) -> Iterator[Trace]:
    """Yield one normalized ``Trace`` per record of a captured-trace dataset.

    Streams records from the source (parquet / JSONL / HuggingFace-hub resolve), turns each into
    a :class:`SessionSpans` (via ``mapping``; the source's own mapping is the default), and
    yields ``normalize(SessionSpans(...))``. Honors ``limit`` by stopping after N records
    (bounded working set). Raises a typed :class:`NormalizationError` on a
    missing extra, an unreadable source / bad mapping, or an unrecognized
    source type; never a partial or silently-dropped result.

    ``frame`` defaults to ``"discovery"``, deliberately NOT to ``SessionSpans.frame``'s own default.
    ``frame`` is an analyst's claim about where data came from, and
    ``"production-representative"`` is the one value that authorizes a headline rate
    (``sampling.assert_production_frame``); so inheriting it by omission meant a public benchmark
    slice arrived pre-authorized to make a production claim nobody had made. A caller who has not
    said where the data came from gets the frame that unlocks nothing.
    """
    if limit is not None and limit < 0:
        raise NormalizationError(f"limit must be non-negative, got {limit}")
    effective_mapping = mapping or getattr(source, "mapping", DatasetMapping())
    semconv = getattr(source, "semconv", "auto")
    for count, record in enumerate(_iter_records(source)):
        if limit is not None and count >= limit:
            break
        yield normalize(_record_to_session(record, effective_mapping, semconv, frame))

iter_sessions

iter_sessions(source, *, mapping=None, frame='discovery')

Yield each record of a dataset as its SessionSpans, or the error mapping it.

For a caller that needs each record before normalization, as calibrated-agent-evaluation ingest does to measure the source side of its no-silent-loss gates, and that reports a record which cannot be mapped on its own line instead of losing the rest of the dataset to it: a record the mapping cannot read is yielded as its NormalizationError, not raised. A failure of the reader itself (an unreadable file, a malformed JSONL line) is raised, because the stream cannot continue. Same reader, mapping and frame default as load_traces.

Source code in src/calibrated_agent_evaluation/normalize/datasets/__init__.py
def iter_sessions(
    source: DatasetSource,
    *,
    mapping: DatasetMapping | None = None,
    frame: Frame = "discovery",
) -> Iterator[SessionSpans | NormalizationError]:
    """Yield each record of a dataset as its :class:`SessionSpans`, or the error mapping it.

    For a caller that needs each record before normalization, as
    ``calibrated-agent-evaluation ingest`` does to measure the source side of its no-silent-loss
    gates, and that reports a record which cannot be mapped on its own line instead of losing the
    rest of the dataset to it: a record the mapping cannot read is yielded as its
    :class:`NormalizationError`, not raised. A failure of the reader itself (an unreadable file, a
    malformed JSONL line) is raised, because the stream cannot continue. Same reader, mapping and
    ``frame`` default as :func:`load_traces`.
    """
    effective_mapping = mapping or getattr(source, "mapping", DatasetMapping())
    semconv = getattr(source, "semconv", "auto")
    for record in _iter_records(source):
        try:
            yield _record_to_session(record, effective_mapping, semconv, frame)
        except NormalizationError as exc:
            yield exc