Skip to content

calibrated_agent_evaluation.normalize

calibrated_agent_evaluation.normalize

The trajectory normalizer: normalize(source) -> Trace.

The single seam that turns a vendor-shaped trace (Inspect .eval, generic OTLP, a Strands session) into the one normalized Trace, so the methodology plane is written once against Trace/Span and never sees a vendor type. It dispatches on the source descriptor, runs the chosen adapter to a vendor-free intermediate, applies the ownership gate (owned vs degraded probed), validates against the schema, and returns the Trace.

This package is the second member of the methodology-plane vendor allowlist (alongside backends/, execute/): the adapters import inspect_ai / opentelemetry.proto; lazily, inside their modules; so importing calibrated_agent_evaluation.normalize needs no opt-in extra, and a missing extra surfaces as a typed NormalizationError naming the installable string, never a bare ImportError.

NormalizerSource module-attribute

NormalizerSource = (
    InspectEvalLog | OTelSpans | StrandsSession
)

NormalizationError

Bases: CalibratedAgentEvaluationDataError

The normalizer could not produce a valid Trace from a source.

Raised (never swallowed and never partially returned) on an unrecognized/unsupported source, framework kind, or OTel semantic convention; a missing opt-in extra (the message names the install command for it); a vendor/parser/ filesystem read failure (chained via from); or a mapped Trace that fails the schema validate().

Recovery dataclass

Recovery(
    ownership,
    reason,
    recovered_spans,
    recovered_tool_calls,
    convention=None,
)

What the adapter recovered, stated before the ownership gate collapsed it.

The gate's output alone cannot answer "did normalization work?", because a probed trace has spans == [] whether the adapter recovered nothing or recovered a full tree whose every node turned out to be empty. Those are a defect and a declared degradation respectively, and the ingest gates have to tell them apart: the loss gate refuses tool-execution spans the adapter failed to link, while the degrade rule degrades an all-contentless tree and explicitly must not refuse it.

So the pre-collapse counts travel beside the trace rather than being re-derived downstream. reason is None for an owned trace and otherwise the message that was logged.

ownership instance-attribute

ownership

reason instance-attribute

reason

recovered_spans instance-attribute

recovered_spans

recovered_tool_calls instance-attribute

recovered_tool_calls

convention class-attribute instance-attribute

convention = None

InspectEvalLog dataclass

InspectEvalLog(
    path,
    trace_id=None,
    frame="production-representative",
    taxonomy_cell=dict(),
    sample_id=None,
)

An Inspect .eval (or .json) log to normalize.

trace_id/taxonomy_cell are read from the sample metadata when present; frame is analyst-assigned and defaults to "production-representative" (override per capture).

An Inspect log holds one sample per dataset row, so a real eval log is normally multi-sample. Use normalize_all for one Trace per sample; on a multi-sample log refuses unless sample_id names which one to read, rather than silently evaluating the log at 1/N of its data.

Attributes:

Name Type Description
path Path

The .eval or .json log file.

trace_id str | None

The trace id; read from the sample metadata when omitted.

frame Frame

The sampling frame.

taxonomy_cell dict[str, str]

The taxonomy cell; read from the sample metadata when omitted.

sample_id str | None

The sample to read from a multi-sample log.

path instance-attribute

path

trace_id class-attribute instance-attribute

trace_id = None

frame class-attribute instance-attribute

frame = 'production-representative'

taxonomy_cell class-attribute instance-attribute

taxonomy_cell = field(default_factory=dict)

sample_id class-attribute instance-attribute

sample_id = None

OTelSpans dataclass

OTelSpans(
    path,
    semconv="auto",
    content_location="events",
    trace_id=None,
    frame="production-representative",
    taxonomy_cell=dict(),
)

A generic OpenTelemetry source.

path is a frozen OTLP-JSON file (read with stdlib json) or a binary .otlp protobuf (needs the [otel] extra). semconv selects the convention branch. content_location is a best-effort preference hint, not a hard gate: the reader tries the named placement first, then falls back to the other, because real traces mix span attributes and span events; falling back is what prevents the silent-drop of event-located content. The header fields resolve in the layered order (convention attribute → this override → OTel-traceId hex / defaults).

Attributes:

Name Type Description
path Path

The OTLP-JSON or binary .otlp file.

semconv Semconv | Convention

The OTel GenAI convention to read with, or "auto".

content_location ContentLocation

Where to look for message content first.

trace_id str | None

The trace id, when the spans don't carry one.

frame Frame

The sampling frame.

taxonomy_cell dict[str, str]

The taxonomy cell, when the spans don't carry one.

path instance-attribute

path

semconv class-attribute instance-attribute

semconv = 'auto'

content_location class-attribute instance-attribute

content_location = 'events'

trace_id class-attribute instance-attribute

trace_id = None

frame class-attribute instance-attribute

frame = 'production-representative'

taxonomy_cell class-attribute instance-attribute

taxonomy_cell = field(default_factory=dict)

SessionSpans dataclass

SessionSpans(
    spans,
    semconv="auto",
    trace_id=None,
    frame="production-representative",
    taxonomy_cell=dict(),
)

One session's raw OTLP-JSON spans, in memory, to normalize via the convention layer.

spans is the list of span dicts (canonical OTLP-JSON shape) for a single session/record. semconv="auto" runs detection; a named Convention forces one adapter. The header fields resolve in the layered order (convention attribute → this override → wire/default). This is what a dataset row becomes before the convention adapter turns it into a Trace.

Attributes:

Name Type Description
spans list[dict[str, Any]]

The session's spans as OTLP-JSON dicts.

semconv Convention

The convention to read with, or "auto".

trace_id str | None

The trace id, when the spans don't carry one.

frame Frame

The sampling frame.

taxonomy_cell dict[str, str]

The taxonomy cell, when the spans don't carry one.

spans instance-attribute

spans

semconv class-attribute instance-attribute

semconv = 'auto'

trace_id class-attribute instance-attribute

trace_id = None

frame class-attribute instance-attribute

frame = 'production-representative'

taxonomy_cell class-attribute instance-attribute

taxonomy_cell = field(default_factory=dict)

StrandsSession dataclass

StrandsSession(
    path,
    mode="latest",
    trace_id=None,
    frame="production-representative",
    taxonomy_cell=dict(),
)

An OTel-native Strands export.

Strands is pattern-agnostic (Graph / Swarm / Agent-as-Tools all share one span-name grammar); mode selects the legacy-vs-latest GenAI convention. Headers resolve in the layered order.

Attributes:

Name Type Description
path Path

The exported OTLP-JSON file.

mode Literal['legacy', 'latest']

The GenAI convention generation the export uses.

trace_id str | None

The trace id, when the spans don't carry one.

frame Frame

The sampling frame.

taxonomy_cell dict[str, str]

The taxonomy cell, when the spans don't carry one.

path instance-attribute

path

mode class-attribute instance-attribute

mode = 'latest'

trace_id class-attribute instance-attribute

trace_id = None

frame class-attribute instance-attribute

frame = 'production-representative'

taxonomy_cell class-attribute instance-attribute

taxonomy_cell = field(default_factory=dict)

tag_span

tag_span(set_attribute, *, trace_id, frame, taxonomy_cell)

Stamp the library's header convention onto an OTel span at instrumentation time.

set_attribute is the span's set_attribute(key, value) method (passed in so this helper carries no opentelemetry import; the boundary stays clean). Call it on the root invoke_agent span so the normalizer reads trace_id/frame/taxonomy_cell from the wire (the first layer of header resolution) rather than falling back to synthesized values.

Source code in src/calibrated_agent_evaluation/normalize/guidance.py
def tag_span(
    set_attribute: Callable[[str, Any], object],
    *,
    trace_id: str,
    frame: str,
    taxonomy_cell: dict[str, str],
) -> None:
    """Stamp the library's header convention onto an OTel span at instrumentation time.

    ``set_attribute`` is the span's ``set_attribute(key, value)`` method (passed in so this
    helper carries no ``opentelemetry`` import; the boundary stays clean). Call it on the root
    ``invoke_agent`` span so the normalizer reads ``trace_id``/``frame``/``taxonomy_cell`` from
    the wire (the first layer of header resolution) rather than falling back to synthesized values.
    """
    set_attribute(ATTR_TRACE_ID, trace_id)
    set_attribute(ATTR_FRAME, frame)
    for key, value in taxonomy_cell.items():
        set_attribute(f"{ATTR_TAXONOMY_PREFIX}{key}", value)

normalize

normalize(source)

Normalize one raw trace source into a validated Trace.

Returns an owned Trace when a span tree with per-step content was recovered, or a degraded probed Trace (spans == [], terminal I/O preserved) when it was not. Raises NormalizationError (chained via from) on an unrecognized/unsupported source, a missing opt-in extra, a vendor/parser/ filesystem read failure, or a mapped Trace that fails validate(). It never returns a partially-populated Trace.

Parameters:

Name Type Description Default
source NormalizerSource | SessionSpans

The raw trace to read: a file source such as OTelSpans or InspectEvalLog, or in-memory SessionSpans.

required

Returns:

Type Description
Trace

The validated trace, owned or probed.

Raises:

Type Description
NormalizationError

The source can't be read, isn't recognized, needs a missing extra, or maps to a trace that fails validation.

Source code in src/calibrated_agent_evaluation/normalize/__init__.py
def normalize(source: NormalizerSource | SessionSpans) -> Trace:
    """Normalize one raw trace source into a validated :class:`Trace`.

    Returns an ``owned`` ``Trace`` when a span tree with per-step content was recovered, or a
    degraded ``probed`` ``Trace`` (``spans == []``, terminal I/O preserved) when it was not.
    Raises :class:`NormalizationError` (chained via ``from``) on an
    unrecognized/unsupported source, a missing opt-in extra, a vendor/parser/
    filesystem read failure, or a mapped ``Trace`` that fails ``validate()``.
    It never returns a partially-populated ``Trace``.

    Args:
        source: The raw trace to read: a file source such as ``OTelSpans`` or ``InspectEvalLog``,
            or in-memory ``SessionSpans``.

    Returns:
        The validated trace, ``owned`` or ``probed``.

    Raises:
        NormalizationError: The source can't be read, isn't recognized, needs a missing extra, or
            maps to a trace that fails validation.
    """
    trace, _recovery = normalize_with_recovery(source)
    return trace

normalize_all

normalize_all(source)

Normalize a source into EVERY validated Trace it holds: one per unit of evaluation.

For an InspectEvalLog that unit is a sample, and a real eval log holds one per dataset row: the adapter used to read the first sample and drain the rest away, so an eval over N rows produced 1 trace and every downstream rate, count and gate decision came from 1/N of the data with no error anywhere. Every other source is one trace by construction, so this returns a single-element list for them and callers need no special-casing.

Parameters:

Name Type Description Default
source NormalizerSource | SessionSpans

The raw trace source to read.

required

Returns:

Type Description
list[Trace]

One validated trace per unit of evaluation in the source.

Raises:

Type Description
NormalizationError

As for normalize.

Source code in src/calibrated_agent_evaluation/normalize/__init__.py
def normalize_all(source: NormalizerSource | SessionSpans) -> list[Trace]:
    """Normalize a source into EVERY validated ``Trace`` it holds: one per unit of evaluation.

    For an :class:`InspectEvalLog` that unit is a **sample**, and a real eval log holds one per
    dataset row: the adapter used to read the first sample and drain the rest away, so an eval over
    N rows produced 1 trace and every downstream rate, count and gate decision came from 1/N of the
    data with no error anywhere. Every other source is one trace by construction, so this returns a
    single-element list for them and callers need no special-casing.

    Args:
        source: The raw trace source to read.

    Returns:
        One validated trace per unit of evaluation in the source.

    Raises:
        NormalizationError: As for :func:`normalize`.
    """
    if isinstance(source, InspectEvalLog):
        from calibrated_agent_evaluation.normalize.inspect_adapter import to_intermediates

        traces = [_validated(trace) for trace in _traces_from(to_intermediates(source))]
        _refuse_duplicate_ids(source, traces)
        return traces
    return [normalize(source)]

normalize_with_recovery

normalize_with_recovery(source)

normalize, additionally returning what was recovered before the ownership collapse.

For a caller that has to distinguish "the adapter recovered nothing" from "the adapter recovered a tree whose every node was empty"; the ingest gates, which refuse the first and degrade the second. Both produce a probed trace with spans == [], so the finished Trace cannot tell them apart on its own.

Source code in src/calibrated_agent_evaluation/normalize/__init__.py
def normalize_with_recovery(
    source: NormalizerSource | SessionSpans,
) -> tuple[Trace, Recovery]:
    """:func:`normalize`, additionally returning what was recovered before the ownership collapse.

    For a caller that has to distinguish "the adapter recovered nothing" from "the adapter recovered
    a tree whose every node was empty"; the ingest gates, which refuse the first and degrade the
    second. Both produce a ``probed`` trace with ``spans == []``, so the finished ``Trace`` cannot
    tell them apart on its own.
    """
    intermediate = _to_intermediate(source)
    trace, recovery = resolve_with_recovery(intermediate)
    try:
        # Only schema-valid Traces leave the normalizer. A mapping bug (duplicate seq,
        # dangling parent, ...) surfaces as SchemaError → wrapped typed, never a raw SchemaError.
        validate(trace)
    except SchemaError as exc:
        raise NormalizationError(
            f"normalized trace {trace.trace_id!r} failed schema validation: {exc}"
        ) from exc
    return trace, recovery