calibrated_agent_evaluation.normalize¶
calibrated_agent_evaluation.normalize ¶
The trajectory normalizer: normalize(source) -> Trace.
The single seam that turns a vendor-shaped trace (Inspect .eval, generic OTLP, a Strands
session) into the one normalized Trace, so the
methodology plane is written once against Trace/Span and never sees a vendor type.
It dispatches on the source descriptor, runs the chosen adapter to a vendor-free intermediate,
applies the ownership gate (owned vs degraded probed), validates against the schema,
and returns the Trace.
This package is the second member of the methodology-plane vendor allowlist (alongside
backends/, execute/): the adapters import inspect_ai / opentelemetry.proto;
lazily, inside their modules; so importing calibrated_agent_evaluation.normalize needs no opt-in
extra, and a missing extra surfaces as a typed NormalizationError naming the installable
string, never a bare ImportError.
NormalizerSource
module-attribute
¶
NormalizationError ¶
Bases: CalibratedAgentEvaluationDataError
The normalizer could not produce a valid Trace from a source.
Raised (never swallowed and never partially returned) on an unrecognized/unsupported
source, framework kind, or OTel semantic convention; a missing opt-in extra
(the message names the install command for it); a vendor/parser/
filesystem read failure (chained via from); or a mapped Trace that fails
the schema validate().
Recovery
dataclass
¶
What the adapter recovered, stated before the ownership gate collapsed it.
The gate's output alone cannot answer "did normalization work?", because a probed trace has
spans == [] whether the adapter recovered nothing or recovered a full tree whose every node
turned out to be empty. Those are a defect and a declared degradation respectively, and the
ingest gates have to tell them apart: the loss gate refuses tool-execution spans the adapter
failed to link, while the degrade rule degrades an all-contentless tree and explicitly must
not refuse it.
So the pre-collapse counts travel beside the trace rather than being re-derived downstream.
reason is None for an owned trace and otherwise the message that was logged.
InspectEvalLog
dataclass
¶
InspectEvalLog(
path,
trace_id=None,
frame="production-representative",
taxonomy_cell=dict(),
sample_id=None,
)
An Inspect .eval (or .json) log to normalize.
trace_id/taxonomy_cell are read from the sample metadata when present; frame
is analyst-assigned and defaults to "production-representative" (override per capture).
An Inspect log holds one sample per dataset row, so a real eval log is normally
multi-sample. Use normalize_all for one Trace
per sample; on a multi-sample log refuses unless sample_id names which one to read, rather
than silently evaluating the log at 1/N of its data.
Attributes:
| Name | Type | Description |
|---|---|---|
path |
Path
|
The |
trace_id |
str | None
|
The trace id; read from the sample metadata when omitted. |
frame |
Frame
|
The sampling frame. |
taxonomy_cell |
dict[str, str]
|
The taxonomy cell; read from the sample metadata when omitted. |
sample_id |
str | None
|
The sample to read from a multi-sample log. |
OTelSpans
dataclass
¶
OTelSpans(
path,
semconv="auto",
content_location="events",
trace_id=None,
frame="production-representative",
taxonomy_cell=dict(),
)
A generic OpenTelemetry source.
path is a frozen OTLP-JSON file (read with stdlib json) or a binary .otlp
protobuf (needs the [otel] extra). semconv selects the convention branch.
content_location is a best-effort preference hint, not a hard gate: the reader tries
the named placement first, then falls back to the other, because real traces mix span
attributes and span events; falling back is what prevents the silent-drop of
event-located content. The header fields resolve in the layered order (convention attribute →
this override → OTel-traceId hex / defaults).
Attributes:
| Name | Type | Description |
|---|---|---|
path |
Path
|
The OTLP-JSON or binary |
semconv |
Semconv | Convention
|
The OTel GenAI convention to read with, or |
content_location |
ContentLocation
|
Where to look for message content first. |
trace_id |
str | None
|
The trace id, when the spans don't carry one. |
frame |
Frame
|
The sampling frame. |
taxonomy_cell |
dict[str, str]
|
The taxonomy cell, when the spans don't carry one. |
SessionSpans
dataclass
¶
SessionSpans(
spans,
semconv="auto",
trace_id=None,
frame="production-representative",
taxonomy_cell=dict(),
)
One session's raw OTLP-JSON spans, in memory, to normalize via the convention layer.
spans is the list of span dicts (canonical OTLP-JSON shape) for a single session/record.
semconv="auto" runs detection; a named Convention forces one adapter.
The header fields resolve in the layered order (convention attribute → this override →
wire/default).
This is what a dataset row becomes before the convention adapter turns it into a Trace.
Attributes:
| Name | Type | Description |
|---|---|---|
spans |
list[dict[str, Any]]
|
The session's spans as OTLP-JSON dicts. |
semconv |
Convention
|
The convention to read with, or |
trace_id |
str | None
|
The trace id, when the spans don't carry one. |
frame |
Frame
|
The sampling frame. |
taxonomy_cell |
dict[str, str]
|
The taxonomy cell, when the spans don't carry one. |
StrandsSession
dataclass
¶
StrandsSession(
path,
mode="latest",
trace_id=None,
frame="production-representative",
taxonomy_cell=dict(),
)
An OTel-native Strands export.
Strands is pattern-agnostic (Graph / Swarm / Agent-as-Tools all share one span-name
grammar); mode selects the legacy-vs-latest GenAI convention. Headers resolve in the
layered order.
Attributes:
| Name | Type | Description |
|---|---|---|
path |
Path
|
The exported OTLP-JSON file. |
mode |
Literal['legacy', 'latest']
|
The GenAI convention generation the export uses. |
trace_id |
str | None
|
The trace id, when the spans don't carry one. |
frame |
Frame
|
The sampling frame. |
taxonomy_cell |
dict[str, str]
|
The taxonomy cell, when the spans don't carry one. |
tag_span ¶
Stamp the library's header convention onto an OTel span at instrumentation time.
set_attribute is the span's set_attribute(key, value) method (passed in so this
helper carries no opentelemetry import; the boundary stays clean). Call it on the root
invoke_agent span so the normalizer reads trace_id/frame/taxonomy_cell from
the wire (the first layer of header resolution) rather than falling back to synthesized values.
Source code in src/calibrated_agent_evaluation/normalize/guidance.py
normalize ¶
Normalize one raw trace source into a validated Trace.
Returns an owned Trace when a span tree with per-step content was recovered, or a
degraded probed Trace (spans == [], terminal I/O preserved) when it was not.
Raises NormalizationError (chained via from) on an
unrecognized/unsupported source, a missing opt-in extra, a vendor/parser/
filesystem read failure, or a mapped Trace that fails validate().
It never returns a partially-populated Trace.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
NormalizerSource | SessionSpans
|
The raw trace to read: a file source such as |
required |
Returns:
| Type | Description |
|---|---|
Trace
|
The validated trace, |
Raises:
| Type | Description |
|---|---|
NormalizationError
|
The source can't be read, isn't recognized, needs a missing extra, or maps to a trace that fails validation. |
Source code in src/calibrated_agent_evaluation/normalize/__init__.py
normalize_all ¶
Normalize a source into EVERY validated Trace it holds: one per unit of evaluation.
For an InspectEvalLog that unit is a sample, and a real eval log holds one per
dataset row: the adapter used to read the first sample and drain the rest away, so an eval over
N rows produced 1 trace and every downstream rate, count and gate decision came from 1/N of the
data with no error anywhere. Every other source is one trace by construction, so this returns a
single-element list for them and callers need no special-casing.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
NormalizerSource | SessionSpans
|
The raw trace source to read. |
required |
Returns:
| Type | Description |
|---|---|
list[Trace]
|
One validated trace per unit of evaluation in the source. |
Raises:
| Type | Description |
|---|---|
NormalizationError
|
As for |
Source code in src/calibrated_agent_evaluation/normalize/__init__.py
normalize_with_recovery ¶
normalize, additionally returning what was recovered before the ownership collapse.
For a caller that has to distinguish "the adapter recovered nothing" from "the adapter recovered
a tree whose every node was empty"; the ingest gates, which refuse the first and degrade the
second. Both produce a probed trace with spans == [], so the finished Trace cannot
tell them apart on its own.