calibrated_agent_evaluation.normalize.conventions¶
calibrated_agent_evaluation.normalize.conventions ¶
OTel GenAI trace-convention adapters.
Each convention adapter turns a list of raw OTLP-JSON spans (one session) into the
vendor-free Intermediate, so the shared tree
builder + ownership gate + schema validation produce the normalized Trace. The generic OTel
adapter already ships the gen_ai/legacy and gen_ai/latest
(invoke_agent/execute_tool) grammars inside otel_adapter; this package adds the three
widely-emitted conventions the wild actually uses; gen_ai/messages (Claude Code / current
OpenLLMetry), legacy openllmetry (Traceloop dotted-index), and openinference
(Arize/Phoenix); plus a hybrid detect front-door and a registry so an unshipped
convention is addable without editing core.
The convention layer is shared: both a single OTLP file (OTelSpans) and a dataset
row (SessionSpans) route through the same adapters, so a convention is reconstructed by
exactly one adapter regardless of how the spans arrived. This package is vendor-free;
it reads plain OTLP-JSON attribute keys via otlp_json; no openllmetry/openinference
runtime dependency exists.
ConventionAdapter ¶
AgentNode
dataclass
¶
AgentNode(
agent,
role,
parent,
input=dict(),
key=None,
parent_key=None,
output=dict(),
tool_calls=list(),
span_id=None,
started_at=None,
ended_at=None,
order=0,
status="completed",
usage=None,
)
One recovered agent in the run (vendor-free intermediate).
agent is the human agent name; role its normalized role; parent the agent
name of its parent (None for the entry node). input/output are the recovered
per-step content (empty dicts when the source carried none; the ownership gate reads that).
span_id lets an adapter pin the {trace_id}/{agent} id; when None the
builder derives it. order is the adapter's execution order hint used to assign seq.
status is the recovered per-step outcome: an adapter that can read the source's error
signal sets "errored", and one that cannot leaves the "completed" default; which is
a claim about the adapter, so it must not be read as a claim that the step succeeded.
Intermediate
dataclass
¶
Intermediate(
trace_id,
frame,
taxonomy_cell,
trace_input,
trace_output,
nodes=list(),
captured_at=None,
degrade_reason=None,
convention=None,
)
The vendor-free hand-off from an adapter to the tree builder + ownership gate.
nodes are the recovered agents (already ordered); trace_input/trace_output are
the terminal request/decision; the headers (trace_id/frame/taxonomy_cell) are
resolved by the adapter in the layered header order. degrade_reason is set by an adapter
that recovered no per-step content and wants the ownership gate to build a probed trace.
ToolInvocation
dataclass
¶
One recovered leaf tool call (no nested agent): becomes a Span.tool_calls entry.
An agent/subagent boundary is NOT a ToolInvocation; the adapter promotes it to its own
AgentNode (the tool-vs-span discriminator). args/result are already
decoded to dicts by the adapter's content reader.
get_adapter ¶
Return the adapter for kind, else raise naming what's registered.
Source code in src/calibrated_agent_evaluation/normalize/conventions/registry.py
register_convention ¶
Register a user convention adapter under kind.
Refuses to shadow a built-in kind or to silently replace an already-registered custom kind;
both raise a typed NormalizationError, so a registration clash is loud, not silent.