Skip to content

calibrated_agent_evaluation.normalize.conventions

calibrated_agent_evaluation.normalize.conventions

OTel GenAI trace-convention adapters.

Each convention adapter turns a list of raw OTLP-JSON spans (one session) into the vendor-free Intermediate, so the shared tree builder + ownership gate + schema validation produce the normalized Trace. The generic OTel adapter already ships the gen_ai/legacy and gen_ai/latest (invoke_agent/execute_tool) grammars inside otel_adapter; this package adds the three widely-emitted conventions the wild actually uses; gen_ai/messages (Claude Code / current OpenLLMetry), legacy openllmetry (Traceloop dotted-index), and openinference (Arize/Phoenix); plus a hybrid detect front-door and a registry so an unshipped convention is addable without editing core.

The convention layer is shared: both a single OTLP file (OTelSpans) and a dataset row (SessionSpans) route through the same adapters, so a convention is reconstructed by exactly one adapter regardless of how the spans arrived. This package is vendor-free; it reads plain OTLP-JSON attribute keys via otlp_json; no openllmetry/openinference runtime dependency exists.

ConventionAdapter

Bases: Protocol

Turns one session's raw spans + headers into the vendor-free Intermediate.

AgentNode dataclass

AgentNode(
    agent,
    role,
    parent,
    input=dict(),
    key=None,
    parent_key=None,
    output=dict(),
    tool_calls=list(),
    span_id=None,
    started_at=None,
    ended_at=None,
    order=0,
    status="completed",
    usage=None,
)

One recovered agent in the run (vendor-free intermediate).

agent is the human agent name; role its normalized role; parent the agent name of its parent (None for the entry node). input/output are the recovered per-step content (empty dicts when the source carried none; the ownership gate reads that). span_id lets an adapter pin the {trace_id}/{agent} id; when None the builder derives it. order is the adapter's execution order hint used to assign seq. status is the recovered per-step outcome: an adapter that can read the source's error signal sets "errored", and one that cannot leaves the "completed" default; which is a claim about the adapter, so it must not be read as a claim that the step succeeded.

agent instance-attribute

agent

role instance-attribute

role

parent instance-attribute

parent

input class-attribute instance-attribute

input = field(default_factory=dict)

key class-attribute instance-attribute

key = None

parent_key class-attribute instance-attribute

parent_key = None

output class-attribute instance-attribute

output = field(default_factory=dict)

tool_calls class-attribute instance-attribute

tool_calls = field(default_factory=list)

span_id class-attribute instance-attribute

span_id = None

started_at class-attribute instance-attribute

started_at = None

ended_at class-attribute instance-attribute

ended_at = None

order class-attribute instance-attribute

order = 0

status class-attribute instance-attribute

status = 'completed'

usage class-attribute instance-attribute

usage = None

Intermediate dataclass

Intermediate(
    trace_id,
    frame,
    taxonomy_cell,
    trace_input,
    trace_output,
    nodes=list(),
    captured_at=None,
    degrade_reason=None,
    convention=None,
)

The vendor-free hand-off from an adapter to the tree builder + ownership gate.

nodes are the recovered agents (already ordered); trace_input/trace_output are the terminal request/decision; the headers (trace_id/frame/taxonomy_cell) are resolved by the adapter in the layered header order. degrade_reason is set by an adapter that recovered no per-step content and wants the ownership gate to build a probed trace.

trace_id instance-attribute

trace_id

frame instance-attribute

frame

taxonomy_cell instance-attribute

taxonomy_cell

trace_input instance-attribute

trace_input

trace_output instance-attribute

trace_output

nodes class-attribute instance-attribute

nodes = field(default_factory=list)

captured_at class-attribute instance-attribute

captured_at = None

degrade_reason class-attribute instance-attribute

degrade_reason = None

convention class-attribute instance-attribute

convention = None

ToolInvocation dataclass

ToolInvocation(name, args, result)

One recovered leaf tool call (no nested agent): becomes a Span.tool_calls entry.

An agent/subagent boundary is NOT a ToolInvocation; the adapter promotes it to its own AgentNode (the tool-vs-span discriminator). args/result are already decoded to dicts by the adapter's content reader.

name instance-attribute

name

args instance-attribute

args

result instance-attribute

result

get_adapter

get_adapter(kind)

Return the adapter for kind, else raise naming what's registered.

Source code in src/calibrated_agent_evaluation/normalize/conventions/registry.py
def get_adapter(kind: str) -> ConventionAdapter:
    """Return the adapter for ``kind``, else raise naming what's registered."""
    try:
        return _REGISTRY[kind]
    except KeyError as exc:
        raise NormalizationError(
            f"no convention adapter registered for {kind!r}; "
            f"registered conventions are {sorted(_REGISTRY)}"
        ) from exc

register_convention

register_convention(kind, adapter)

Register a user convention adapter under kind.

Refuses to shadow a built-in kind or to silently replace an already-registered custom kind; both raise a typed NormalizationError, so a registration clash is loud, not silent.

Source code in src/calibrated_agent_evaluation/normalize/conventions/registry.py
def register_convention(kind: str, adapter: ConventionAdapter) -> None:
    """Register a user convention adapter under ``kind``.

    Refuses to shadow a built-in kind or to silently replace an already-registered custom kind;
    both raise a typed :class:`NormalizationError`, so a registration clash is loud, not silent.
    """
    if kind in _BUILTIN_KINDS:
        raise NormalizationError(
            f"cannot register convention {kind!r}: it is a built-in adapter and may not be "
            f"overridden (built-ins: {sorted(_BUILTIN_KINDS)})"
        )
    if kind in _REGISTRY:
        raise NormalizationError(
            f"cannot register convention {kind!r}: a custom adapter is already registered under "
            f"that kind; unregister or choose a distinct kind"
        )
    _REGISTRY[kind] = adapter

registered_kinds

registered_kinds()

The currently-registered convention kinds (built-in + user).

Source code in src/calibrated_agent_evaluation/normalize/conventions/registry.py
def registered_kinds() -> frozenset[str]:
    """The currently-registered convention kinds (built-in + user)."""
    return frozenset(_REGISTRY)