Skip to content

calibrated_agent_evaluation.schema

calibrated_agent_evaluation.schema

Normalized-schema contract for the library's data plane.

The vendor-neutral Trace / Span / Label types every plane serializes to, the validate gate that enforces the schema invariants, the typed error hierarchy, and the SCHEMA_VERSION constant. Nothing here imports a vendor SDK type; this package is the firewall the methodology plane reads through.

SCHEMA_VERSION module-attribute

SCHEMA_VERSION = '1'

Frame module-attribute

Frame = Literal['discovery', 'production-representative']

LabelKind module-attribute

LabelKind = Literal[
    "human", "llm-judge", "code", "policy-engine"
]

Outcome module-attribute

Outcome = Literal['PASS', 'FAIL']

Ownership module-attribute

Ownership = Literal['owned', 'probed']

Role module-attribute

Role = Literal['specialist', 'supervisor', 'tool']

SpanStatus module-attribute

SpanStatus = Literal['completed', 'errored']

BackendError

Bases: CalibratedAgentEvaluationDataError

A configured backend is unreachable or rejected a request.

Raised without any silent fallback to the local backend; failure surfaces honestly, never silently degrades worlds.

CalibratedAgentEvaluationDataError

Bases: Exception

Base class for every data-plane error.

SchemaError

Bases: CalibratedAgentEvaluationDataError

An artifact violates the normalized-schema contract.

Raised on a missing/empty trace_id or a Label whose expected is set while critique is empty/missing; expected may never substitute for the human critique.

SchemaVersionError

Bases: CalibratedAgentEvaluationDataError

load read an artifact whose schema_version ≠ the running SCHEMA_VERSION.

Raised rather than coercing the data; refuse-loudly is the honest v1 behavior; migrators are later work.

TraceNotFoundError

Bases: BackendError

The backend answered, and it holds no trace with the requested id.

Distinct from its parent so a caller can tell "this id is free" from "the read failed": only the first may be acted on as an absence.

UnregisteredArtifactError

Bases: CalibratedAgentEvaluationDataError

dump/load hit an artifact type with no registered codec.

Raised rather than silently skipping the artifact; a silent skip would make the no-lock-in round-trip lie.

Label pydantic-model

Bases: CalibratedAgentEvaluationModel

The universal eval-result object, mapped to a Langfuse Score on write.

Binary outcome and a required, non-empty critique are structural: a non-binary or critique-less label cannot be represented. Carries the crew-attribution fields (culprit_span / manifestation_span / handoff_implicated / span_defect) and codebook_version; each references a trace_id and an optional span_id.

Config:

  • frozen: True
  • extra: forbid

Fields:

trace_id pydantic-field

trace_id

span_id pydantic-field

span_id

kind pydantic-field

kind

outcome pydantic-field

outcome

critique pydantic-field

critique

failure_mode pydantic-field

failure_mode = None

culprit_span pydantic-field

culprit_span = None

manifestation_span pydantic-field

manifestation_span = None

handoff_implicated pydantic-field

handoff_implicated = None

span_defect pydantic-field

span_defect = False

codebook_version pydantic-field

codebook_version = None

expected pydantic-field

expected = None

rater_id pydantic-field

rater_id = None

Span pydantic-model

Bases: CalibratedAgentEvaluationModel

One agent step in an owned trace.

Preserves the handoff input received, the tool_calls made, the output emitted, and explicit tree ordering via parent + seq (Langfuse has no native ordering, so seq is carried explicitly).

usage carries the per-step token counts when the source reported them. It exists because normalizing used to discard them: OTel sources carry gen_ai.usage.* per agent span, the schema had nowhere to put it, and so a normalized Trace could describe a run's topology while being silent about what the run cost; which is much of why a team ingests traces at all. None means the source reported nothing, and is deliberately different from a TokenUsage whose fields are all None: the second says a usage record existed and was empty.

Config:

  • frozen: True
  • extra: forbid

Fields:

span_id pydantic-field

span_id

trace_id pydantic-field

trace_id

agent pydantic-field

agent

role pydantic-field

role

parent pydantic-field

parent

seq pydantic-field

seq

status pydantic-field

status

input pydantic-field

input

tool_calls pydantic-field

tool_calls

output pydantic-field

output

started_at pydantic-field

started_at = None

ended_at pydantic-field

ended_at = None

usage pydantic-field

usage = None

occurrence pydantic-field

occurrence = 1

TokenUsage pydantic-model

Bases: CalibratedAgentEvaluationModel

Tokens a single agent step consumed, as the source reported them.

Every field is optional and independently so, because a source may report a total without its parts (or the reverse), and a normalizer must never compute one from the others; a derived total is indistinguishable from a reported one once it is stored, and the two disagree whenever a provider counts cached or reasoning tokens outside the pair. Absent means not reported, never zero.

Config:

  • frozen: True
  • extra: forbid

Fields:

input_tokens pydantic-field

input_tokens = None

output_tokens pydantic-field

output_tokens = None

total_tokens pydantic-field

total_tokens = None

ToolCall pydantic-model

Bases: CalibratedAgentEvaluationModel

One tool invocation inside a span. Reconstructed by the normalizer.

Config:

  • frozen: True
  • extra: forbid

Fields:

name pydantic-field

name

args pydantic-field

args

result pydantic-field

result

Trace pydantic-model

Bases: CalibratedAgentEvaluationModel

The central object.

An owned trace carries a populated spans tree; a probed trace is input/output only with spans == [] (and an endpoint_fingerprint). frame, ownership, and taxonomy_cell are first-class queryable dimensions.

tool_calls live ONLY on Span; Trace deliberately has no tool_calls field. A single-agent owned trace is therefore represented as a one-span tree: a single root Span (parent=None, seq=0) that holds the agent's tool_calls (see fixtures/normalized/ORD-1001.json). The methodology plane reads tool calls from exactly one place (span.tool_calls) for both single- and multi-agent traces; no branching on agent architecture (target-agnosticism). The Trace-level input/output remain the terminal request/decision; the root span's input/output mirror them for a single-agent trace.

Config:

  • frozen: True
  • extra: forbid

Fields:

trace_id pydantic-field

trace_id

frame pydantic-field

frame

ownership pydantic-field

ownership

input pydantic-field

input

output pydantic-field

output

taxonomy_cell pydantic-field

taxonomy_cell

spans pydantic-field

spans

captured_at pydantic-field

captured_at = None

endpoint_fingerprint pydantic-field

endpoint_fingerprint = None

session_id pydantic-field

session_id = None

schema_version pydantic-field

schema_version = SCHEMA_VERSION

label_from_dict

label_from_dict(data)

Rebuild a Label from the dict to_dict produced.

A document written before rater_id existed is unattributed, not malformed: the field defaults to None.

Source code in src/calibrated_agent_evaluation/schema/serde.py
def label_from_dict(data: dict[str, Any]) -> Label:
    """Rebuild a ``Label`` from the dict :func:`to_dict` produced.

    A document written before ``rater_id`` existed is unattributed, not malformed: the field
    defaults to ``None``.
    """
    return Label.model_validate(data)

latest_verdicts

latest_verdicts(labels)

Keep one machine verdict per target, failure mode and rater: the newest. Order is kept.

labels must be oldest first, which is the order get_scores returns. A judge re-run writes a new verdict for every target it already scored, and a live model words its critique differently each time, so dedup_labels keeps both rows. Counting both doubles n with no new evidence, which narrows every interval and can turn a failing gate green.

Only LLM-judge and code-scorer verdicts with a rater_id are collapsed, keyed on (trace_id, span_id, kind, failure_mode, rater_id): that rater id names the judge and its exact version, so two such rows are the same judgment made twice. Human labels and verdicts with no recorded rater pass through unchanged, since two of them on one target may be different people or different judges.

Parameters:

Name Type Description Default
labels Iterable[Label]

Labels in write order, oldest first.

required

Returns:

Type Description
list[Label]

The labels with every superseded machine verdict removed.

Source code in src/calibrated_agent_evaluation/schema/serde.py
def latest_verdicts(labels: Iterable[Label]) -> list[Label]:
    """Keep one machine verdict per target, failure mode and rater: the newest. Order is kept.

    ``labels`` must be oldest first, which is the order ``get_scores`` returns. A judge re-run
    writes a new verdict for every target it already scored, and a live model words its critique
    differently each time, so :func:`dedup_labels` keeps both rows. Counting both doubles n with
    no new evidence, which narrows every interval and can turn a failing gate green.

    Only LLM-judge and code-scorer verdicts with a ``rater_id`` are collapsed, keyed on
    ``(trace_id, span_id, kind, failure_mode, rater_id)``: that rater id names the judge and its
    exact version, so two such rows are the same judgment made twice. Human labels and verdicts
    with no recorded rater pass through unchanged, since two of them on one target may be
    different people or different judges.

    Args:
        labels: Labels in write order, oldest first.

    Returns:
        The labels with every superseded machine verdict removed.
    """
    ordered = list(labels)
    newest: dict[tuple[object, ...], int] = {}
    for index, label in enumerate(ordered):
        if label.kind in _MACHINE_KINDS and label.rater_id is not None:
            key = (label.trace_id, label.span_id, label.kind, label.failure_mode, label.rater_id)
            newest[key] = index
    keep = set(newest.values())
    return [
        label
        for index, label in enumerate(ordered)
        if index in keep or label.kind not in _MACHINE_KINDS or label.rater_id is None
    ]

span_from_dict

span_from_dict(data)

Rebuild a Span from the dict to_dict produced.

Source code in src/calibrated_agent_evaluation/schema/serde.py
def span_from_dict(data: dict[str, Any]) -> Span:
    """Rebuild a ``Span`` from the dict :func:`to_dict` produced."""
    return Span.model_validate(data)

to_dict

to_dict(artifact)

Return a JSON-serializable dict for a normalized artifact (nested objects included).

Parameters:

Name Type Description Default
artifact Trace | Span | Label | ToolCall

The artifact to convert.

required

Returns:

Type Description
dict[str, Any]

The artifact as plain dicts and lists.

Source code in src/calibrated_agent_evaluation/schema/serde.py
def to_dict(artifact: Trace | Span | Label | ToolCall) -> dict[str, Any]:
    """Return a JSON-serializable dict for a normalized artifact (nested objects included).

    Args:
        artifact: The artifact to convert.

    Returns:
        The artifact as plain dicts and lists.
    """
    return artifact.model_dump(mode="json")

trace_from_dict

trace_from_dict(data)

Rebuild a Trace from the dict to_dict produced.

Parameters:

Name Type Description Default
data dict[str, Any]

The dict form of a trace.

required

Returns:

Type Description
Trace

The trace.

Raises:

Type Description
SchemaError

The dict carries no schema_version.

Source code in src/calibrated_agent_evaluation/schema/serde.py
def trace_from_dict(data: dict[str, Any]) -> Trace:
    """Rebuild a ``Trace`` from the dict :func:`to_dict` produced.

    Args:
        data: The dict form of a trace.

    Returns:
        The trace.

    Raises:
        SchemaError: The dict carries no ``schema_version``.
    """
    _require_schema_version(data)
    return Trace.model_validate(data)