calibrated_agent_evaluation.schema¶
calibrated_agent_evaluation.schema ¶
Normalized-schema contract for the library's data plane.
The vendor-neutral Trace / Span / Label types every plane serializes to,
the validate gate that enforces the schema invariants, the typed
error hierarchy, and the SCHEMA_VERSION constant. Nothing here imports a vendor SDK
type; this package is the firewall the methodology plane reads through.
BackendError ¶
Bases: CalibratedAgentEvaluationDataError
A configured backend is unreachable or rejected a request.
Raised without any silent fallback to the local backend; failure surfaces honestly, never silently degrades worlds.
SchemaError ¶
Bases: CalibratedAgentEvaluationDataError
An artifact violates the normalized-schema contract.
Raised on a missing/empty trace_id or a Label whose expected is
set while critique is empty/missing; expected may never substitute
for the human critique.
SchemaVersionError ¶
Bases: CalibratedAgentEvaluationDataError
load read an artifact whose schema_version ≠ the running SCHEMA_VERSION.
Raised rather than coercing the data; refuse-loudly is the honest v1 behavior; migrators are later work.
TraceNotFoundError ¶
Bases: BackendError
The backend answered, and it holds no trace with the requested id.
Distinct from its parent so a caller can tell "this id is free" from "the read failed": only the first may be acted on as an absence.
UnregisteredArtifactError ¶
Bases: CalibratedAgentEvaluationDataError
dump/load hit an artifact type with no registered codec.
Raised rather than silently skipping the artifact; a silent skip would make the no-lock-in round-trip lie.
Label
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
The universal eval-result object, mapped to a Langfuse Score on write.
Binary outcome and a required, non-empty critique are structural:
a non-binary or critique-less label cannot be represented. Carries the crew-attribution
fields (culprit_span / manifestation_span / handoff_implicated / span_defect)
and codebook_version; each references a trace_id and an optional span_id.
Config:
frozen:Trueextra:forbid
Fields:
-
trace_id(str) -
span_id(str | None) -
kind(LabelKind) -
outcome(Outcome) -
critique(str) -
failure_mode(str | None) -
culprit_span(str | None) -
manifestation_span(str | None) -
handoff_implicated(str | None) -
span_defect(bool) -
codebook_version(str | None) -
expected(dict[str, Any] | None) -
rater_id(str | None)
Span
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
One agent step in an owned trace.
Preserves the handoff input received, the tool_calls made, the output
emitted, and explicit tree ordering via parent + seq (Langfuse has no native
ordering, so seq is carried explicitly).
usage carries the per-step token counts when the source reported them. It exists because
normalizing used to discard them: OTel sources carry gen_ai.usage.* per agent span, the
schema had nowhere to put it, and so a normalized Trace could describe a run's topology
while being silent about what the run cost; which is much of why a team ingests traces at all.
None means the source reported nothing, and is deliberately different from a
TokenUsage whose fields are all None: the second says a usage record existed and
was empty.
Config:
frozen:Trueextra:forbid
Fields:
-
span_id(str) -
trace_id(str) -
agent(str) -
role(Role) -
parent(str | None) -
seq(StrictInt) -
status(SpanStatus) -
input(dict[str, Any]) -
tool_calls(list[ToolCall]) -
output(dict[str, Any]) -
started_at(str | None) -
ended_at(str | None) -
usage(TokenUsage | None) -
occurrence(StrictInt)
TokenUsage
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
Tokens a single agent step consumed, as the source reported them.
Every field is optional and independently so, because a source may report a total without its parts (or the reverse), and a normalizer must never compute one from the others; a derived total is indistinguishable from a reported one once it is stored, and the two disagree whenever a provider counts cached or reasoning tokens outside the pair. Absent means not reported, never zero.
Config:
frozen:Trueextra:forbid
Fields:
-
input_tokens(int | None) -
output_tokens(int | None) -
total_tokens(int | None)
ToolCall
pydantic-model
¶
Trace
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
The central object.
An owned trace carries a populated spans tree; a probed trace is
input/output only with spans == [] (and an endpoint_fingerprint). frame,
ownership, and taxonomy_cell are first-class queryable dimensions.
tool_calls live ONLY on Span; Trace deliberately has no
tool_calls field. A single-agent owned trace is therefore represented as a
one-span tree: a single root Span (parent=None, seq=0) that holds the
agent's tool_calls (see fixtures/normalized/ORD-1001.json). The methodology
plane reads tool calls from exactly one place (span.tool_calls) for both single-
and multi-agent traces; no branching on agent architecture (target-agnosticism).
The Trace-level input/output remain the terminal request/decision; the
root span's input/output mirror them for a single-agent trace.
Config:
frozen:Trueextra:forbid
Fields:
-
trace_id(str) -
frame(Frame) -
ownership(Ownership) -
input(dict[str, Any]) -
output(dict[str, Any]) -
taxonomy_cell(dict[str, str]) -
spans(list[Span]) -
captured_at(str | None) -
endpoint_fingerprint(str | None) -
session_id(str | None) -
schema_version(str)
label_from_dict ¶
Rebuild a Label from the dict to_dict produced.
A document written before rater_id existed is unattributed, not malformed: the field
defaults to None.
Source code in src/calibrated_agent_evaluation/schema/serde.py
latest_verdicts ¶
Keep one machine verdict per target, failure mode and rater: the newest. Order is kept.
labels must be oldest first, which is the order get_scores returns. A judge re-run
writes a new verdict for every target it already scored, and a live model words its critique
differently each time, so dedup_labels keeps both rows. Counting both doubles n with
no new evidence, which narrows every interval and can turn a failing gate green.
Only LLM-judge and code-scorer verdicts with a rater_id are collapsed, keyed on
(trace_id, span_id, kind, failure_mode, rater_id): that rater id names the judge and its
exact version, so two such rows are the same judgment made twice. Human labels and verdicts
with no recorded rater pass through unchanged, since two of them on one target may be
different people or different judges.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
labels
|
Iterable[Label]
|
Labels in write order, oldest first. |
required |
Returns:
| Type | Description |
|---|---|
list[Label]
|
The labels with every superseded machine verdict removed. |
Source code in src/calibrated_agent_evaluation/schema/serde.py
span_from_dict ¶
to_dict ¶
Return a JSON-serializable dict for a normalized artifact (nested objects included).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
artifact
|
Trace | Span | Label | ToolCall
|
The artifact to convert. |
required |
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
The artifact as plain dicts and lists. |
Source code in src/calibrated_agent_evaluation/schema/serde.py
trace_from_dict ¶
Rebuild a Trace from the dict to_dict produced.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
dict[str, Any]
|
The dict form of a trace. |
required |
Returns:
| Type | Description |
|---|---|
Trace
|
The trace. |
Raises:
| Type | Description |
|---|---|
SchemaError
|
The dict carries no |