calibrated_agent_evaluation.ingest¶
calibrated_agent_evaluation.ingest ¶
Trace ingestion & capture.
The first-class input boundary: turn an agent run you captured, a directory of trace dumps, or a Langfuse project into normalized Traces in the store; with no bespoke glue and no silent loss.
Public surface is re-exported here so callers write
from calibrated_agent_evaluation.ingest import run_ingest. Vendor SDKs (OpenTelemetry, Langfuse)
are reached only through the allowlisted execute/ and backends/ planes; nothing under
ingest/ imports a vendor type.
IngestReport
dataclass
¶
The aggregate of an ingest invocation; its exit_code is the process code.
exit_code
property
¶
0 iff every source ingested (or this was a --dry-run preview); else 1.
A dry run writes nothing and is a preview, so it never fails the process; a real run exits non-zero if any source was skipped or refused; never a silent exit-zero on a drop.
probed
property
¶
Ingested sources that degraded to outcome-only (ownership="probed").
A subset of ingested, not a fourth status: the trace parsed, passed every gate, and
was persisted faithfully; what degraded is how much of it there was to persist. Counted
here so the aggregate header can say so, because ownership was previously visible only
on the per-source lines and in a logger.warning nobody reading the report sees.
IngestResult
dataclass
¶
IngestResult(
source,
status,
adapter=None,
span_count=0,
tool_call_count=0,
ownership=None,
trace_id=None,
reason=None,
contentless_node_count=0,
unrecovered_tool_spans=0,
)
The per-source outcome the recovery summary reports.
IngestAbortedError ¶
Bases: IngestError
The store failed part-way through a batch, so the rest of the batch was not ingested.
The traces written before the failure stay written. written names them, so the caller can
tell what is in the store without re-reading it; the message names the source being ingested
when the store failed.
Source code in src/calibrated_agent_evaluation/ingest/errors.py
IngestError ¶
Bases: CalibratedAgentEvaluationDataError
An ingest source would lose data, or cannot be ingested faithfully.
Raised (never swallowed) when a normalized Trace would drop tool calls the source carried, when a source has duplicate span ids, or when a captured agent raises or emits no spans. The message names the offending source and the fix.
SampleRecord
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
How a sampled ingest drew its traces, so the draw can be repeated exactly.
population_size is how many candidates the draw chose from, sample_size how many it
kept, and the seed rides on provenance.seed. drawn names the population members the
draw selected (normalized trace ids for a store pull, source paths for files); trace_ids
names the traces that then landed in the store, which is shorter than drawn exactly when a
sampled source was refused or skipped, and the ingest report says which. cell is the
taxonomy cell the population was restricted to, empty when the whole population was in scope,
and
excluded_unknown_cell counts candidates left out of that stratum because their cell was
unknown. frame is the frame the sampled traces were stored with: a seeded simple random
sample of the population is production-representative, which is the point of drawing one.
Config:
frozen:Trueextra:forbid
Fields:
-
sample_id(str) -
source(str) -
population_size(int) -
sample_size(int) -
drawn(tuple[str, ...]) -
trace_ids(tuple[str, ...]) -
frame(Frame) -
cell(dict[str, str]) -
excluded_unknown_cell(int) -
schema_version(str) -
provenance(ProvenanceStamp)
capture_and_ingest ¶
capture_and_ingest(
agent,
*,
service_name,
trace_id=None,
backend,
dry_run=False,
register_global=True,
require_owned=False,
overwrite=False,
agent_label=None,
)
Run an in-process agent under OTel capture, then ingest the captured trace.
Captures the spans agent() emits in this process (a subprocess cannot share the
in-memory exporter, so capture is in-process; an instrumented callable, like 's executor),
encodes canonical OTLP-JSON, and runs it through the same file-ingest path (detect → gate →
persist) so the no-silent-loss guarantee applies. If the agent emits no span, raises
IngestError rather than persisting an empty trace. Needs the [strands]
extra for the OTel SDK; its absence is a typed error naming the extra.
trace_id is optional and defaults to the captured OTel trace id, unique per run by
construction. It used to default (via the CLI) to --service, which itself defaults to the
constant "agent"; so two captures with no explicit ids resolved to the same trace id, and
the second silently replaced the first under INSERT OR REPLACE while both runs reported
1 ingested and exit 0.
overwrite must be set to replace a trace id the store already holds. Without it, an explicit
trace_id that already exists is refused rather than clobbering the earlier capture; an id
collision is at least as likely to be a mistake as an intention, and only the caller knows.
An agent that raises is reported as an IngestError naming agent_label (the CLI
passes the --agent spec; it defaults to the callable's qualified name), chained from the
agent's own exception, and nothing is persisted: a run that died part-way is not a trace of the
agent's behaviour, and persisting the spans it left would read as a complete run.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
agent
|
Callable[[], object]
|
A no-argument callable that runs the agent once in this process. |
required |
service_name
|
str
|
The OTel service name the capture records spans under. |
required |
trace_id
|
str | None
|
The trace id to store; defaults to the captured OTel trace id. |
None
|
backend
|
TraceStoreBackend
|
The store to write to. |
required |
dry_run
|
bool
|
Report what would happen without writing. |
False
|
register_global
|
bool
|
Install the capture's tracer provider as the global one. |
True
|
require_owned
|
bool
|
Refuse a capture that degrades to |
False
|
overwrite
|
bool
|
Replace a trace the store already holds under |
False
|
agent_label
|
str | None
|
The name to use for the agent in error messages. |
None
|
Returns:
| Type | Description |
|---|---|
IngestReport
|
The ingest report for the captured trace. |
Raises:
| Type | Description |
|---|---|
IngestError
|
The agent raised, emitted no spans, the OTel SDK is missing, the trace id is
already stored without |
Source code in src/calibrated_agent_evaluation/ingest/core.py
1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 | |
contentless_node_count ¶
Count normalized spans that came back with nothing a judge or labeler could read.
The complement of recovered_content_count, named and exposed because the absence is
what gets reported: a tree of correct agents and correctly-named tool calls with {} in every
content field is structurally valid, passes the tool-count gate, and is worthless to read. This
figure is the one thing in the recovery summary that says so.
Source code in src/calibrated_agent_evaluation/ingest/core.py
ingest_langfuse ¶
ingest_langfuse(
source,
*,
backend,
dry_run=False,
require_owned=False,
overwrite=False,
allow_partial_tools=False,
sample_size=None,
sample_seed=None,
sample_id="sample",
cell=None,
source_label="langfuse",
source_kind="langfuse",
frame=None,
)
Pull every trace from a Langfuse source into the destination backend.
A source that implements TraceSource
(LangfuseBackend does) is read with pull, which returns traces Calibrated Agent
Evaluation wrote as normalized Traces and traces anything else wrote as OTLP-JSON. The
foreign ones go through the same path as a file (detect the convention, normalize, the
no-silent-loss gates), so they are ingested, refused or skipped exactly as the same spans in a
file would be. A trace that cannot be read is skipped with its reason and the pull carries
on; it used to raise out of query and abort the pull on the first foreign trace.
Any other backend is read with query, its own traces only.
overwrite and allow_partial_tools mean what they mean for run_ingest.
source_kind names where each trace came from in its result's source
(<source_kind>:<trace id>), and is the adapter of a trace the source could not hand
over; a source that reuses this function, such as AgentCore, passes its own.
sample_size + sample_seed ingest a seeded random sample of the pulled population
instead of all of it, tag the sampled traces production-representative and return the
draw as the report's SampleRecord. cell
restricts the population to traces in that taxonomy cell, which makes the draw a stratified one;
a pulled trace whose cell is unknown (a foreign export carries none) cannot be in the stratum,
so it is left out of the population and counted on the record. frame stamps every pulled
trace with that sampling frame, for a caller that knows how the source's traces were drawn (an
AgentCore online config that scores every session, or a random share of them, reads a random
sample of the agent's traffic); a sample is always production-representative, so frame
is refused together with sample_size.
The source's listing raises BackendError before any trace is yielded when it is
unreachable, so nothing is reported ingested. Vendor-free here: this speaks the
backend protocols and plain OTLP-JSON dicts, never a Langfuse type.
Source code in src/calibrated_agent_evaluation/ingest/core.py
1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 | |
probed_refusal_error ¶
Build the --require-owned refusal for a trace that degraded to outcome-only.
Not a no-silent-loss refusal: a probed trace is a faithful record of an under-instrumented
source, and refusing it by default would reject legitimate outcome-only data. It is refused only
when the caller has declared that step-level data is a requirement; which is the honest place
for that judgement, because whether outcome-only traces are usable depends on what the corpus is
for, not on the trace.
Source code in src/calibrated_agent_evaluation/ingest/core.py
recovered_content_count ¶
Count normalized spans that came back carrying content (per-step I/O or tool args/result).
Source code in src/calibrated_agent_evaluation/ingest/core.py
run_ingest ¶
run_ingest(
paths,
*,
backend,
fmt="auto",
dry_run=False,
require_owned=False,
trace_id_override=None,
convention="auto",
overwrite=False,
allow_partial_tools=False,
frame="discovery",
sample_size=None,
sample_seed=None,
sample_id="sample",
)
Ingest each trace file: detect → normalize → gate → persist (006/011; dry-run).
A source that can't be parsed or matched is recorded as skipped (never aborts the batch,
never a silent drop -); a lossy source is refused. Only a faithfully
persisted source is ingested; the report's exit_code is non-zero if any other status.
Backend-agnostic, so the same call persists to local or Langfuse identically.
trace_id_override forces the trace id for a single-source call, bypassing the
filename-stem default (capture_and_ingest uses it to pass the captured OTel trace id).
convention declares the OTel GenAI convention to read OTLP and dataset sources with
(calibrated-agent-evaluation ingest --convention), for a trace whose signals match more than
one; "auto" detects it.
require_owned refuses a source that degrades to ownership="probed" instead of persisting
it; for a corpus that will be analyzed step by step, where an outcome-only trace is not a
smaller result but an unusable one. Off by default, because an outcome-only trace is a faithful
record of an under-instrumented agent and refusing it unasked would reject real data.
A trace id the store already holds with different content is refused unless overwrite is
set; re-ingesting an identical trace is a no-op success. A trace that recovered some but not
all of its source tool calls is refused unless allow_partial_tools is set, and then reports
the shortfall in unrecovered_tool_spans.
frame is the sampling frame for a source whose spans carry no
calibrated_agent_evaluation.frame attribute. It defaults to "discovery", the frame that
authorizes no headline rate, because a file says nothing about how its traces were sampled;
"production-representative" is a claim the caller makes, never one inherited by omission.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
paths
|
Iterable[Path]
|
The trace files or directories to ingest. |
required |
backend
|
TraceStoreBackend
|
The store to write to. |
required |
fmt
|
str
|
The source format, or |
'auto'
|
dry_run
|
bool
|
Report what would happen without writing. |
False
|
require_owned
|
bool
|
Refuse a source that degrades to |
False
|
trace_id_override
|
str | None
|
The trace id to use for a single source. |
None
|
convention
|
str
|
The OTel GenAI convention to read with, or |
'auto'
|
overwrite
|
bool
|
Replace a stored trace whose id a source resolves to. |
False
|
allow_partial_tools
|
bool
|
Ingest a trace that recovered only some of its tool calls. |
False
|
frame
|
Frame
|
The sampling frame for sources that do not declare one. |
'discovery'
|
sample_size
|
int | None
|
Ingest a seeded random sample of this many sources instead of all of them. |
None
|
sample_seed
|
int | None
|
The seed for that draw; required with |
None
|
sample_id
|
str
|
The id the written
|
'sample'
|
Returns:
| Type | Description |
|---|---|
IngestReport
|
One result per source, and the exit code for the batch. |
Raises:
| Type | Description |
|---|---|
IngestError
|
Two sources in the batch resolve to the same trace id, or the sample arguments are incomplete or ask for more sources than the population holds. |
Source code in src/calibrated_agent_evaluation/ingest/core.py
943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 | |
source_content_span_count ¶
Count source spans that carry recoverable per-step content, in ANY known placement.
The content analogue of source_signals' tool-span count, and computed on the source for
the same reason: it is the only way to tell "the adapter dropped it" from "the run had none".
Counts a span that carries any content-bearing legacy event, or a non-empty value on any key a
registered placement family reads
(content_placements); deliberately
placement-agnostic, because the whole point is to compare what was on the wire against what came
back regardless of where it rode. The keys come from the registry rather than a list here so the
source side of the comparison can never know fewer placements than the reader does.
An empty list or object ("[]", "{}") is the key's presence, not content: counting it
refused a source with nothing in any placement, blaming a placement the adapter did not read.
Source code in src/calibrated_agent_evaluation/ingest/core.py
source_signals ¶
From a raw OTLP-JSON payload, return (tool_execution_span_count, duplicate_span_labels).
Computed on the SOURCE, independent of normalization, so the gate can distinguish "the source
had tool spans but normalize recovered none" (the silent-drop failure mode) from "the agent
genuinely made no tool calls" (a legitimate zero-tool trace). A tool-execution span is one whose
gen_ai.operation.name is execute_tool or that carries a gen_ai.tool.name attribute.
The OpenInference and legacy OpenLLMetry conventions emit no tool span at all; their calls are indexed attributes on an LLM span. For those the count is a floor: the most distinct calls any one span lists (an id counts once, an id-less entry once each), which no faithful reconstruction can come back under, whatever history the other spans resend.
Source code in src/calibrated_agent_evaluation/ingest/core.py
verify_no_silent_loss ¶
verify_no_silent_loss(
source_tool_span_count,
dup_span_names,
trace,
*,
source_content_spans=0,
recovery=None,
)
Refuse a trace that would silently lose data; no-op on a faithful one.
Raises IngestError when the source has duplicate span ids, when the source
carried tool-execution spans but the normalized trace recovered zero tool calls,
or when the source carried per-step content and none of it survived ('s content half;
a trace whose tools are all present by name with empty args and results passes the count
check while having lost everything a judge would read). A genuinely tool-free or content-free
source is never refused: both gates compare the source against the result, so "nothing was
there" and "something was dropped" stay different claims.
source_content_spans defaults to 0, i.e. the content gate is off unless the caller measured
the source; a caller that cannot measure it must not have its trace refused for that.
The content gate reads the recovered side through the trace's own ownership, because a
probed trace has spans == [] by definition and keeps its content in the terminal
input/output envelope instead. Counting only span content refused every
legitimately-degraded trace as a placement mismatch; wrong (nothing was lost, and the
degradation had already been announced with a reason) and unactionable (there is no placement
to go and fix). Whether an outcome-only trace is usable is the caller's call, taken through
require_owned.
The tool gate reads recovery when the caller supplies it, so it measures what the ADAPTER
linked rather than what survived the ownership collapse. The loss gate and the degrade rule ask
different
questions of the same probed, spans == [] trace; "did the adapter fail to link these
tool spans?" (refuse) versus "was every node it linked empty?" (degrade, and explicitly do not
refuse); and the finished Trace cannot tell them apart, because both look identical once the
spans are gone. Without recovery the gate falls back to the post-collapse count, which is
the conservative reading for a caller that did not measure.
Source code in src/calibrated_agent_evaluation/ingest/core.py
detect_source ¶
Return the normalizer source descriptor for path.
fmt forces a format (strands / otel / inspect); auto (default) detects it.
trace_id overrides the filename-stem default (the batch collision resolver passes one).
Raises IngestError for an unrecognized shape; the caller records that as a skipped
source, never a silent drop.
frame applies to a source whose spans carry no calibrated_agent_evaluation.frame; it
defaults to "discovery" because a file says nothing about how its traces were sampled.
Source code in src/calibrated_agent_evaluation/ingest/detect.py
draw_sample ¶
Draw size ids from population without replacement, reproducibly.
The population is sorted before the draw, so the same ids and seed give the same sample whatever order the store listed them in. The returned ids are in sorted order too.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
population
|
Sequence[str]
|
Every candidate id, in any order; duplicates are collapsed. |
required |
size
|
int
|
How many to draw. |
required |
seed
|
int
|
The seed; the same seed and population always draw the same sample. |
required |
Returns:
| Type | Description |
|---|---|
list[str]
|
The drawn ids, sorted. |
Raises:
| Type | Description |
|---|---|
IngestError
|
|
Source code in src/calibrated_agent_evaluation/ingest/sample.py
dump_samples ¶
Write records to sample_record.{csv,json} under path.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
records
|
Sequence[SampleRecord]
|
The sample records to persist. |
required |
path
|
str | Path
|
The directory to write into. |
required |
fmt
|
str
|
Which file a later |
'csv'
|
Raises:
| Type | Description |
|---|---|
IngestError
|
A bad |
Source code in src/calibrated_agent_evaluation/ingest/sample.py
load_samples ¶
Read sample records back from path; a missing file is an empty store.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str | Path
|
The directory |
required |
fmt
|
str
|
|
'csv'
|
Returns:
| Type | Description |
|---|---|
list[SampleRecord]
|
The stored records. |
Raises:
| Type | Description |
|---|---|
IngestError
|
A schema-version mismatch, or any read, parse or validation failure. |