Skip to content

calibrated_agent_evaluation.ingest

calibrated_agent_evaluation.ingest

Trace ingestion & capture.

The first-class input boundary: turn an agent run you captured, a directory of trace dumps, or a Langfuse project into normalized Traces in the store; with no bespoke glue and no silent loss.

Public surface is re-exported here so callers write from calibrated_agent_evaluation.ingest import run_ingest. Vendor SDKs (OpenTelemetry, Langfuse) are reached only through the allowlisted execute/ and backends/ planes; nothing under ingest/ imports a vendor type.

IngestStatus module-attribute

IngestStatus = Literal['ingested', 'skipped', 'refused']

IngestReport dataclass

IngestReport(results=list(), dry_run=False, sample=None)

The aggregate of an ingest invocation; its exit_code is the process code.

results class-attribute instance-attribute

results = field(default_factory=list)

dry_run class-attribute instance-attribute

dry_run = False

sample class-attribute instance-attribute

sample = None

exit_code property

exit_code

0 iff every source ingested (or this was a --dry-run preview); else 1.

A dry run writes nothing and is a preview, so it never fails the process; a real run exits non-zero if any source was skipped or refused; never a silent exit-zero on a drop.

ingested property

ingested

How many inputs were ingested.

refused property

refused

How many inputs were refused.

skipped property

skipped

How many inputs were skipped.

probed property

probed

Ingested sources that degraded to outcome-only (ownership="probed").

A subset of ingested, not a fourth status: the trace parsed, passed every gate, and was persisted faithfully; what degraded is how much of it there was to persist. Counted here so the aggregate header can say so, because ownership was previously visible only on the per-source lines and in a logger.warning nobody reading the report sees.

IngestResult dataclass

IngestResult(
    source,
    status,
    adapter=None,
    span_count=0,
    tool_call_count=0,
    ownership=None,
    trace_id=None,
    reason=None,
    contentless_node_count=0,
    unrecovered_tool_spans=0,
)

The per-source outcome the recovery summary reports.

source instance-attribute

source

status instance-attribute

status

adapter class-attribute instance-attribute

adapter = None

span_count class-attribute instance-attribute

span_count = 0

tool_call_count class-attribute instance-attribute

tool_call_count = 0

ownership class-attribute instance-attribute

ownership = None

trace_id class-attribute instance-attribute

trace_id = None

reason class-attribute instance-attribute

reason = None

contentless_node_count class-attribute instance-attribute

contentless_node_count = 0

unrecovered_tool_spans class-attribute instance-attribute

unrecovered_tool_spans = 0

IngestAbortedError

IngestAbortedError(message, *, written)

Bases: IngestError

The store failed part-way through a batch, so the rest of the batch was not ingested.

The traces written before the failure stay written. written names them, so the caller can tell what is in the store without re-reading it; the message names the source being ingested when the store failed.

Source code in src/calibrated_agent_evaluation/ingest/errors.py
def __init__(self, message: str, *, written: tuple[str, ...]) -> None:
    super().__init__(message)
    self.written = written

written instance-attribute

written = written

IngestError

Bases: CalibratedAgentEvaluationDataError

An ingest source would lose data, or cannot be ingested faithfully.

Raised (never swallowed) when a normalized Trace would drop tool calls the source carried, when a source has duplicate span ids, or when a captured agent raises or emits no spans. The message names the offending source and the fix.

SampleRecord pydantic-model

Bases: CalibratedAgentEvaluationModel

How a sampled ingest drew its traces, so the draw can be repeated exactly.

population_size is how many candidates the draw chose from, sample_size how many it kept, and the seed rides on provenance.seed. drawn names the population members the draw selected (normalized trace ids for a store pull, source paths for files); trace_ids names the traces that then landed in the store, which is shorter than drawn exactly when a sampled source was refused or skipped, and the ingest report says which. cell is the taxonomy cell the population was restricted to, empty when the whole population was in scope, and excluded_unknown_cell counts candidates left out of that stratum because their cell was unknown. frame is the frame the sampled traces were stored with: a seeded simple random sample of the population is production-representative, which is the point of drawing one.

Config:

  • frozen: True
  • extra: forbid

Fields:

sample_id pydantic-field

sample_id

source pydantic-field

source

population_size pydantic-field

population_size

sample_size pydantic-field

sample_size

drawn pydantic-field

drawn

trace_ids pydantic-field

trace_ids

frame pydantic-field

frame = 'production-representative'

cell pydantic-field

cell

excluded_unknown_cell pydantic-field

excluded_unknown_cell = 0

schema_version pydantic-field

schema_version = SAMPLE_SCHEMA_VERSION

provenance pydantic-field

provenance

capture_and_ingest

capture_and_ingest(
    agent,
    *,
    service_name,
    trace_id=None,
    backend,
    dry_run=False,
    register_global=True,
    require_owned=False,
    overwrite=False,
    agent_label=None,
)

Run an in-process agent under OTel capture, then ingest the captured trace.

Captures the spans agent() emits in this process (a subprocess cannot share the in-memory exporter, so capture is in-process; an instrumented callable, like 's executor), encodes canonical OTLP-JSON, and runs it through the same file-ingest path (detect → gate → persist) so the no-silent-loss guarantee applies. If the agent emits no span, raises IngestError rather than persisting an empty trace. Needs the [strands] extra for the OTel SDK; its absence is a typed error naming the extra.

trace_id is optional and defaults to the captured OTel trace id, unique per run by construction. It used to default (via the CLI) to --service, which itself defaults to the constant "agent"; so two captures with no explicit ids resolved to the same trace id, and the second silently replaced the first under INSERT OR REPLACE while both runs reported 1 ingested and exit 0.

overwrite must be set to replace a trace id the store already holds. Without it, an explicit trace_id that already exists is refused rather than clobbering the earlier capture; an id collision is at least as likely to be a mistake as an intention, and only the caller knows.

An agent that raises is reported as an IngestError naming agent_label (the CLI passes the --agent spec; it defaults to the callable's qualified name), chained from the agent's own exception, and nothing is persisted: a run that died part-way is not a trace of the agent's behaviour, and persisting the spans it left would read as a complete run.

Parameters:

Name Type Description Default
agent Callable[[], object]

A no-argument callable that runs the agent once in this process.

required
service_name str

The OTel service name the capture records spans under.

required
trace_id str | None

The trace id to store; defaults to the captured OTel trace id.

None
backend TraceStoreBackend

The store to write to.

required
dry_run bool

Report what would happen without writing.

False
register_global bool

Install the capture's tracer provider as the global one.

True
require_owned bool

Refuse a capture that degrades to probed.

False
overwrite bool

Replace a trace the store already holds under trace_id.

False
agent_label str | None

The name to use for the agent in error messages.

None

Returns:

Type Description
IngestReport

The ingest report for the captured trace.

Raises:

Type Description
IngestError

The agent raised, emitted no spans, the OTel SDK is missing, the trace id is already stored without overwrite, or the store can't be read to check it.

Source code in src/calibrated_agent_evaluation/ingest/core.py
def capture_and_ingest(
    agent: Callable[[], object],
    *,
    service_name: str,
    trace_id: str | None = None,
    backend: TraceStoreBackend,
    dry_run: bool = False,
    register_global: bool = True,
    require_owned: bool = False,
    overwrite: bool = False,
    agent_label: str | None = None,
) -> IngestReport:
    """Run an in-process agent under OTel capture, then ingest the captured trace.

    Captures the spans ``agent()`` emits in **this** process (a subprocess cannot share the
    in-memory exporter, so capture is in-process; an instrumented callable, like 's executor),
    encodes canonical OTLP-JSON, and runs it through the same file-ingest path (detect → gate →
    persist) so the no-silent-loss guarantee applies. If the agent emits no span, raises
    :class:`IngestError` rather than persisting an empty trace. Needs the ``[strands]``
    extra for the OTel SDK; its absence is a typed error naming the extra.

    ``trace_id`` is optional and defaults to the **captured OTel trace id**, unique per run by
    construction. It used to default (via the CLI) to ``--service``, which itself defaults to the
    constant ``"agent"``; so two captures with no explicit ids resolved to the same trace id, and
    the second silently replaced the first under ``INSERT OR REPLACE`` while both runs reported
    ``1 ingested`` and exit 0.

    ``overwrite`` must be set to replace a trace id the store already holds. Without it, an explicit
    ``trace_id`` that already exists is refused rather than clobbering the earlier capture; an id
    collision is at least as likely to be a mistake as an intention, and only the caller knows.

    An agent that raises is reported as an :class:`IngestError` naming ``agent_label`` (the CLI
    passes the ``--agent`` spec; it defaults to the callable's qualified name), chained from the
    agent's own exception, and nothing is persisted: a run that died part-way is not a trace of the
    agent's behaviour, and persisting the spans it left would read as a complete run.

    Args:
        agent: A no-argument callable that runs the agent once in this process.
        service_name: The OTel service name the capture records spans under.
        trace_id: The trace id to store; defaults to the captured OTel trace id.
        backend: The store to write to.
        dry_run: Report what would happen without writing.
        register_global: Install the capture's tracer provider as the global one.
        require_owned: Refuse a capture that degrades to ``probed``.
        overwrite: Replace a trace the store already holds under ``trace_id``.
        agent_label: The name to use for the agent in error messages.

    Returns:
        The ingest report for the captured trace.

    Raises:
        IngestError: The agent raised, emitted no spans, the OTel SDK is missing, the trace id is
            already stored without ``overwrite``, or the store can't be read to check it.
    """
    try:
        from calibrated_agent_evaluation.execute.strands_capture import (
            CaptureProviderError,
            capture_spans,
            write_otlp_json,
        )
    except ImportError as exc:  # missing extra → typed, actionable
        # Name the module that failed. The bare "install the extra" hint was all a user saw when
        # the extra itself lacked a dependency, so they reinstalled what they had.
        missing = exc.name or str(exc)
        raise IngestError(
            "calibrated-agent-evaluation capture needs the OTel SDK and OTLP encoder (cannot "
            f"import {missing!r}); "
            f"{install_hint('strands')}"
        ) from exc

    label = agent_label or getattr(agent, "__qualname__", None) or repr(agent)
    with capture_spans(service_name) as cap:
        if register_global:
            try:
                cap.register_global()
            except CaptureProviderError as exc:
                raise IngestError(f"cannot capture {label!r}: {exc}") from exc
        try:
            agent()
        except Exception as exc:  # the user's code; any failure is reported, never swallowed
            raise IngestError(
                f"agent {label!r} raised {type(exc).__name__}: {exc}; the run did not finish, so "
                "nothing was persisted"
            ) from exc
    # Read only after the `with` exits. `SpanCapture.__exit__` force-flushes the provider, and a
    # span that ends during that flush (an instrumentation closing its span from a flush hook, or a
    # buffering processor draining) reaches the exporter only then. Reading inside the block
    # dropped those spans and reported the capture as complete.
    spans = cap.finished_spans
    if not spans:  # a broken/uninstrumented agent that emitted nothing
        raise IngestError(
            f"captured 0 spans from {label!r}: the agent emitted no OpenTelemetry spans "
            "(is it instrumented? did it exit before a span closed?), so nothing was stored"
        )
    with tempfile.TemporaryDirectory(prefix="calibrated-agent-evaluation-capture-") as tmp:
        # The file name seeds the trace id (see `detect._trace_id_for`), so an explicit id rides
        # through the name and an absent one leaves the adapter to read the captured OTel traceId.
        stem = trace_id or "calibrated-agent-evaluation-capture"
        otlp_path = Path(tmp) / f"{stem}.otlp.json"
        write_otlp_json(spans, otlp_path)
        resolved = trace_id or wire_trace_id(otlp_path)
        if resolved is not None and not dry_run and not overwrite:
            _refuse_existing_trace(backend, resolved, explicit=trace_id is not None)
        return run_ingest(
            [otlp_path],
            backend=backend,
            dry_run=dry_run,
            require_owned=require_owned,
            trace_id_override=None if trace_id else resolved,
            # `_refuse_existing_trace` above already ran the stricter capture check (a capture
            # refuses even an identical stored trace unless told to replace it).
            overwrite=True,
        )

contentless_node_count

contentless_node_count(trace)

Count normalized spans that came back with nothing a judge or labeler could read.

The complement of recovered_content_count, named and exposed because the absence is what gets reported: a tree of correct agents and correctly-named tool calls with {} in every content field is structurally valid, passes the tool-count gate, and is worthless to read. This figure is the one thing in the recovery summary that says so.

Source code in src/calibrated_agent_evaluation/ingest/core.py
def contentless_node_count(trace: Trace) -> int:
    """Count normalized spans that came back with nothing a judge or labeler could read.

    The complement of :func:`recovered_content_count`, named and exposed because the *absence* is
    what gets reported: a tree of correct agents and correctly-named tool calls with ``{}`` in every
    content field is structurally valid, passes the tool-count gate, and is worthless to read. This
    figure is the one thing in the recovery summary that says so.
    """
    return len(trace.spans) - recovered_content_count(trace)

ingest_langfuse

ingest_langfuse(
    source,
    *,
    backend,
    dry_run=False,
    require_owned=False,
    overwrite=False,
    allow_partial_tools=False,
    sample_size=None,
    sample_seed=None,
    sample_id="sample",
    cell=None,
    source_label="langfuse",
    source_kind="langfuse",
    frame=None,
)

Pull every trace from a Langfuse source into the destination backend.

A source that implements TraceSource (LangfuseBackend does) is read with pull, which returns traces Calibrated Agent Evaluation wrote as normalized Traces and traces anything else wrote as OTLP-JSON. The foreign ones go through the same path as a file (detect the convention, normalize, the no-silent-loss gates), so they are ingested, refused or skipped exactly as the same spans in a file would be. A trace that cannot be read is skipped with its reason and the pull carries on; it used to raise out of query and abort the pull on the first foreign trace.

Any other backend is read with query, its own traces only.

overwrite and allow_partial_tools mean what they mean for run_ingest. source_kind names where each trace came from in its result's source (<source_kind>:<trace id>), and is the adapter of a trace the source could not hand over; a source that reuses this function, such as AgentCore, passes its own.

sample_size + sample_seed ingest a seeded random sample of the pulled population instead of all of it, tag the sampled traces production-representative and return the draw as the report's SampleRecord. cell restricts the population to traces in that taxonomy cell, which makes the draw a stratified one; a pulled trace whose cell is unknown (a foreign export carries none) cannot be in the stratum, so it is left out of the population and counted on the record. frame stamps every pulled trace with that sampling frame, for a caller that knows how the source's traces were drawn (an AgentCore online config that scores every session, or a random share of them, reads a random sample of the agent's traffic); a sample is always production-representative, so frame is refused together with sample_size.

The source's listing raises BackendError before any trace is yielded when it is unreachable, so nothing is reported ingested. Vendor-free here: this speaks the backend protocols and plain OTLP-JSON dicts, never a Langfuse type.

Source code in src/calibrated_agent_evaluation/ingest/core.py
def ingest_langfuse(
    source: TraceStoreBackend | TraceSource,
    *,
    backend: TraceStoreBackend,
    dry_run: bool = False,
    require_owned: bool = False,
    overwrite: bool = False,
    allow_partial_tools: bool = False,
    sample_size: int | None = None,
    sample_seed: int | None = None,
    sample_id: str = "sample",
    cell: dict[str, str] | None = None,
    source_label: str = "langfuse",
    source_kind: str = "langfuse",
    frame: Frame | None = None,
) -> IngestReport:
    """Pull every trace from a Langfuse source into the destination backend.

    A source that implements :class:`~calibrated_agent_evaluation.backends.base.TraceSource`
    (``LangfuseBackend`` does) is read with ``pull``, which returns traces Calibrated Agent
    Evaluation wrote as normalized ``Trace``s and traces anything else wrote as OTLP-JSON. The
    foreign ones go through the same path as a file (detect the convention, normalize, the
    no-silent-loss gates), so they are ingested, refused or skipped exactly as the same spans in a
    file would be. A trace that cannot be read is ``skipped`` with its reason and the pull carries
    on; it used to raise out of ``query`` and abort the pull on the first foreign trace.

    Any other backend is read with ``query``, its own traces only.

    ``overwrite`` and ``allow_partial_tools`` mean what they mean for :func:`run_ingest`.
    ``source_kind`` names where each trace came from in its result's ``source``
    (``<source_kind>:<trace id>``), and is the ``adapter`` of a trace the source could not hand
    over; a source that reuses this function, such as AgentCore, passes its own.

    ``sample_size`` + ``sample_seed`` ingest a seeded random sample of the pulled population
    instead of all of it, tag the sampled traces ``production-representative`` and return the
    draw as the report's :class:`~calibrated_agent_evaluation.ingest.sample.SampleRecord`. ``cell``
    restricts the population to traces in that taxonomy cell, which makes the draw a stratified one;
    a pulled trace whose cell is unknown (a foreign export carries none) cannot be in the stratum,
    so it is left out of the population and counted on the record. ``frame`` stamps every pulled
    trace with that sampling frame, for a caller that knows how the source's traces were drawn (an
    AgentCore online config that scores every session, or a random share of them, reads a random
    sample of the agent's traffic); a sample is always ``production-representative``, so ``frame``
    is refused together with ``sample_size``.

    The source's listing raises ``BackendError`` before any trace is yielded when it is
    unreachable, so nothing is reported ingested. Vendor-free here: this speaks the
    backend protocols and plain OTLP-JSON dicts, never a Langfuse type.
    """
    if frame is not None and sample_size is not None:
        raise IngestError(
            "a sampled ingest is always production-representative, so it takes no frame; pass "
            "either the sample size or the frame"
        )
    pulled: Iterable[SourceTrace]
    if isinstance(source, TraceSource):
        pulled = source.pull()
    else:
        pulled = (SourceTrace(source_id=t.trace_id, trace=t) for t in source.query())

    # Traces Calibrated Agent Evaluation wrote go first, so when a foreign export of the same run
    # resolves to the same trace id, the faithful copy is the one kept and the foreign one is
    # refused as a repeat.
    items = sorted(pulled, key=lambda item: item.trace is None)
    drawn: list[str] | None = None
    stamp_frame: Frame | None = frame
    excluded = 0
    population = len(items)
    if sample_size is not None:
        if sample_seed is None:
            raise IngestError(
                "a sample needs a seed so the draw can be repeated; pass the seed alongside the "
                "sample size"
            )
        if cell:
            eligible = [item for item in items if _item_in_cell(item, cell)]
            excluded = len(items) - len(eligible)
            items = eligible
        population = len(items)
        # Keyed on the normalized trace id where the source has one, so the record's draw names
        # the traces a reader can look up rather than the store's internal id for them.
        keyed = {_item_key(item): item for item in items}
        drawn = draw_sample(list(keyed), size=sample_size, seed=sample_seed)
        items = [keyed[key] for key in drawn]
        stamp_frame = "production-representative"
    claimed: dict[str, str] = {}
    results: list[IngestResult] = []
    with tempfile.TemporaryDirectory(prefix="calibrated-agent-evaluation-langfuse-") as tmp:
        for index, item in enumerate(items):
            label = f"{source_kind}:{item.source_id}"
            try:
                results.extend(
                    _ingest_pulled(
                        item,
                        label,
                        Path(tmp) / f"{index}.otlp.json",
                        backend=backend,
                        dry_run=dry_run,
                        require_owned=require_owned,
                        claimed=claimed,
                        overwrite=overwrite,
                        allow_partial_tools=allow_partial_tools,
                        stamp_frame=stamp_frame,
                        source_kind=source_kind,
                    )
                )
            except BackendError as exc:
                _raise_aborted(label, results, exc)
    return IngestReport(
        results=results,
        dry_run=dry_run,
        sample=_sample_record(
            sample_id,
            source=source_label,
            drawn=drawn,
            population_size=population,
            seed=sample_seed,
            results=results,
            cell=cell,
            excluded_unknown_cell=excluded,
        ),
    )

probed_refusal_error

probed_refusal_error(trace_id, reason=None)

Build the --require-owned refusal for a trace that degraded to outcome-only.

Not a no-silent-loss refusal: a probed trace is a faithful record of an under-instrumented source, and refusing it by default would reject legitimate outcome-only data. It is refused only when the caller has declared that step-level data is a requirement; which is the honest place for that judgement, because whether outcome-only traces are usable depends on what the corpus is for, not on the trace.

Source code in src/calibrated_agent_evaluation/ingest/core.py
def probed_refusal_error(trace_id: str, reason: str | None = None) -> IngestError:
    """Build the ``--require-owned`` refusal for a trace that degraded to outcome-only.

    Not a no-silent-loss refusal: a ``probed`` trace is a faithful record of an under-instrumented
    source, and refusing it by default would reject legitimate outcome-only data. It is refused only
    when the caller has declared that step-level data is a requirement; which is the honest place
    for that judgement, because whether outcome-only traces are usable depends on what the corpus is
    for, not on the trace.
    """
    detail = f" ({reason})" if reason else ""
    return IngestError(
        f"trace {trace_id!r} degraded to ownership='probed'{detail} and --require-owned was set; "
        "it carries terminal input/output only, with no per-step spans, so step-level attribution "
        "and per-agent rates have nothing to read; re-capture with tool/agent instrumentation, or "
        "drop --require-owned to accept outcome-only data"
    )

recovered_content_count

recovered_content_count(trace)

Count normalized spans that came back carrying content (per-step I/O or tool args/result).

Source code in src/calibrated_agent_evaluation/ingest/core.py
def recovered_content_count(trace: Trace) -> int:
    """Count normalized spans that came back carrying content (per-step I/O or tool args/result)."""
    return sum(
        1
        for span in trace.spans
        if span.input or span.output or any(call.args or call.result for call in span.tool_calls)
    )

run_ingest

run_ingest(
    paths,
    *,
    backend,
    fmt="auto",
    dry_run=False,
    require_owned=False,
    trace_id_override=None,
    convention="auto",
    overwrite=False,
    allow_partial_tools=False,
    frame="discovery",
    sample_size=None,
    sample_seed=None,
    sample_id="sample",
)

Ingest each trace file: detect → normalize → gate → persist (006/011; dry-run).

A source that can't be parsed or matched is recorded as skipped (never aborts the batch, never a silent drop -); a lossy source is refused. Only a faithfully persisted source is ingested; the report's exit_code is non-zero if any other status. Backend-agnostic, so the same call persists to local or Langfuse identically.

trace_id_override forces the trace id for a single-source call, bypassing the filename-stem default (capture_and_ingest uses it to pass the captured OTel trace id).

convention declares the OTel GenAI convention to read OTLP and dataset sources with (calibrated-agent-evaluation ingest --convention), for a trace whose signals match more than one; "auto" detects it.

require_owned refuses a source that degrades to ownership="probed" instead of persisting it; for a corpus that will be analyzed step by step, where an outcome-only trace is not a smaller result but an unusable one. Off by default, because an outcome-only trace is a faithful record of an under-instrumented agent and refusing it unasked would reject real data.

A trace id the store already holds with different content is refused unless overwrite is set; re-ingesting an identical trace is a no-op success. A trace that recovered some but not all of its source tool calls is refused unless allow_partial_tools is set, and then reports the shortfall in unrecovered_tool_spans.

frame is the sampling frame for a source whose spans carry no calibrated_agent_evaluation.frame attribute. It defaults to "discovery", the frame that authorizes no headline rate, because a file says nothing about how its traces were sampled; "production-representative" is a claim the caller makes, never one inherited by omission.

Parameters:

Name Type Description Default
paths Iterable[Path]

The trace files or directories to ingest.

required
backend TraceStoreBackend

The store to write to.

required
fmt str

The source format, or "auto" to detect it per file.

'auto'
dry_run bool

Report what would happen without writing.

False
require_owned bool

Refuse a source that degrades to probed.

False
trace_id_override str | None

The trace id to use for a single source.

None
convention str

The OTel GenAI convention to read with, or "auto".

'auto'
overwrite bool

Replace a stored trace whose id a source resolves to.

False
allow_partial_tools bool

Ingest a trace that recovered only some of its tool calls.

False
frame Frame

The sampling frame for sources that do not declare one.

'discovery'
sample_size int | None

Ingest a seeded random sample of this many sources instead of all of them.

None
sample_seed int | None

The seed for that draw; required with sample_size.

None
sample_id str

The id the written SampleRecord carries.

'sample'

Returns:

Type Description
IngestReport

One result per source, and the exit code for the batch.

Raises:

Type Description
IngestError

Two sources in the batch resolve to the same trace id, or the sample arguments are incomplete or ask for more sources than the population holds.

Source code in src/calibrated_agent_evaluation/ingest/core.py
def run_ingest(
    paths: Iterable[Path],
    *,
    backend: TraceStoreBackend,
    fmt: str = "auto",
    dry_run: bool = False,
    require_owned: bool = False,
    trace_id_override: str | None = None,
    convention: str = "auto",
    overwrite: bool = False,
    allow_partial_tools: bool = False,
    frame: Frame = "discovery",
    sample_size: int | None = None,
    sample_seed: int | None = None,
    sample_id: str = "sample",
) -> IngestReport:
    """Ingest each trace file: detect → normalize → gate → persist (006/011;  dry-run).

    A source that can't be parsed or matched is recorded as ``skipped`` (never aborts the batch,
    never a silent drop -); a lossy source is ``refused``. Only a faithfully
    persisted source is ``ingested``; the report's ``exit_code`` is non-zero if any other status.
    Backend-agnostic, so the same call persists to local or Langfuse identically.

    ``trace_id_override`` forces the trace id for a single-source call, bypassing the
    filename-stem default (``capture_and_ingest`` uses it to pass the captured OTel trace id).

    ``convention`` declares the OTel GenAI convention to read OTLP and dataset sources with
    (``calibrated-agent-evaluation ingest --convention``), for a trace whose signals match more than
    one; ``"auto"`` detects it.

    ``require_owned`` refuses a source that degrades to ``ownership="probed"`` instead of persisting
    it; for a corpus that will be analyzed step by step, where an outcome-only trace is not a
    smaller result but an unusable one. Off by default, because an outcome-only trace is a faithful
    record of an under-instrumented agent and refusing it unasked would reject real data.

    A trace id the store already holds with different content is refused unless ``overwrite`` is
    set; re-ingesting an identical trace is a no-op success. A trace that recovered some but not
    all of its source tool calls is refused unless ``allow_partial_tools`` is set, and then reports
    the shortfall in ``unrecovered_tool_spans``.

    ``frame`` is the sampling frame for a source whose spans carry no
    ``calibrated_agent_evaluation.frame`` attribute. It defaults to ``"discovery"``, the frame that
    authorizes no headline rate, because a file says nothing about how its traces were sampled;
    ``"production-representative"`` is a claim the caller makes, never one inherited by omission.

    Args:
        paths: The trace files or directories to ingest.
        backend: The store to write to.
        fmt: The source format, or ``"auto"`` to detect it per file.
        dry_run: Report what would happen without writing.
        require_owned: Refuse a source that degrades to ``probed``.
        trace_id_override: The trace id to use for a single source.
        convention: The OTel GenAI convention to read with, or ``"auto"``.
        overwrite: Replace a stored trace whose id a source resolves to.
        allow_partial_tools: Ingest a trace that recovered only some of its tool calls.
        frame: The sampling frame for sources that do not declare one.
        sample_size: Ingest a seeded random sample of this many sources instead of all of them.
        sample_seed: The seed for that draw; required with ``sample_size``.
        sample_id: The id the written
            :class:`~calibrated_agent_evaluation.ingest.sample.SampleRecord` carries.

    Returns:
        One result per source, and the exit code for the batch.

    Raises:
        IngestError: Two sources in the batch resolve to the same trace id, or the sample
            arguments are incomplete or ask for more sources than the population holds.
    """
    listed = list(paths)
    drawn: list[str] | None = None
    population = len(listed)
    stamp_frame: Frame | None = None
    if sample_size is not None:
        if sample_seed is None:
            raise IngestError(
                "a sample needs a seed so the draw can be repeated; pass the seed alongside the "
                "sample size"
            )
        drawn = draw_sample([str(path) for path in listed], size=sample_size, seed=sample_seed)
        keep = set(drawn)
        listed = [path for path in listed if str(path) in keep]
        # A seeded random sample of the population IS production-representative, whatever the
        # exported traces happen to be tagged.
        stamp_frame = "production-representative"
    if trace_id_override is not None:
        # Single-source path (`capture_and_ingest`): the caller already resolved the id, so the
        # filename-stem default and its collision machinery have nothing to contribute.
        if len(listed) != 1:
            raise IngestError(f"trace_id_override applies to exactly one source, got {len(listed)}")
        return IngestReport(
            results=_ingest_path(
                listed[0],
                backend=backend,
                fmt=fmt,
                dry_run=dry_run,
                require_owned=require_owned,
                trace_id=trace_id_override,
                convention=convention,
                overwrite=overwrite,
                allow_partial_tools=allow_partial_tools,
                frame=frame,
                stamp_frame=stamp_frame,
            ),
            dry_run=dry_run,
        )
    # A dataset's trace ids come from its records, not its file name, so datasets stay out of the
    # stem-collision resolver; `claimed` catches a record id that repeats across the batch.
    single = [path for path in listed if detect_dataset(path, fmt) is None]
    overrides, collisions = resolve_batch_trace_ids(single)
    colliding_paths = {src for sources in collisions.values() for src in sources}
    claimed: dict[str, str] = {}

    results: list[IngestResult] = []
    for path in listed:
        if str(path) in colliding_paths:
            tid = next(t for t, srcs in collisions.items() if str(path) in srcs)
            results.append(
                IngestResult(
                    source=str(path),
                    status="refused",
                    reason=str(trace_id_collision_error(tid, collisions[tid])),
                )
            )
            continue
        try:
            results.extend(
                _ingest_path(
                    path,
                    backend=backend,
                    fmt=fmt,
                    dry_run=dry_run,
                    require_owned=require_owned,
                    trace_id=overrides.get(path),
                    claimed=claimed,
                    convention=convention,
                    overwrite=overwrite,
                    allow_partial_tools=allow_partial_tools,
                    frame=frame,
                    stamp_frame=stamp_frame,
                )
            )
        except BackendError as exc:
            _raise_aborted(str(path), results, exc)
    return IngestReport(
        results=results,
        dry_run=dry_run,
        sample=_sample_record(
            sample_id,
            source="files",
            drawn=drawn,
            population_size=population,
            seed=sample_seed,
            results=results,
        ),
    )

source_content_span_count

source_content_span_count(otlp)

Count source spans that carry recoverable per-step content, in ANY known placement.

The content analogue of source_signals' tool-span count, and computed on the source for the same reason: it is the only way to tell "the adapter dropped it" from "the run had none". Counts a span that carries any content-bearing legacy event, or a non-empty value on any key a registered placement family reads (content_placements); deliberately placement-agnostic, because the whole point is to compare what was on the wire against what came back regardless of where it rode. The keys come from the registry rather than a list here so the source side of the comparison can never know fewer placements than the reader does.

An empty list or object ("[]", "{}") is the key's presence, not content: counting it refused a source with nothing in any placement, blaming a placement the adapter did not read.

Source code in src/calibrated_agent_evaluation/ingest/core.py
def source_content_span_count(otlp: dict[str, Any]) -> int:
    """Count source spans that carry recoverable per-step content, in ANY known placement.

    The content analogue of :func:`source_signals`' tool-span count, and computed on the source for
    the same reason: it is the only way to tell "the adapter dropped it" from "the run had none".
    Counts a span that carries any content-bearing legacy event, or a non-empty value on any key a
    registered placement family reads
    (:mod:`~calibrated_agent_evaluation.normalize.content_placements`); deliberately
    placement-agnostic, because the whole point is to compare what was on the wire against what came
    back regardless of where it rode. The keys come from the registry rather than a list here so the
    source side of the comparison can never know fewer placements than the reader does.

    An empty list or object (``"[]"``, ``"{}"``) is the key's presence, not content: counting it
    refused a source with nothing in any placement, blaming a placement the adapter did not read.
    """
    keys = tuple(
        dict.fromkeys(
            key for role in _CONTENT_ROLES for key in content_placements.placement_keys(role)
        )
    )
    events = (
        EVENT_USER_MESSAGE,
        EVENT_ASSISTANT_MESSAGE,
        EVENT_TOOL_MESSAGE,
        EVENT_CHOICE,
        EVENT_INFERENCE_DETAILS,
    )
    count = 0
    for span in iter_spans(otlp):
        if any(_has_payload(latest_content(span, key)) for key in keys):
            count += 1
            continue
        if any(event for name in events for event in event_attributes(span, name)):
            count += 1
    return count

source_signals

source_signals(otlp)

From a raw OTLP-JSON payload, return (tool_execution_span_count, duplicate_span_labels).

Computed on the SOURCE, independent of normalization, so the gate can distinguish "the source had tool spans but normalize recovered none" (the silent-drop failure mode) from "the agent genuinely made no tool calls" (a legitimate zero-tool trace). A tool-execution span is one whose gen_ai.operation.name is execute_tool or that carries a gen_ai.tool.name attribute.

The OpenInference and legacy OpenLLMetry conventions emit no tool span at all; their calls are indexed attributes on an LLM span. For those the count is a floor: the most distinct calls any one span lists (an id counts once, an id-less entry once each), which no faithful reconstruction can come back under, whatever history the other spans resend.

Source code in src/calibrated_agent_evaluation/ingest/core.py
def source_signals(otlp: dict[str, Any]) -> tuple[int, list[str]]:
    """From a raw OTLP-JSON payload, return ``(tool_execution_span_count, duplicate_span_labels)``.

    Computed on the SOURCE, independent of normalization, so the gate can distinguish "the source
    had tool spans but normalize recovered none" (the silent-drop failure mode) from "the agent
    genuinely made no tool calls" (a legitimate zero-tool trace). A tool-execution span is one whose
    ``gen_ai.operation.name`` is ``execute_tool`` or that carries a ``gen_ai.tool.name`` attribute.

    The OpenInference and legacy OpenLLMetry conventions emit no tool span at all; their calls are
    indexed attributes on an LLM span. For those the count is a floor: the most distinct calls any
    one span lists (an id counts once, an id-less entry once each), which no faithful
    reconstruction can come back under, whatever history the other spans resend.
    """
    spans = list(iter_spans(otlp))
    # A tool span that directly wraps an agent invocation is a sub-agent run as a tool, which the
    # adapters recover as an agent node rather than a tool call, so it is not counted as a tool.
    wraps_agent = {
        span.get("parentSpanId")
        for span in spans
        if attributes(span).get(ATTR_OP) == "invoke_agent" and span.get("parentSpanId")
    }
    tool_spans = 0
    seen: set[str] = set()
    dup_labels: dict[str, str] = {}
    for span in spans:
        attrs = attributes(span)
        is_tool = attrs.get(ATTR_OP) == "execute_tool" or ATTR_TOOL_NAME in attrs
        if is_tool and span.get("spanId") not in wraps_agent:
            tool_spans += 1
        span_id = span.get("spanId")
        if isinstance(span_id, str):
            if span_id in seen:
                label = attrs.get(ATTR_AGENT_NAME) or span.get("name") or span_id
                dup_labels[span_id] = str(label)
            seen.add(span_id)
    floor = max((_indexed_tool_call_count(attributes(span)) for span in spans), default=0)
    return max(tool_spans, floor), sorted(set(dup_labels.values()))

verify_no_silent_loss

verify_no_silent_loss(
    source_tool_span_count,
    dup_span_names,
    trace,
    *,
    source_content_spans=0,
    recovery=None,
)

Refuse a trace that would silently lose data; no-op on a faithful one.

Raises IngestError when the source has duplicate span ids, when the source carried tool-execution spans but the normalized trace recovered zero tool calls, or when the source carried per-step content and none of it survived ('s content half; a trace whose tools are all present by name with empty args and results passes the count check while having lost everything a judge would read). A genuinely tool-free or content-free source is never refused: both gates compare the source against the result, so "nothing was there" and "something was dropped" stay different claims.

source_content_spans defaults to 0, i.e. the content gate is off unless the caller measured the source; a caller that cannot measure it must not have its trace refused for that.

The content gate reads the recovered side through the trace's own ownership, because a probed trace has spans == [] by definition and keeps its content in the terminal input/output envelope instead. Counting only span content refused every legitimately-degraded trace as a placement mismatch; wrong (nothing was lost, and the degradation had already been announced with a reason) and unactionable (there is no placement to go and fix). Whether an outcome-only trace is usable is the caller's call, taken through require_owned.

The tool gate reads recovery when the caller supplies it, so it measures what the ADAPTER linked rather than what survived the ownership collapse. The loss gate and the degrade rule ask different questions of the same probed, spans == [] trace; "did the adapter fail to link these tool spans?" (refuse) versus "was every node it linked empty?" (degrade, and explicitly do not refuse); and the finished Trace cannot tell them apart, because both look identical once the spans are gone. Without recovery the gate falls back to the post-collapse count, which is the conservative reading for a caller that did not measure.

Source code in src/calibrated_agent_evaluation/ingest/core.py
def verify_no_silent_loss(
    source_tool_span_count: int,
    dup_span_names: list[str],
    trace: Trace,
    *,
    source_content_spans: int = 0,
    recovery: Recovery | None = None,
) -> None:
    """Refuse a trace that would silently lose data; no-op on a faithful one.

    Raises :class:`IngestError` when the source has duplicate span ids, when the source
    carried tool-execution spans but the normalized ``trace`` recovered zero tool calls,
    or when the source carried per-step content and none of it survived ('s content half;
    a trace whose tools are all present *by name* with empty args and results passes the count
    check while having lost everything a judge would read). A genuinely tool-free or content-free
    source is never refused: both gates compare the source against the result, so "nothing was
    there" and "something was dropped" stay different claims.

    ``source_content_spans`` defaults to 0, i.e. the content gate is off unless the caller measured
    the source; a caller that cannot measure it must not have its trace refused for that.

    The content gate reads the recovered side through the trace's own ownership, because a
    ``probed`` trace has ``spans == []`` by definition and keeps its content in the terminal
    ``input``/``output`` envelope instead. Counting only span content refused every
    legitimately-degraded trace as a placement mismatch; wrong (nothing was lost, and the
    degradation had already been announced with a reason) and unactionable (there is no placement
    to go and fix). Whether an outcome-only trace is usable is the caller's call, taken through
    ``require_owned``.

    The tool gate reads ``recovery`` when the caller supplies it, so it measures what the ADAPTER
    linked rather than what survived the ownership collapse. The loss gate and the degrade rule ask
    different
    questions of the same ``probed``, ``spans == []`` trace; "did the adapter fail to link these
    tool spans?" (refuse) versus "was every node it linked empty?" (degrade, and explicitly do not
    refuse); and the finished Trace cannot tell them apart, because both look identical once the
    spans are gone. Without ``recovery`` the gate falls back to the post-collapse count, which is
    the conservative reading for a caller that did not measure.
    """
    if dup_span_names:
        raise duplicate_span_error(dup_span_names)
    recovered = (
        recovery.recovered_tool_calls
        if recovery is not None
        else sum(len(span.tool_calls) for span in trace.spans)
    )
    if source_tool_span_count > 0 and recovered == 0:
        raise tool_loss_error(source_tool_span_count)
    if source_content_spans > 0 and not _recovered_any_content(trace, source_content_spans):
        raise content_loss_error(source_content_spans)

detect_source

detect_source(
    path,
    fmt="auto",
    *,
    trace_id=None,
    convention="auto",
    frame="discovery",
)

Return the normalizer source descriptor for path.

fmt forces a format (strands / otel / inspect); auto (default) detects it. trace_id overrides the filename-stem default (the batch collision resolver passes one). Raises IngestError for an unrecognized shape; the caller records that as a skipped source, never a silent drop.

frame applies to a source whose spans carry no calibrated_agent_evaluation.frame; it defaults to "discovery" because a file says nothing about how its traces were sampled.

Source code in src/calibrated_agent_evaluation/ingest/detect.py
def detect_source(
    path: Path,
    fmt: str = "auto",
    *,
    trace_id: str | None = None,
    convention: str = "auto",
    frame: Frame = "discovery",
) -> NormalizerSource:
    """Return the normalizer source descriptor for ``path``.

    ``fmt`` forces a format (``strands`` / ``otel`` / ``inspect``); ``auto`` (default) detects it.
    ``trace_id`` overrides the filename-stem default (the batch collision resolver passes one).
    Raises :class:`IngestError` for an unrecognized shape; the caller records that as a skipped
    source, never a silent drop.

    ``frame`` applies to a source whose spans carry no ``calibrated_agent_evaluation.frame``; it
    defaults to ``"discovery"`` because a file says nothing about how its traces were sampled.
    """
    tid = trace_id or _trace_id_for(path)
    if convention != "auto":
        # A declared convention (`calibrated-agent-evaluation ingest --convention`) is the answer to
        # detect's ambiguity refusal, so it means the OTel reader with that grammar, skipping the
        # Strands sniff.
        if fmt not in ("auto", "otel"):
            raise IngestError(f"--convention applies to OTLP sources, not --format {fmt}")
        return OTelSpans(
            path=path,
            trace_id=tid,
            semconv=cast("Semconv | Convention", convention),
            frame=frame,
        )
    if fmt == "strands":
        return StrandsSession(path=path, trace_id=tid, frame=frame)
    if fmt == "otel":
        return OTelSpans(path=path, trace_id=tid, frame=frame)
    if fmt == "inspect":
        return InspectEvalLog(path=path, trace_id=tid, frame=frame)
    if fmt != "auto":
        raise IngestError(f"unknown --format {fmt!r}; expected {' | '.join(FORMATS)}")

    if path.suffix == ".eval":
        return InspectEvalLog(path=path, trace_id=tid, frame=frame)
    if path.suffix == ".otlp":
        # Binary protobuf; the generic adapter decodes it (the [otel] extra). Strands is detected
        # from OTLP-JSON only, so a binary Strands export needs --format strands.
        return OTelSpans(path=path, trace_id=tid, frame=frame)
    if path.name.endswith(".otlp.json") or path.suffix == ".json":
        otlp = _load_otlp(path)
        if _looks_strands(otlp):
            return StrandsSession(path=path, trace_id=tid, frame=frame)
        return OTelSpans(path=path, trace_id=tid, frame=frame)
    raise IngestError(f"unrecognized trace source {path.name!r}; expected one of {_SUPPORTED}")

draw_sample

draw_sample(population, *, size, seed)

Draw size ids from population without replacement, reproducibly.

The population is sorted before the draw, so the same ids and seed give the same sample whatever order the store listed them in. The returned ids are in sorted order too.

Parameters:

Name Type Description Default
population Sequence[str]

Every candidate id, in any order; duplicates are collapsed.

required
size int

How many to draw.

required
seed int

The seed; the same seed and population always draw the same sample.

required

Returns:

Type Description
list[str]

The drawn ids, sorted.

Raises:

Type Description
IngestError

size is not positive, the population is empty, or the population is smaller than size (a short draw is not a smaller sample, it is a different population, so it is refused rather than quietly reduced).

Source code in src/calibrated_agent_evaluation/ingest/sample.py
def draw_sample(population: Sequence[str], *, size: int, seed: int) -> list[str]:
    """Draw ``size`` ids from ``population`` without replacement, reproducibly.

    The population is sorted before the draw, so the same ids and seed give the same sample
    whatever order the store listed them in. The returned ids are in sorted order too.

    Args:
        population: Every candidate id, in any order; duplicates are collapsed.
        size: How many to draw.
        seed: The seed; the same seed and population always draw the same sample.

    Returns:
        The drawn ids, sorted.

    Raises:
        IngestError: ``size`` is not positive, the population is empty, or the population is
            smaller than ``size`` (a short draw is not a smaller sample, it is a different
            population, so it is refused rather than quietly reduced).
    """
    if size < 1:
        raise IngestError(f"a sample needs at least one trace, got {size}")
    candidates = sorted(set(population))
    if not candidates:
        raise IngestError(
            "nothing to sample: the population is empty. Widen the scope (drop --cell, or point "
            "at the project or files that hold the traces)"
        )
    if size > len(candidates):
        raise IngestError(
            f"asked for a sample of {size} but the population holds {len(candidates)} trace(s); "
            "a short draw would be a different population, so it is refused. Lower the sample "
            "size or widen the scope"
        )
    rng = random.Random(seed)  # noqa: S311  # nosec B311 - a seeded statistical draw, not crypto
    return sorted(rng.sample(candidates, size))

dump_samples

dump_samples(records, path, fmt='csv')

Write records to sample_record.{csv,json} under path.

Parameters:

Name Type Description Default
records Sequence[SampleRecord]

The sample records to persist.

required
path str | Path

The directory to write into.

required
fmt str

Which file a later load_samples reads back (both are always written).

'csv'

Raises:

Type Description
IngestError

A bad fmt or any filesystem failure.

Source code in src/calibrated_agent_evaluation/ingest/sample.py
def dump_samples(records: Sequence[SampleRecord], path: str | Path, fmt: str = "csv") -> None:
    """Write ``records`` to ``sample_record.{csv,json}`` under ``path``.

    Args:
        records: The sample records to persist.
        path: The directory to write into.
        fmt: Which file a later :func:`load_samples` reads back (both are always written).

    Raises:
        IngestError: A bad ``fmt`` or any filesystem failure.
    """
    _STORE.dump(records, path, fmt)

load_samples

load_samples(path, fmt='csv')

Read sample records back from path; a missing file is an empty store.

Parameters:

Name Type Description Default
path str | Path

The directory dump_samples wrote.

required
fmt str

"csv" or "json".

'csv'

Returns:

Type Description
list[SampleRecord]

The stored records.

Raises:

Type Description
IngestError

A schema-version mismatch, or any read, parse or validation failure.

Source code in src/calibrated_agent_evaluation/ingest/sample.py
def load_samples(path: str | Path, fmt: str = "csv") -> list[SampleRecord]:
    """Read sample records back from ``path``; a missing file is an empty store.

    Args:
        path: The directory :func:`dump_samples` wrote.
        fmt: ``"csv"`` or ``"json"``.

    Returns:
        The stored records.

    Raises:
        IngestError: A schema-version mismatch, or any read, parse or validation failure.
    """
    return _STORE.load(path, fmt)