Skip to content

calibrated_agent_evaluation.backends

calibrated_agent_evaluation.backends

Data-plane backends for Calibrated Agent Evaluation.

The TraceStoreBackend / LabelBackend protocols are defined against the normalized schema only; no vendor type appears in any method signature. LocalBackend (stdlib sqlite3 + CSV/JSON) is the no-config offline default; LangfuseBackend is the opt-in both-worlds adapter (the langfuse extra).

LangfuseBackend and the dashboard sinks are served lazily via __getattr__ (mirroring execute/__init__) so import calibrated_agent_evaluation.backends on a bare offline install pulls neither the Langfuse vendor SDK nor the run/calibrate stats chain the dashboard sinks import (scipy/pyarrow). This keeps the offline CLI path zero-third-party at import time the light stdlib-only LocalBackend + get_backend are still eager.

DEFAULT_LOCAL_ROOT module-attribute

DEFAULT_LOCAL_ROOT = '.calibrated-agent-evaluation'

LabelBackend

Bases: Protocol

A human-labeling work-list source/sink, defined against normalized Label only.

No implementation ships: this is the managed-labeling seam, deferred until a managed labeling backend is built. The review app reads its work list from a TraceStoreBackend and writes labels with put_score instead. The interface is fixed so a future labeling backend speaks the normalized schema, never a vendor labeling type.

next_item

next_item()

Return the next trace awaiting a human label, or None when the work-list is empty.

Source code in src/calibrated_agent_evaluation/backends/base.py
def next_item(self) -> Trace | None:
    """Return the next trace awaiting a human label, or ``None`` when the work-list is empty."""
    ...

submit

submit(label)

Record a human label. Raises SchemaError on invalid input.

Source code in src/calibrated_agent_evaluation/backends/base.py
def submit(self, label: Label) -> None:
    """Record a human label. Raises ``SchemaError`` on invalid input."""
    ...

ScanningBackend

Bases: Protocol

A store that can hold traces it cannot return as normalized Traces.

scan

scan(frame=None, cell=None, ownership=None)

query, but a trace it cannot return is a SkippedTrace, not a raise.

Source code in src/calibrated_agent_evaluation/backends/base.py
def scan(
    self,
    frame: Frame | None = None,
    cell: dict[str, str] | None = None,
    ownership: Ownership | None = None,
) -> TraceScan:
    """``query``, but a trace it cannot return is a :class:`SkippedTrace`, not a raise."""
    ...

SkippedTrace pydantic-model

Bases: CalibratedAgentEvaluationModel

A trace a whole-store read left out, and why.

Config:

  • frozen: True
  • extra: forbid

Fields:

trace_id pydantic-field

trace_id

reason pydantic-field

reason

SourceTrace pydantic-model

Bases: CalibratedAgentEvaluationModel

One trace read from a source store, in the form the store could give it.

Exactly one of trace, otlp and error is set:

  • trace: the store holds a Calibrated Agent Evaluation-normalized trace and returned it losslessly;
  • otlp: the store holds a trace something else wrote, mapped to a canonical OTLP-JSON document for the ingest path to detect and normalize like a file;
  • error: this one trace could not be read, and why. The ingest path reports it as skipped and carries on, because one unreadable trace is no reason to drop the rest of the pull.

Plain types only, so a vendor shape never crosses into the ingest plane.

Config:

  • frozen: True
  • extra: forbid

Fields:

Validators:

  • _exactly_one_form

source_id pydantic-field

source_id

trace pydantic-field

trace = None

otlp pydantic-field

otlp = None

error pydantic-field

error = None

session_id pydantic-field

session_id = None

TraceScan pydantic-model

Bases: CalibratedAgentEvaluationModel

A whole-store read: the traces read, and every trace left out with its reason.

skipped is how a read stays honest about what it did not return. A store shared with another client holds traces Calibrated Agent Evaluation did not write; a stage reads the rest and says how many it left out and why, instead of aborting on the first foreign trace or dropping it unseen.

Config:

  • frozen: True
  • extra: forbid

Fields:

traces pydantic-field

traces

skipped pydantic-field

skipped = ()

TraceSource

Bases: Protocol

A store that can be pulled from, including traces it did not normalize itself.

pull

pull()

Yield every trace in the store as a SourceTrace.

Raises BackendError before yielding anything if the store cannot be listed (unreachable or rejected credentials); a failure on one trace is a SourceTrace.error, not a raise.

Source code in src/calibrated_agent_evaluation/backends/base.py
def pull(self) -> Iterator[SourceTrace]:
    """Yield every trace in the store as a :class:`SourceTrace`.

    Raises ``BackendError`` before yielding anything if the store cannot be listed (unreachable
    or rejected credentials); a failure on one trace is a ``SourceTrace.error``, not a raise.
    """
    ...

TraceStoreBackend

Bases: Protocol

Persist and retrieve normalized traces and their labels through one interface.

The methodology plane is written once against this protocol and works regardless of which concrete backend is configured.

put_trace

put_trace(trace)

Persist a trace (and its span tree). Raises SchemaError on invalid input.

Source code in src/calibrated_agent_evaluation/backends/base.py
def put_trace(self, trace: Trace) -> None:
    """Persist a trace (and its span tree). Raises ``SchemaError`` on invalid input."""
    ...

get_trace

get_trace(trace_id)

Return the trace for trace_id, equal field-for-field to what was written.

Raises BackendError if no such trace exists or the backend is unreachable.

Source code in src/calibrated_agent_evaluation/backends/base.py
def get_trace(self, trace_id: str) -> Trace:
    """Return the trace for ``trace_id``, equal field-for-field to what was written.

    Raises ``BackendError`` if no such trace exists or the backend is unreachable.
    """
    ...

query

query(frame=None, cell=None, ownership=None)

Return the traces matching the given queryable dimensions.

The locked query surface; frame / cell / ownership only; richer querying is out of scope. An empty match returns [].

Source code in src/calibrated_agent_evaluation/backends/base.py
def query(
    self,
    frame: Frame | None = None,
    cell: dict[str, str] | None = None,
    ownership: Ownership | None = None,
) -> list[Trace]:
    """Return the traces matching the given queryable dimensions.

    The locked query surface; ``frame`` / ``cell`` / ``ownership`` only; richer
    querying is out of scope. An empty match returns ``[]``.
    """
    ...

put_score

put_score(label)

Persist a label. Raises SchemaError on invalid input.

Source code in src/calibrated_agent_evaluation/backends/base.py
def put_score(self, label: Label) -> None:
    """Persist a label. Raises ``SchemaError`` on invalid input."""
    ...

get_scores

get_scores(trace_id, span_id=None)

Return the labels for a trace (optionally narrowed to one span).

Source code in src/calibrated_agent_evaluation/backends/base.py
def get_scores(self, trace_id: str, span_id: str | None = None) -> list[Label]:
    """Return the labels for a trace (optionally narrowed to one span)."""
    ...

retract_score

retract_score(label)

Retract a stored label, so no later get_scores returns it.

label is matched by full content (schema.serde.label_identity), the same identity the read-side dedup uses, so exactly that label goes and a different verdict on the same target stays. Putting the same label again afterwards stores it again. Retracting a label that is not (yet) readable is not an error: an eventually-consistent store cannot tell "absent" from "still ingesting", and the retraction must win over a put that lands late. Raises BackendError if the backend is unreachable or the retraction is not written.

Source code in src/calibrated_agent_evaluation/backends/base.py
def retract_score(self, label: Label) -> None:
    """Retract a stored label, so no later ``get_scores`` returns it.

    ``label`` is matched by full content (``schema.serde.label_identity``), the same identity
    the read-side dedup uses, so exactly that label goes and a different verdict on the same
    target stays. Putting the same label again afterwards stores it again. Retracting a label
    that is not (yet) readable is not an error: an eventually-consistent store cannot tell
    "absent" from "still ingesting", and the retraction must win over a put that lands late.
    Raises ``BackendError`` if the backend is unreachable or the retraction is not written.
    """
    ...

LocalBackend

LocalBackend(root='.calibrated-agent-evaluation')

A TraceStoreBackend backed by stdlib sqlite3 and the local filesystem.

Spans are stored in their own table (for the indexed tree reconstruction) but are also embedded in the trace doc blob, which is the source of truth on read. Use it as a context manager, or call close, to release the database connection.

Parameters:

Name Type Description Default
root str | Path

The directory holding the store, created if missing. ":memory:" keeps the store in memory, for tests.

'.calibrated-agent-evaluation'
Source code in src/calibrated_agent_evaluation/backends/local.py
def __init__(self, root: str | Path = ".calibrated-agent-evaluation") -> None:
    if root == ":memory:":
        self._db_path = ":memory:"
    else:
        root_path = Path(root)
        root_path.mkdir(parents=True, exist_ok=True)
        self._db_path = str(root_path / "store.sqlite3")
    # One connection, used from the thread that opened it (sqlite3's default check).
    self._conn = sqlite3.connect(self._db_path)
    self._conn.executescript(_SCHEMA)
    self._conn.commit()
    _migrate(self._conn)

put_trace

put_trace(trace)

Store trace and its spans, replacing any trace with the same trace_id.

Parameters:

Name Type Description Default
trace Trace

The normalized trace to store.

required

Raises:

Type Description
SchemaError

The trace fails validation; nothing is written.

Source code in src/calibrated_agent_evaluation/backends/local.py
def put_trace(self, trace: Trace) -> None:
    """Store ``trace`` and its spans, replacing any trace with the same ``trace_id``.

    Args:
        trace: The normalized trace to store.

    Raises:
        SchemaError: The trace fails validation; nothing is written.
    """
    validate(trace)  # refuse invalid artifacts before any write
    doc = json.dumps(to_dict(trace))
    cell_feature = trace.taxonomy_cell.get("feature")
    with self._conn:
        self._conn.execute(
            "INSERT OR REPLACE INTO traces "
            "(trace_id, frame, ownership, cell_feature, doc) VALUES (?, ?, ?, ?, ?)",
            (trace.trace_id, trace.frame, trace.ownership, cell_feature, doc),
        )
        self._conn.execute("DELETE FROM spans WHERE trace_id = ?", (trace.trace_id,))
        self._conn.executemany(
            "INSERT OR REPLACE INTO spans (span_id, trace_id, seq, doc) VALUES (?, ?, ?, ?)",
            [
                (span.span_id, span.trace_id, span.seq, json.dumps(to_dict(span)))
                for span in trace.spans
            ],
        )

get_trace

get_trace(trace_id)

Return the stored trace for trace_id, field for field as it was written.

Parameters:

Name Type Description Default
trace_id str

The id of the trace to read.

required

Returns:

Type Description
Trace

The stored trace.

Raises:

Type Description
TraceNotFoundError

No trace with that id is stored.

Source code in src/calibrated_agent_evaluation/backends/local.py
def get_trace(self, trace_id: str) -> Trace:
    """Return the stored trace for ``trace_id``, field for field as it was written.

    Args:
        trace_id: The id of the trace to read.

    Returns:
        The stored trace.

    Raises:
        TraceNotFoundError: No trace with that id is stored.
    """
    row = self._conn.execute(
        "SELECT doc FROM traces WHERE trace_id = ?", (trace_id,)
    ).fetchone()
    if row is None:
        raise TraceNotFoundError(f"no trace with trace_id {trace_id!r} in local store")
    return _trace_from_doc(row[0])

query

query(frame=None, cell=None, ownership=None)

Return the stored traces that match every given filter, ordered by trace_id.

Parameters:

Name Type Description Default
frame Frame | None

Keep only traces in this sampling frame.

None
cell dict[str, str] | None

Keep only traces whose taxonomy cell has all of these key/value pairs.

None
ownership Ownership | None

Keep only owned or only probed traces.

None

Returns:

Type Description
list[Trace]

The matching traces; an empty list when none match.

Source code in src/calibrated_agent_evaluation/backends/local.py
def query(
    self,
    frame: Frame | None = None,
    cell: dict[str, str] | None = None,
    ownership: Ownership | None = None,
) -> list[Trace]:
    """Return the stored traces that match every given filter, ordered by ``trace_id``.

    Args:
        frame: Keep only traces in this sampling frame.
        cell: Keep only traces whose taxonomy cell has all of these key/value pairs.
        ownership: Keep only owned or only probed traces.

    Returns:
        The matching traces; an empty list when none match.
    """
    clauses: list[str] = []
    params: list[Any] = []
    if frame is not None:
        clauses.append("frame = ?")
        params.append(frame)
    if ownership is not None:
        clauses.append("ownership = ?")
        params.append(ownership)
    if cell is not None and "feature" in cell:
        clauses.append("cell_feature = ?")
        params.append(cell["feature"])
    sql = "SELECT doc FROM traces"
    if clauses:
        sql += " WHERE " + " AND ".join(clauses)
    sql += " ORDER BY trace_id"
    traces = [_trace_from_doc(r[0]) for r in self._conn.execute(sql, params)]
    # Remaining cell keys (beyond the indexed `feature`) filter in-memory.
    if cell:
        extra = {k: v for k, v in cell.items() if k != "feature"}
        if extra:
            traces = [
                t for t in traces if all(t.taxonomy_cell.get(k) == v for k, v in extra.items())
            ]
    return traces

put_score

put_score(label)

Append a label. Identity-duplicate rows are collapsed by dedup_labels on read.

Deliberately an append rather than an upsert. labels.rowid_pk is an autoincrement primary key and nothing in the existing schema carries the identity, so an upsert needs either a migration of every store on disk or an identity column that the Langfuse backend could not mirror anyway (its create_score API has no update). Deduping on read gives both backends one behaviour and keeps every already-written store readable.

rater_id is projected into its own column, like trace_id/span_id, but stays out of the (trace_id, span_id) index: every read filters on the target, none on the rater, and the index is a lookup path, not a uniqueness key. Two raters' labels on one target are kept apart by the full-content identity dedup_labels applies, which includes rater_id, so widening the index would change no result.

Source code in src/calibrated_agent_evaluation/backends/local.py
def put_score(self, label: Label) -> None:
    """Append a label. Identity-duplicate rows are collapsed by :func:`dedup_labels` on read.

    Deliberately an append rather than an upsert. `labels.rowid_pk` is an autoincrement primary
    key and nothing in the existing schema carries the identity, so an upsert needs either a
    migration of every store on disk or an identity column that the Langfuse backend could not
    mirror anyway (its `create_score` API has no update). Deduping on read gives both backends
    one behaviour and keeps every already-written store readable.

    ``rater_id`` is projected into its own column, like ``trace_id``/``span_id``, but stays out
    of the ``(trace_id, span_id)`` index: every read filters on the target, none on the rater,
    and the index is a lookup path, not a uniqueness key. Two raters' labels on one target are
    kept apart by the full-content identity :func:`dedup_labels` applies, which includes
    ``rater_id``, so widening the index would change no result.
    """
    validate(label)  # refuse invalid labels before any write
    with self._conn:
        self._conn.execute(
            "INSERT INTO labels (trace_id, span_id, doc, rater_id) VALUES (?, ?, ?, ?)",
            (label.trace_id, label.span_id, json.dumps(to_dict(label)), label.rater_id),
        )

get_scores

get_scores(trace_id, span_id=None)

Return the labels stored for a trace, oldest first, with duplicates collapsed.

Parameters:

Name Type Description Default
trace_id str

The trace whose labels to read.

required
span_id str | None

Narrow the result to labels on this span.

None

Returns:

Type Description
list[Label]

The labels; an empty list when the trace has none.

Source code in src/calibrated_agent_evaluation/backends/local.py
def get_scores(self, trace_id: str, span_id: str | None = None) -> list[Label]:
    """Return the labels stored for a trace, oldest first, with duplicates collapsed.

    Args:
        trace_id: The trace whose labels to read.
        span_id: Narrow the result to labels on this span.

    Returns:
        The labels; an empty list when the trace has none.
    """
    if span_id is None:
        rows = self._conn.execute(
            "SELECT doc FROM labels WHERE trace_id = ? ORDER BY rowid_pk", (trace_id,)
        )
    else:
        rows = self._conn.execute(
            "SELECT doc FROM labels WHERE trace_id = ? AND span_id = ? ORDER BY rowid_pk",
            (trace_id, span_id),
        )
    # `ORDER BY rowid_pk` makes the last byte-identical row the last written, which is what
    # `dedup_labels` keeps. `put_score` appends (see its docstring), so re-putting one label
    # leaves several identical rows, and an undeduped read doubled every rate over them.
    return dedup_labels(label_from_dict(json.loads(r[0])) for r in rows)

retract_score

retract_score(label)

Delete every stored row carrying exactly label (see TraceStoreBackend).

A real delete, because SQLite's is synchronous and exact. Every byte-identical row goes: an idempotent re-put leaves several rows that read back as one label, and leaving any of them would make the retraction a no-op on read. Retracting a label that is not stored deletes nothing, matching the Langfuse backend, which cannot check synchronously.

Source code in src/calibrated_agent_evaluation/backends/local.py
def retract_score(self, label: Label) -> None:
    """Delete every stored row carrying exactly ``label`` (see ``TraceStoreBackend``).

    A real delete, because SQLite's is synchronous and exact. Every byte-identical row goes:
    an idempotent re-put leaves several rows that read back as one label, and leaving any of
    them would make the retraction a no-op on read. Retracting a label that is not stored
    deletes nothing, matching the Langfuse backend, which cannot check synchronously.
    """
    identity = label_identity(label)
    rows = self._conn.execute(
        "SELECT rowid_pk, doc FROM labels WHERE trace_id = ? AND span_id IS ?",
        (label.trace_id, label.span_id),
    ).fetchall()
    doomed = [
        (pk,)
        for pk, doc in rows
        if label_identity(label_from_dict(json.loads(doc))) == identity
    ]
    with self._conn:
        self._conn.executemany("DELETE FROM labels WHERE rowid_pk = ?", doomed)

all_traces

all_traces()

Return every stored trace (dump support).

Source code in src/calibrated_agent_evaluation/backends/local.py
def all_traces(self) -> list[Trace]:
    """Return every stored trace (dump support)."""
    rows = self._conn.execute("SELECT doc FROM traces ORDER BY trace_id")
    return [_trace_from_doc(r[0]) for r in rows]

all_scores

all_scores()

Return every stored label (dump support).

Source code in src/calibrated_agent_evaluation/backends/local.py
def all_scores(self) -> list[Label]:
    """Return every stored label (dump support)."""
    rows = self._conn.execute("SELECT doc FROM labels ORDER BY rowid_pk")
    # `ORDER BY rowid_pk` makes the last byte-identical row the last written, which is what
    # `dedup_labels` keeps. `put_score` appends (see its docstring), so re-putting one label
    # leaves several identical rows, and an undeduped read doubled every rate over them.
    return dedup_labels(label_from_dict(json.loads(r[0])) for r in rows)

wipe

wipe()

Delete every artifact (the dump→wipe→load no-lock-in proof).

Source code in src/calibrated_agent_evaluation/backends/local.py
def wipe(self) -> None:
    """Delete every artifact (the dump→wipe→load no-lock-in proof)."""
    with self._conn:
        self._conn.execute("DELETE FROM traces")
        self._conn.execute("DELETE FROM spans")
        self._conn.execute("DELETE FROM labels")

close

close()

Close the database connection. The backend can't be used afterwards.

Source code in src/calibrated_agent_evaluation/backends/local.py
def close(self) -> None:
    """Close the database connection. The backend can't be used afterwards."""
    self._conn.close()

DashboardSink

Bases: Protocol

Publishes a run report's numeric series to a dashboard target.

publish

publish(report)

Publish report's numeric series; raise on a target failure (never silently no-op).

Source code in src/calibrated_agent_evaluation/backends/dashboard.py
def publish(self, report: RunReport) -> None:
    """Publish ``report``'s numeric series; raise on a target failure (never silently no-op)."""
    ...

LangfuseDashboardSink

LangfuseDashboardSink(
    client,
    *,
    local=None,
    flush_timeout_seconds=_DEFAULT_FLUSH_TIMEOUT_SECONDS,
)

Publishes the numeric series to Langfuse as numeric scores for dashboard rendering.

Mirrors LangfuseBackend: publish calls auth_check first and raises BackendError on an unreachable or rejecting backend before writing, and every score is a synchronous REST create, so a score the server refuses raises instead of vanishing. The local sink (if provided) is written first and is authoritative; the Langfuse leg is additive. Client injected for tests; from_credentials builds the real MIT SDK client.

Source code in src/calibrated_agent_evaluation/backends/dashboard.py
def __init__(
    self,
    client: LangfuseScoreClient,
    *,
    local: LocalDashboardSink | None = None,
    flush_timeout_seconds: float = _DEFAULT_FLUSH_TIMEOUT_SECONDS,
) -> None:
    self._client = client
    self._local = local
    self._flush_timeout_seconds = flush_timeout_seconds

publish

publish(report)

Publish the numeric series to Langfuse (and the local sink first, if configured).

Source code in src/calibrated_agent_evaluation/backends/dashboard.py
def publish(self, report: RunReport) -> None:
    """Publish the numeric series to Langfuse (and the local sink first, if configured)."""
    if self._local is not None:
        self._local.publish(report)  # authoritative round-trip artifact, written first

    # The async OTel write path does NOT raise on a dead backend, so probe first.
    try:
        reachable = self._client.auth_check()
    except Exception as exc:  # connection/auth failure -> typed BackendError (never silent)
        raise BackendError(
            f"Langfuse dashboard publish failed: backend unreachable ({exc})"
        ) from exc
    if not reachable:
        raise BackendError(
            "Langfuse dashboard publish failed: backend unreachable or auth rejected"
        )

    # Deduplicate by (name, kind) before writing. A score create is append-only (the same root
    # cause as the label-retraction duplicates) and the trace id is seeded on run_id, so
    # re-publishing a run appended a second copy of every trend point and the chart showed each
    # value twice. Dedup here rather than on read, because a dashboard sink has no read path.
    points = _dedup_points(numeric_series(report))
    trace_id = self._client.create_trace_id(
        seed=f"calibrated-agent-evaluation-run-{report.run_id}"
    )
    try:
        # A score on a trace id Langfuse has never seen is accepted and then never readable
        # (verified against Langfuse 3.202), so the run gets a trace of its own first.
        span = self._client.start_span(
            trace_context={"trace_id": trace_id},
            name="calibrated-agent-evaluation-run",
            metadata={"run_id": report.run_id, "mode": report.mode},
        )
        span.update_trace(
            name=f"calibrated-agent-evaluation run {report.run_id}",
            metadata={"run_id": report.run_id},
        )
        span.end()
    except Exception as exc:  # any SDK failure -> typed BackendError (no raw error escapes)
        raise BackendError(
            f"Langfuse dashboard publish failed while opening the run trace: {exc}"
        ) from exc
    # The span travels on the queued OTel path; send it before the scores that hang off it.
    self._flush_bounded()
    for point in points:
        if point.value is None:
            # A refused correction has no number to chart; publish the refusal itself as a
            # categorical score so the dashboard shows it rather than a gap.
            self._create_score(
                trace_id,
                point.name,
                "refused",
                "CATEGORICAL",
                f"{point.kind} for run {point.run_id}: {point.refusal_reason}",
            )
        else:
            self._create_score(trace_id, point.name, point.value, "NUMERIC", _comment(point))

from_credentials classmethod

from_credentials(
    public_key,
    secret_key,
    host,
    *,
    timeout_seconds=5,
    local=None,
    client_factory=None,
)

Build against the real MIT SDK client (or an injected factory in tests).

timeout_seconds bounds every request including the auth_check probe (the SDK accepts timeout in seconds, verified langfuse==3.15.0), so an unreachable backend fails fast.

Source code in src/calibrated_agent_evaluation/backends/dashboard.py
@classmethod
def from_credentials(
    cls,
    public_key: str,
    secret_key: str,
    host: str,
    *,
    timeout_seconds: int = 5,
    local: LocalDashboardSink | None = None,
    client_factory: Callable[..., Any] | None = None,
) -> LangfuseDashboardSink:
    """Build against the real MIT SDK client (or an injected factory in tests).

    ``timeout_seconds`` bounds every request including the ``auth_check`` probe (the SDK accepts
    ``timeout`` in seconds, verified langfuse==3.15.0), so an unreachable backend fails fast.
    """
    factory: Callable[..., Any]
    if (
        client_factory is None
    ):  # pragma: no cover - real SDK path, exercised by the integration test
        from langfuse import Langfuse  # opt-in [langfuse] extra; MIT build

        factory = Langfuse
    else:
        factory = client_factory
    raw = factory(
        public_key=public_key, secret_key=secret_key, host=host, timeout=timeout_seconds
    )
    return cls(cast("LangfuseScoreClient", raw), local=local)

LocalDashboardSink

LocalDashboardSink(directory)

The authoritative file sink: numeric series -> CSV+JSON, round-trippable.

Offline, dependency-free, identical in both worlds. Writes both files under directory; load reads the JSON back field-for-field. Any filesystem/parse failure is wrapped into RunStoreError (no raw OSError/JSONDecodeError escapes -- "no silent failure").

Source code in src/calibrated_agent_evaluation/backends/dashboard.py
def __init__(self, directory: str | Path) -> None:
    self._dir = Path(directory)

publish

publish(report)

Write the numeric series to dashboard_series.{csv,json} under the directory.

Source code in src/calibrated_agent_evaluation/backends/dashboard.py
def publish(self, report: RunReport) -> None:
    """Write the numeric series to ``dashboard_series.{csv,json}`` under the directory."""
    points = numeric_series(report)
    rows = [p.model_dump(mode="json") for p in points]
    try:
        self._dir.mkdir(parents=True, exist_ok=True)
        atomic_write_text(self._dir / _JSON_NAME, json.dumps(rows, indent=2, sort_keys=True))
        fieldnames = [
            "name",
            "value",
            "kind",
            "run_id",
            "lo",
            "hi",
            "interval_method",
            "p_value",
            "refusal_reason",
        ]
        buffer = io.StringIO(newline="")
        writer = csv.DictWriter(buffer, fieldnames=fieldnames)
        writer.writeheader()
        writer.writerows(rows)
        atomic_write_text(self._dir / _CSV_NAME, buffer.getvalue())
    except OSError as exc:
        raise RunStoreError(f"failed to write dashboard series to {self._dir}: {exc}") from exc

load

load()

Read the numeric series back field-for-field (the round-trip).

Source code in src/calibrated_agent_evaluation/backends/dashboard.py
def load(self) -> list[NumericPoint]:
    """Read the numeric series back field-for-field (the round-trip)."""
    file = self._dir / _JSON_NAME
    if not file.exists():
        return []
    try:
        rows = json.loads(file.read_text(encoding="utf-8"))
    except OSError as exc:
        raise RunStoreError(f"failed to read dashboard series from {self._dir}: {exc}") from exc
    except json.JSONDecodeError as exc:
        raise RunStoreError(
            f"dashboard series in {self._dir} is not valid JSON: {exc}"
        ) from exc
    try:
        return [NumericPoint.model_validate(r) for r in rows]
    except RunError as exc:
        raise RunStoreError(f"malformed dashboard series in {self._dir} ({exc})") from exc

LangfuseBackend

LangfuseBackend(
    client,
    *,
    telemetry_disabled=True,
    flush_timeout_seconds=20.0,
    export_tap=None,
)

A TraceStoreBackend over an unmodified Langfuse v3 deployment.

Writes block until the Langfuse client confirms its queue drained, so a returned write is durable. Build one with from_credentials; the constructor takes an injected client.

Parameters:

Name Type Description Default
client LangfuseLike

A Langfuse v3 client, or a stand-in with the same surface.

required
telemetry_disabled bool

Recorded for provenance; the client never sends usage telemetry.

True
flush_timeout_seconds float

How long a write waits for the client's queue to drain before it fails with BackendError.

20.0
export_tap ExportTap | None

Counts the client's failed span exports, so a trace write whose export failed raises instead of returning. from_credentials wires one in.

None
Source code in src/calibrated_agent_evaluation/backends/langfuse.py
def __init__(
    self,
    client: LangfuseLike,
    *,
    telemetry_disabled: bool = True,
    flush_timeout_seconds: float = 20.0,
    export_tap: ExportTap | None = None,
) -> None:
    self._client = client
    self._export_tap = export_tap
    self.telemetry_disabled = telemetry_disabled
    # Bounds the blocking flush() (see _flush_bounded). Defaults higher than the per-request
    # timeout because flush waits on the whole queued batch to drain, not one request.
    self._flush_timeout_seconds = flush_timeout_seconds

telemetry_disabled instance-attribute

telemetry_disabled = telemetry_disabled

from_credentials classmethod

from_credentials(
    public_key,
    secret_key,
    host,
    *,
    timeout_seconds=5,
    flush_timeout_seconds=20.0,
    client_factory=None,
)

Construct against the OSS build with telemetry disabled.

Telemetry-off is enforced in two real places, NOT by the client alone: (a) the OSS server reads TELEMETRY_ENABLED (set in docker/langfuse/docker-compose.yml), and (b) this adapter restricts itself to the MIT/OSS public surface; the EE_GATED_METHODS denylist (asserted in tests) is the client-side guarantee that no enterprise-licensed feature is invoked. The v3 Python client does NOT read TELEMETRY_ENABLED (it only reads LANGFUSE_TRACING_ENABLED / the tracing_enabled kwarg, which gate our own span writes; flipping those would break put_trace, so we must not). We still set the server-recognized TELEMETRY_ENABLED=false in-process while the client is built; a harmless belt-and-suspenders for a co-located server that reads the process env; and restore the prior value after, so constructing a backend has no lasting global side effect. client_factory is an injection seam for tests; in production it imports the real MIT SDK. The real langfuse.Langfuse client structurally satisfies LangfuseLike at the call sites the adapter uses (verified surface).

timeout_seconds bounds every API request; including the synchronous reachability probe (auth_check); so an unreachable backend fails fast instead of hanging. The real SDK accepts timeout (seconds; verified langfuse==3.15.0) and defaults it to 5; we surface it as an explicit, tunable parameter rather than rely on that default. flush_timeout_seconds separately bounds the blocking flush() after a write; the SDK's flush does an unbounded queue.join() (verified langfuse==3.15.0) that hangs forever if the ingestion queue wedges, so _flush_bounded caps it and raises a typed BackendError on timeout rather than blocking the caller indefinitely.

Source code in src/calibrated_agent_evaluation/backends/langfuse.py
@classmethod
def from_credentials(
    cls,
    public_key: str,
    secret_key: str,
    host: str,
    *,
    timeout_seconds: int = 5,
    flush_timeout_seconds: float = 20.0,
    client_factory: Callable[..., Any] | None = None,
) -> LangfuseBackend:
    """Construct against the OSS build with telemetry disabled.

    Telemetry-off is enforced in two real places, NOT by the client alone: (a) the OSS
    *server* reads ``TELEMETRY_ENABLED`` (set in ``docker/langfuse/docker-compose.yml``), and
    (b) this adapter restricts itself to the MIT/OSS public surface; the ``EE_GATED_METHODS``
    denylist (asserted in tests) is the client-side guarantee that no enterprise-licensed
    feature is invoked. The v3 Python client does NOT read ``TELEMETRY_ENABLED`` (it only
    reads ``LANGFUSE_TRACING_ENABLED`` / the ``tracing_enabled`` kwarg, which gate our own
    span *writes*; flipping those would break ``put_trace``, so we must not). We still set the
    server-recognized ``TELEMETRY_ENABLED=false`` in-process while the client is built; a
    harmless belt-and-suspenders for a co-located server that reads the process env; and
    restore the prior value after, so constructing a backend has no lasting global side effect.
    ``client_factory`` is an injection seam for tests; in production it imports the real MIT
    SDK. The real ``langfuse.Langfuse`` client structurally satisfies :class:`LangfuseLike`
    at the call sites the adapter uses (verified surface).

    ``timeout_seconds`` bounds every API request; including the synchronous reachability
    probe (``auth_check``); so an unreachable backend fails fast instead of hanging. The
    real SDK accepts ``timeout`` (seconds; verified langfuse==3.15.0) and defaults it to 5;
    we surface it as an explicit, tunable parameter rather than rely on that default.
    ``flush_timeout_seconds`` separately bounds the blocking ``flush()`` after a write; the
    SDK's ``flush`` does an unbounded ``queue.join()`` (verified langfuse==3.15.0) that hangs
    forever if the ingestion queue wedges, so :meth:`_flush_bounded` caps it and raises a typed
    ``BackendError`` on timeout rather than blocking the caller indefinitely.
    """
    factory: Callable[..., Any]
    if client_factory is None:  # pragma: no cover - real SDK path, integration-tested
        from langfuse import Langfuse  # opt-in [langfuse] extra; MIT build

        factory = Langfuse
    else:
        factory = client_factory
    # The client gets a provider of its own (the SDK's documented `tracer_provider` argument)
    # rather than taking over the process's global one, so it exports only the spans this
    # backend writes, and its failed exports are counted (see
    # `calibrated_agent_evaluation.backends._export_tap`).
    tap = ExportTap()
    provider = tapped_tracer_provider(tap)
    # cast: the real langfuse.Langfuse client structurally provides every member of
    # LangfuseLike at the call sites used here (surface verified against langfuse==3.15.0);
    # pyright cannot prove the wider real signatures match the protocol.
    # Server-side telemetry env (the v3 client itself ignores it; see docstring); scoped to
    # construction and restored after so there is no lasting global os.environ mutation.
    with _scoped_env("TELEMETRY_ENABLED", "false"):
        raw_client = factory(
            public_key=public_key,
            secret_key=secret_key,
            host=host,
            timeout=timeout_seconds,
            tracer_provider=provider,
        )
    return cls(
        cast("LangfuseLike", raw_client),
        telemetry_disabled=True,
        flush_timeout_seconds=flush_timeout_seconds,
        export_tap=tap,
    )

check_reachable

check_reachable()

Check that Langfuse can be reached and accepts these credentials, writing nothing.

It is the synchronous probe every write runs first: the client's auth_check, bounded by its request timeout. Run it before costly work whose results go here, so an unreachable host or refused keys stop the work before it starts.

Raises:

Type Description
BackendError

Langfuse can't be reached, or it refuses the credentials.

Source code in src/calibrated_agent_evaluation/backends/langfuse.py
def check_reachable(self) -> None:
    """Check that Langfuse can be reached and accepts these credentials, writing nothing.

    It is the synchronous probe every write runs first: the client's ``auth_check``, bounded
    by its request timeout. Run it before costly work whose results go here, so an unreachable
    host or refused keys stop the work before it starts.

    Raises:
        BackendError: Langfuse can't be reached, or it refuses the credentials.
    """
    self._require_reachable("reachability check")

put_trace

put_trace(trace)

Write trace and its span tree to Langfuse, parents before children.

Parameters:

Name Type Description Default
trace Trace

The normalized trace to store.

required

Raises:

Type Description
SchemaError

The trace fails validation, or a span names a parent that isn't in the trace; nothing is written.

BackendError

Langfuse is unreachable, rejects the write, or doesn't confirm it within flush_timeout_seconds.

Source code in src/calibrated_agent_evaluation/backends/langfuse.py
def put_trace(self, trace: Trace) -> None:
    """Write ``trace`` and its span tree to Langfuse, parents before children.

    Args:
        trace: The normalized trace to store.

    Raises:
        SchemaError: The trace fails validation, or a span names a parent that isn't in the
            trace; nothing is written.
        BackendError: Langfuse is unreachable, rejects the write, or doesn't confirm it within
            ``flush_timeout_seconds``.
    """
    from calibrated_agent_evaluation.schema.validate import validate

    validate(trace)
    self._require_reachable(f"put_trace({trace.trace_id!r})")
    failures_before = self._export_tap.failures if self._export_tap is not None else 0
    doc = to_dict(trace)
    tags = self._dimension_tags(trace)
    trace_envelope = {k: v for k, v in doc.items() if k != "spans"}
    generation = uuid.uuid4().hex
    trace_metadata = {
        _TRACE_ENVELOPE: trace_envelope,
        _GENERATION: generation,
        _SPAN_COUNT: len(trace.spans),
    }
    # Resolve the write order BEFORE the try: this refusal is about the trace, not the SDK, and
    # must not be caught and re-wrapped as a transport failure.
    write_order = _parents_first(trace.spans)
    _refuse_unresolved_parents(trace.trace_id, write_order)
    try:
        root = self._client.start_span(
            trace_context=self._trace_context(trace.trace_id),
            name=trace.trace_id,
            input=trace.input,
            output=trace.output,
            metadata={_TRACE_ENVELOPE: trace_envelope},
        )
        # Queryable dims as trace tags; full normalized trace (minus spans) in metadata.
        root.update_trace(
            tags=tags,
            metadata=trace_metadata,
            input=trace.input,
            output=trace.output,
            # Langfuse's own session too, so its Sessions view groups a session's traces.
            session_id=_native_session(trace),
        )
        lf_parent_ids: dict[str, str] = {}
        for span in write_order:
            parent_lf_id = lf_parent_ids.get(span.parent) if span.parent else None
            obs = self._client.start_observation(
                trace_context=self._trace_context(trace.trace_id, parent_lf_id),
                name=span.agent,
                as_type="span",
                input=span.input,
                output=span.output,
                # seq persisted explicitly; full span (incl. tool_calls) in metadata.
                metadata={
                    "seq": span.seq,
                    _SPAN_ENVELOPE: to_dict(span),
                    _GENERATION: generation,
                },
            )
            lf_parent_ids[span.span_id] = obs.id
            obs.end()
        root.end()
    except Exception as exc:
        raise self._as_backend_error(exc, f"put_trace({trace.trace_id!r})") from exc
    # Flush outside the try: _flush_bounded already raises a typed BackendError on timeout,
    # so wrapping it again would double-wrap. It is the durability confirmation for this write.
    self._flush_bounded(f"put_trace({trace.trace_id!r})")
    self._refuse_failed_export(f"put_trace({trace.trace_id!r})", failures_before)

put_score

put_score(label)

Write label to Langfuse as a score on its trace (and span, if it names one).

Parameters:

Name Type Description Default
label Label

The label to store.

required

Raises:

Type Description
SchemaError

The label fails validation; nothing is written.

BackendError

Langfuse is unreachable, rejects the write, or doesn't confirm it.

Source code in src/calibrated_agent_evaluation/backends/langfuse.py
def put_score(self, label: Label) -> None:
    """Write ``label`` to Langfuse as a score on its trace (and span, if it names one).

    Args:
        label: The label to store.

    Raises:
        SchemaError: The label fails validation; nothing is written.
        BackendError: Langfuse is unreachable, rejects the write, or doesn't confirm it.
    """
    from calibrated_agent_evaluation.schema.validate import validate

    validate(label)
    self._create_score(
        f"put_score({label.trace_id!r})",
        trace_id=label.trace_id,
        name=_SCORE_NAME,
        value=label.outcome,
        comment=label.critique,
        # Bespoke Label fields (attribution, expected, …) ride in Score.metadata.
        metadata={_LABEL_ENVELOPE: to_dict(label), _WRITTEN_AT: _written_at()},
    )

retract_score

retract_score(label)

Retract label by appending a tombstone score (see TraceStoreBackend).

get_scores then hides every put of this label stamped before the tombstone, including one still in Langfuse's ingestion queue; a later put of the same label is visible again. There is deliberately no read-first existence check: score ingestion is asynchronous (verified live, a score takes seconds to become readable), so an undo pressed right after a verdict would find nothing and refuse to retract a label that is about to appear.

Source code in src/calibrated_agent_evaluation/backends/langfuse.py
def retract_score(self, label: Label) -> None:
    """Retract ``label`` by appending a tombstone score (see ``TraceStoreBackend``).

    ``get_scores`` then hides every put of this label stamped before the tombstone, including
    one still in Langfuse's ingestion queue; a later put of the same label is visible again.
    There is deliberately no read-first existence check: score ingestion is asynchronous
    (verified live, a score takes seconds to become readable), so an undo pressed right after
    a verdict would find nothing and refuse to retract a label that is about to appear.
    """
    self._create_score(
        f"retract_score({label.trace_id!r})",
        trace_id=label.trace_id,
        name=_RETRACTION_SCORE_NAME,
        value="RETRACTED",
        comment=f"retracts a {label.outcome} label",
        metadata={_RETRACTION_ENVELOPE: to_dict(label), _WRITTEN_AT: _written_at()},
    )

get_trace

get_trace(trace_id)

Read the trace Calibrated Agent Evaluation wrote under trace_id back from Langfuse.

Parameters:

Name Type Description Default
trace_id str

The Calibrated Agent Evaluation trace id (not the Langfuse id derived from it).

required

Returns:

Type Description
Trace

The trace, rebuilt from the envelope stored with it.

Raises:

Type Description
TraceNotFoundError

Langfuse has no trace for that id.

BackendError

Langfuse can't be read, or the stored trace can't be rebuilt.

Source code in src/calibrated_agent_evaluation/backends/langfuse.py
def get_trace(self, trace_id: str) -> Trace:
    """Read the trace Calibrated Agent Evaluation wrote under ``trace_id`` back from Langfuse.

    Args:
        trace_id: The Calibrated Agent Evaluation trace id (not the Langfuse id derived from
            it).

    Returns:
        The trace, rebuilt from the envelope stored with it.

    Raises:
        TraceNotFoundError: Langfuse has no trace for that id.
        BackendError: Langfuse can't be read, or the stored trace can't be rebuilt.
    """
    lf_trace_id = self._client.create_trace_id(seed=trace_id)
    try:
        raw = self._client.api.trace.get(lf_trace_id)
    except Exception as exc:
        # The SDK raises an ApiError with status_code 404 (NotFoundError, or a bare ApiError
        # when the 404 body is not JSON); anything else is a failed read, not an absence.
        if getattr(exc, "status_code", None) == 404:
            raise TraceNotFoundError(
                f"no trace with trace_id {trace_id!r} in Langfuse"
            ) from exc
        raise self._as_backend_error(exc, f"get_trace({trace_id!r})") from exc
    if raw is None:
        raise TraceNotFoundError(f"no trace with trace_id {trace_id!r} in Langfuse")
    # Reconstruction is wrapped separately: a partial/corrupt envelope makes trace_from_dict
    # raise a bare KeyError that would otherwise escape the CalibratedAgentEvaluationDataError
    # family the data plane contracts on (query() wraps its reconstruction the same way).
    try:
        return self._reconstruct_trace(raw)
    except BackendError:
        raise
    except Exception as exc:
        raise self._as_backend_error(exc, f"get_trace({trace_id!r})") from exc

query

query(frame=None, cell=None, ownership=None)

Return every stored trace matching all given filters, reading every result page.

Parameters:

Name Type Description Default
frame Frame | None

Keep only traces in this sampling frame.

None
cell dict[str, str] | None

Keep only traces whose taxonomy cell has all of these key/value pairs.

None
ownership Ownership | None

Keep only owned or only probed traces.

None

Returns:

Type Description
list[Trace]

The matching traces.

Raises:

Type Description
BackendError

Langfuse can't be read, or a listed trace can't be rebuilt. Use scan to skip traces Calibrated Agent Evaluation didn't write instead of failing.

Source code in src/calibrated_agent_evaluation/backends/langfuse.py
def query(
    self,
    frame: Frame | None = None,
    cell: dict[str, str] | None = None,
    ownership: Ownership | None = None,
) -> list[Trace]:
    """Return every stored trace matching all given filters, reading every result page.

    Args:
        frame: Keep only traces in this sampling frame.
        cell: Keep only traces whose taxonomy cell has all of these key/value pairs.
        ownership: Keep only owned or only probed traces.

    Returns:
        The matching traces.

    Raises:
        BackendError: Langfuse can't be read, or a listed trace can't be rebuilt. Use
            :meth:`scan` to skip traces Calibrated Agent Evaluation didn't write instead of
            failing.
    """
    tags = self._tags(frame, cell, ownership)
    try:
        # Walk every page; the v3 list endpoint paginates, so reading only the first page
        # would silently truncate a large result set (violates "no silent failure").
        items = list(self._list_all_traces(tags))
        # list items carry observation *ids* only; re-get each for the full detail we read.
        traces = [
            self._reconstruct_trace(self._client.api.trace.get(item.id)) for item in items
        ]
    except Exception as exc:
        raise self._as_backend_error(exc, "query") from exc
    return [t for t in traces if _matches(t, frame, cell, ownership)]

scan

scan(frame=None, cell=None, ownership=None)

query, with every trace Calibrated Agent Evaluation did not write skipped and named.

A project shared with another client holds traces without the calibrated_agent_evaluation_trace envelope, and query raises on the first one. A pipeline stage reads with scan instead: the library's traces come back as query returns them, and each foreign one is a SkippedTrace whose reason says how to bring it in (calibrated-agent-evaluation ingest --from-langfuse, which maps it through the path and its gates) or that it already was, under the same id. Skipped, not mapped on the fly: a stage would otherwise score a trace that has passed none of the ingest gates, under span ids no stored label can name. A Calibrated Agent Evaluation trace that fails to reconstruct still raises, since that is a corrupt copy of our own data, not someone else's.

Source code in src/calibrated_agent_evaluation/backends/langfuse.py
def scan(
    self,
    frame: Frame | None = None,
    cell: dict[str, str] | None = None,
    ownership: Ownership | None = None,
) -> TraceScan:
    """``query``, with every trace Calibrated Agent Evaluation did not write skipped and named.

    A project shared with another client holds traces without the
    ``calibrated_agent_evaluation_trace`` envelope, and ``query`` raises on the first one. A
    pipeline stage reads with ``scan`` instead: the library's traces come back as ``query``
    returns them, and each foreign one is a :class:`SkippedTrace` whose reason says how to bring
    it in (``calibrated-agent-evaluation ingest --from-langfuse``, which maps it through the
    path and its gates) or that it already was, under the same id. Skipped, not mapped on the
    fly: a stage would otherwise score a trace that has passed none of the ingest gates, under
    span ids no stored label can name. A Calibrated Agent Evaluation trace that fails to
    reconstruct still raises, since that is a corrupt copy of our own data, not someone else's.
    """
    tags = self._tags(frame, cell, ownership)
    try:
        items = list(self._list_all_traces(tags))
    except Exception as exc:
        raise self._as_backend_error(exc, "scan") from exc
    traces: list[Trace] = []
    foreign: list[str] = []
    for item in items:
        try:
            raw = self._client.api.trace.get(item.id)
            if self._read_metadata(raw).get(_TRACE_ENVELOPE):
                trace = self._reconstruct_trace(raw)
                if _matches(trace, frame, cell, ownership):
                    traces.append(trace)
            else:
                foreign.append(str(item.id))
        except Exception as exc:
            raise self._as_backend_error(exc, f"scan({item.id!r})") from exc
    ingested = {t.trace_id for t in traces}
    skipped = tuple(
        SkippedTrace(
            trace_id=tid,
            reason=(
                "written by another client; already ingested as the Calibrated Agent "
                "Evaluation trace of the "
                "same id, which is read instead"
                if tid in ingested
                else "written by another client "
                "(no calibrated_agent_evaluation_trace envelope); "
                "`calibrated-agent-evaluation ingest --from-langfuse` brings such traces in"
            ),
        )
        for tid in foreign
    )
    return TraceScan(traces=traces, skipped=skipped)

pull

pull()

Yield every trace in the project, including ones this library did not write.

query reads only the library's own traces: it reconstructs from the calibrated_agent_evaluation_trace envelope and raises on a trace without one, so a single foreign trace aborted the whole read. pull returns each trace in whichever form it can: a Calibrated Agent Evaluation-authored trace reconstructed losslessly as before, a foreign one mapped to OTLP-JSON (langfuse_foreign) for the ingest path to normalize, and a trace that fails either way as a per-trace error.

The listing is read to exhaustion before anything is yielded, so an unreachable or rejecting server raises BackendError before the caller persists a single trace. Re-putting a trace leaves every earlier put's observations server-side; the Calibrated Agent Evaluation-authored path keeps only the latest put's spans, as get_trace does.

Source code in src/calibrated_agent_evaluation/backends/langfuse.py
def pull(self) -> Iterator[SourceTrace]:
    """Yield every trace in the project, including ones this library did not write.

    ``query`` reads only the library's own traces: it reconstructs from the
    ``calibrated_agent_evaluation_trace`` envelope and raises on a trace without one, so a
    single foreign trace aborted the whole read. ``pull`` returns each trace in whichever form
    it can: a Calibrated Agent Evaluation-authored trace reconstructed losslessly as before, a
    foreign one mapped to OTLP-JSON
    (:mod:`~calibrated_agent_evaluation.backends.langfuse_foreign`) for the ingest path to
    normalize, and a trace that fails either way as a per-trace ``error``.

    The listing is read to exhaustion before anything is yielded, so an unreachable or
    rejecting server raises ``BackendError`` before the caller persists a single trace.
    Re-putting a trace leaves every earlier put's observations server-side; the
    Calibrated Agent Evaluation-authored path keeps only the latest put's spans, as
    ``get_trace`` does.
    """
    from calibrated_agent_evaluation.backends.langfuse_foreign import foreign_trace_to_otlp

    try:
        items = list(self._list_all_traces([]))
    except Exception as exc:
        raise self._as_backend_error(exc, "pull") from exc
    for item in items:
        trace_id = str(item.id)
        try:
            raw = self._client.api.trace.get(trace_id)
            if self._read_metadata(raw).get(_TRACE_ENVELOPE):
                yield SourceTrace(source_id=trace_id, trace=self._reconstruct_trace(raw))
            else:
                yield SourceTrace(source_id=trace_id, otlp=foreign_trace_to_otlp(raw))
        # One unreadable trace is reported as its own error, never fatal to the pull; the
        # error is the message the caller shows, so nothing is swallowed.
        except Exception as exc:  # noqa: BLE001
            error = self._as_backend_error(exc, f"pull({trace_id!r})")
            yield SourceTrace(source_id=trace_id, error=str(error))

project

project()

The (id, name) of the project these credentials address.

Langfuse API keys are project-scoped and the public API has no project selector: the project is whichever one the key pair belongs to (verified: api.projects.get() returns exactly that project for a project key, langfuse==3.15.0 against server 3.202.1). So the caller can check that it is reading the project it meant, but cannot choose another.

Source code in src/calibrated_agent_evaluation/backends/langfuse.py
def project(self) -> tuple[str, str]:
    """The ``(id, name)`` of the project these credentials address.

    Langfuse API keys are project-scoped and the public API has no project selector: the
    project is whichever one the key pair belongs to (verified: ``api.projects.get()`` returns
    exactly that project for a project key, langfuse==3.15.0 against server 3.202.1). So the
    caller can check that it is reading the project it meant, but cannot choose another.
    """
    try:
        projects = list(self._client.api.projects.get().data)
    except Exception as exc:
        raise self._as_backend_error(exc, "project lookup") from exc
    if len(projects) != 1:
        raise BackendError(
            f"Langfuse credentials resolve to {len(projects)} projects; use a project-scoped "
            "key pair, which addresses exactly one"
        )
    return str(projects[0].id), str(projects[0].name)

get_scores

get_scores(trace_id, span_id=None)

Return the labels stored for a trace, oldest first, retracted and duplicates removed.

Parameters:

Name Type Description Default
trace_id str

The trace whose labels to read.

required
span_id str | None

Narrow the result to labels on this span.

None

Returns:

Type Description
list[Label]

The labels; an empty list when the trace has none.

Raises:

Type Description
BackendError

Langfuse can't be read, or a stored score isn't a Calibrated Agent Evaluation label.

Source code in src/calibrated_agent_evaluation/backends/langfuse.py
def get_scores(self, trace_id: str, span_id: str | None = None) -> list[Label]:
    """Return the labels stored for a trace, oldest first, retracted and duplicates removed.

    Args:
        trace_id: The trace whose labels to read.
        span_id: Narrow the result to labels on this span.

    Returns:
        The labels; an empty list when the trace has none.

    Raises:
        BackendError: Langfuse can't be read, or a stored score isn't a Calibrated Agent
            Evaluation label.
    """
    lf_trace_id = self._client.create_trace_id(seed=trace_id)
    try:
        # Walk every page; score_v_2.get paginates (data + meta.total_pages, verified shape
        # langfuse==3.15.0), so reading only page 1 would silently drop a heavily-labeled
        # trace's later labels (violates "no silent truncation"), mirroring query()/_list_all.
        # Reconstruction stays INSIDE the try so a partial envelope surfaces as a typed
        # BackendError, not a bare KeyError escaping the CalibratedAgentEvaluationDataError
        # family.
        puts: list[tuple[Label, str]] = []
        retracted_at: dict[str, str] = {}
        for score in self._list_all_scores(lf_trace_id):
            meta = self._read_metadata(score)
            written = str(meta.get(_WRITTEN_AT) or "")
            if meta.get(_LABEL_ENVELOPE) is not None:
                puts.append((self._reconstruct_label(score), written))
            elif meta.get(_RETRACTION_ENVELOPE) is not None:
                key = label_identity(label_from_dict(meta[_RETRACTION_ENVELOPE]))
                retracted_at[key] = max(retracted_at.get(key, ""), written)
    except Exception as exc:
        raise self._as_backend_error(exc, f"get_scores({trace_id!r})") from exc
    # Last write wins per label: a put survives only if it is newer than every tombstone for
    # the same identity. Scores carry no server order we read, so the order is the stamped time,
    # and the result is returned oldest first like LocalBackend's, which `dedup_labels` and
    # `latest_verdicts` rely on to keep the newest write. The sort is stable for equal stamps.
    puts.sort(key=lambda put: put[1])
    labels = [
        lbl
        for lbl, written in puts
        if (key := label_identity(lbl)) not in retracted_at or written > retracted_at[key]
    ]
    if span_id is not None:
        labels = [lbl for lbl in labels if lbl.span_id == span_id]
    # Same append-only root cause as `_reconstruct_trace`'s span dedup: `create_score` has no
    # update, so a re-`put_score` leaves N scores server-side for one label. Keep the last.
    return dedup_labels(labels)

scan_traces

scan_traces(
    backend, *, frame=None, cell=None, ownership=None
)

Read every matching trace from backend, reporting the ones it leaves out.

A backend that can hold foreign traces (ScanningBackend, e.g. LangfuseBackend) scans; any other returns everything query does and skips nothing.

Source code in src/calibrated_agent_evaluation/backends/base.py
def scan_traces(
    backend: TraceStoreBackend,
    *,
    frame: Frame | None = None,
    cell: dict[str, str] | None = None,
    ownership: Ownership | None = None,
) -> TraceScan:
    """Read every matching trace from ``backend``, reporting the ones it leaves out.

    A backend that can hold foreign traces (:class:`ScanningBackend`, e.g. ``LangfuseBackend``)
    scans; any other returns everything ``query`` does and skips nothing.
    """
    if isinstance(backend, ScanningBackend):
        return backend.scan(frame=frame, cell=cell, ownership=ownership)
    if frame is None and cell is None and ownership is None:
        return TraceScan(traces=backend.query())
    return TraceScan(traces=backend.query(frame=frame, cell=cell, ownership=ownership))

backend_session

backend_session(
    configured=None, *, local_root=DEFAULT_LOCAL_ROOT
)

get_backend with the lifetime bound to the block.

Closes only what it created. A configured backend was opened by the caller and is left open; closing someone else's handle is the mirror-image bug of leaking your own.

Parameters:

Name Type Description Default
configured TraceStoreBackend | None

A backend the caller already built; yielded and left open.

None
local_root str | Path

The directory of the default local store.

DEFAULT_LOCAL_ROOT

Yields:

Type Description
TraceStoreBackend

The backend to use inside the block.

Source code in src/calibrated_agent_evaluation/backends/factory.py
@contextmanager
def backend_session(
    configured: TraceStoreBackend | None = None,
    *,
    local_root: str | Path = DEFAULT_LOCAL_ROOT,
) -> Iterator[TraceStoreBackend]:
    """:func:`get_backend` with the lifetime bound to the block.

    Closes only what it created. A ``configured`` backend was opened by the caller and is left
    open; closing someone else's handle is the mirror-image bug of leaking your own.

    Args:
        configured: A backend the caller already built; yielded and left open.
        local_root: The directory of the default local store.

    Yields:
        The backend to use inside the block.
    """
    backend = get_backend(configured, local_root=local_root)
    if configured is not None:
        yield backend
        return
    closer = getattr(backend, "close", None)
    try:
        yield backend
    finally:
        if callable(closer):
            closer()

get_backend

get_backend(
    configured=None, *, local_root=DEFAULT_LOCAL_ROOT
)

Return the backend to use.

If configured is given it is returned unchanged (no silent fallback to local). Otherwise a LocalBackend rooted at local_root is returned. Constructing the LocalBackend against an unwritable root raises (the filesystem error surfaces) rather than silently no-op'ing.

The caller owns the returned backend's lifetime. A default LocalBackend holds an open sqlite3 connection, so a caller that neither uses it as a context manager nor calls close() leaks one per call. Use backend_session where the scope is a block; this function stays available for the callers that hand the backend onward.

Parameters:

Name Type Description Default
configured TraceStoreBackend | None

A backend the caller already built; returned as is.

None
local_root str | Path

The directory of the default local store.

DEFAULT_LOCAL_ROOT

Returns:

Type Description
TraceStoreBackend

configured, or a new LocalBackend at local_root.

Source code in src/calibrated_agent_evaluation/backends/factory.py
def get_backend(
    configured: TraceStoreBackend | None = None,
    *,
    local_root: str | Path = DEFAULT_LOCAL_ROOT,
) -> TraceStoreBackend:
    """Return the backend to use.

    If ``configured`` is given it is returned unchanged (no silent fallback to local).
    Otherwise a ``LocalBackend`` rooted at ``local_root`` is returned. Constructing
    the ``LocalBackend`` against an unwritable root raises (the filesystem error surfaces)
    rather than silently no-op'ing.

    **The caller owns the returned backend's lifetime.** A default ``LocalBackend`` holds an open
    ``sqlite3`` connection, so a caller that neither uses it as a context manager nor calls
    ``close()`` leaks one per call. Use :func:`backend_session` where the scope is a block; this
    function stays available for the callers that hand the backend onward.

    Args:
        configured: A backend the caller already built; returned as is.
        local_root: The directory of the default local store.

    Returns:
        ``configured``, or a new ``LocalBackend`` at ``local_root``.
    """
    if configured is not None:
        return configured
    return LocalBackend(root=local_root)