calibrated_agent_evaluation.backends¶
calibrated_agent_evaluation.backends ¶
Data-plane backends for Calibrated Agent Evaluation.
The TraceStoreBackend / LabelBackend protocols are defined against the
normalized schema only; no vendor type appears in any method signature.
LocalBackend (stdlib sqlite3 + CSV/JSON) is the no-config offline default;
LangfuseBackend is the opt-in both-worlds adapter (the langfuse extra).
LangfuseBackend and the dashboard sinks are served lazily via __getattr__ (mirroring
execute/__init__) so import calibrated_agent_evaluation.backends on a bare offline install
pulls neither the Langfuse vendor SDK nor the run/calibrate stats chain the dashboard sinks
import (scipy/pyarrow). This keeps the offline CLI path zero-third-party at import time
the light stdlib-only LocalBackend + get_backend are still eager.
LabelBackend ¶
Bases: Protocol
A human-labeling work-list source/sink, defined against normalized Label only.
No implementation ships: this is the managed-labeling seam, deferred until a managed
labeling backend is built. The review app reads its work list from a
TraceStoreBackend and writes labels with put_score instead. The interface is fixed so a
future labeling backend speaks the normalized schema, never a vendor labeling type.
next_item ¶
ScanningBackend ¶
Bases: Protocol
A store that can hold traces it cannot return as normalized Traces.
scan ¶
query, but a trace it cannot return is a SkippedTrace, not a raise.
Source code in src/calibrated_agent_evaluation/backends/base.py
SkippedTrace
pydantic-model
¶
SourceTrace
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
One trace read from a source store, in the form the store could give it.
Exactly one of trace, otlp and error is set:
trace: the store holds a Calibrated Agent Evaluation-normalized trace and returned it losslessly;otlp: the store holds a trace something else wrote, mapped to a canonical OTLP-JSON document for the ingest path to detect and normalize like a file;error: this one trace could not be read, and why. The ingest path reports it as skipped and carries on, because one unreadable trace is no reason to drop the rest of the pull.
Plain types only, so a vendor shape never crosses into the ingest plane.
Config:
frozen:Trueextra:forbid
Fields:
-
source_id(str) -
trace(Trace | None) -
otlp(dict[str, Any] | None) -
error(str | None) -
session_id(str | None)
Validators:
-
_exactly_one_form
TraceScan
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
A whole-store read: the traces read, and every trace left out with its reason.
skipped is how a read stays honest about what it did not return. A store shared with
another client holds traces Calibrated Agent Evaluation did not write; a stage reads the rest
and says how many it left out and why, instead of aborting on the first foreign trace or
dropping it unseen.
Config:
frozen:Trueextra:forbid
Fields:
TraceSource ¶
Bases: Protocol
A store that can be pulled from, including traces it did not normalize itself.
pull ¶
Yield every trace in the store as a SourceTrace.
Raises BackendError before yielding anything if the store cannot be listed (unreachable
or rejected credentials); a failure on one trace is a SourceTrace.error, not a raise.
Source code in src/calibrated_agent_evaluation/backends/base.py
TraceStoreBackend ¶
Bases: Protocol
Persist and retrieve normalized traces and their labels through one interface.
The methodology plane is written once against this protocol and works regardless of which concrete backend is configured.
put_trace ¶
get_trace ¶
Return the trace for trace_id, equal field-for-field to what was written.
Raises BackendError if no such trace exists or the backend is unreachable.
query ¶
Return the traces matching the given queryable dimensions.
The locked query surface; frame / cell / ownership only; richer
querying is out of scope. An empty match returns [].
Source code in src/calibrated_agent_evaluation/backends/base.py
put_score ¶
get_scores ¶
retract_score ¶
Retract a stored label, so no later get_scores returns it.
label is matched by full content (schema.serde.label_identity), the same identity
the read-side dedup uses, so exactly that label goes and a different verdict on the same
target stays. Putting the same label again afterwards stores it again. Retracting a label
that is not (yet) readable is not an error: an eventually-consistent store cannot tell
"absent" from "still ingesting", and the retraction must win over a put that lands late.
Raises BackendError if the backend is unreachable or the retraction is not written.
Source code in src/calibrated_agent_evaluation/backends/base.py
LocalBackend ¶
A TraceStoreBackend backed by stdlib sqlite3 and the local filesystem.
Spans are stored in their own table (for the indexed tree reconstruction) but are also
embedded in the trace doc blob, which is the source of truth on read. Use it as a context
manager, or call close, to release the database connection.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
root
|
str | Path
|
The directory holding the store, created if missing. |
'.calibrated-agent-evaluation'
|
Source code in src/calibrated_agent_evaluation/backends/local.py
put_trace ¶
Store trace and its spans, replacing any trace with the same trace_id.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
trace
|
Trace
|
The normalized trace to store. |
required |
Raises:
| Type | Description |
|---|---|
SchemaError
|
The trace fails validation; nothing is written. |
Source code in src/calibrated_agent_evaluation/backends/local.py
get_trace ¶
Return the stored trace for trace_id, field for field as it was written.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
trace_id
|
str
|
The id of the trace to read. |
required |
Returns:
| Type | Description |
|---|---|
Trace
|
The stored trace. |
Raises:
| Type | Description |
|---|---|
TraceNotFoundError
|
No trace with that id is stored. |
Source code in src/calibrated_agent_evaluation/backends/local.py
query ¶
Return the stored traces that match every given filter, ordered by trace_id.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
frame
|
Frame | None
|
Keep only traces in this sampling frame. |
None
|
cell
|
dict[str, str] | None
|
Keep only traces whose taxonomy cell has all of these key/value pairs. |
None
|
ownership
|
Ownership | None
|
Keep only owned or only probed traces. |
None
|
Returns:
| Type | Description |
|---|---|
list[Trace]
|
The matching traces; an empty list when none match. |
Source code in src/calibrated_agent_evaluation/backends/local.py
put_score ¶
Append a label. Identity-duplicate rows are collapsed by dedup_labels on read.
Deliberately an append rather than an upsert. labels.rowid_pk is an autoincrement primary
key and nothing in the existing schema carries the identity, so an upsert needs either a
migration of every store on disk or an identity column that the Langfuse backend could not
mirror anyway (its create_score API has no update). Deduping on read gives both backends
one behaviour and keeps every already-written store readable.
rater_id is projected into its own column, like trace_id/span_id, but stays out
of the (trace_id, span_id) index: every read filters on the target, none on the rater,
and the index is a lookup path, not a uniqueness key. Two raters' labels on one target are
kept apart by the full-content identity dedup_labels applies, which includes
rater_id, so widening the index would change no result.
Source code in src/calibrated_agent_evaluation/backends/local.py
get_scores ¶
Return the labels stored for a trace, oldest first, with duplicates collapsed.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
trace_id
|
str
|
The trace whose labels to read. |
required |
span_id
|
str | None
|
Narrow the result to labels on this span. |
None
|
Returns:
| Type | Description |
|---|---|
list[Label]
|
The labels; an empty list when the trace has none. |
Source code in src/calibrated_agent_evaluation/backends/local.py
retract_score ¶
Delete every stored row carrying exactly label (see TraceStoreBackend).
A real delete, because SQLite's is synchronous and exact. Every byte-identical row goes: an idempotent re-put leaves several rows that read back as one label, and leaving any of them would make the retraction a no-op on read. Retracting a label that is not stored deletes nothing, matching the Langfuse backend, which cannot check synchronously.
Source code in src/calibrated_agent_evaluation/backends/local.py
all_traces ¶
Return every stored trace (dump support).
all_scores ¶
Return every stored label (dump support).
Source code in src/calibrated_agent_evaluation/backends/local.py
wipe ¶
Delete every artifact (the dump→wipe→load no-lock-in proof).
Source code in src/calibrated_agent_evaluation/backends/local.py
DashboardSink ¶
LangfuseDashboardSink ¶
LangfuseDashboardSink(
client,
*,
local=None,
flush_timeout_seconds=_DEFAULT_FLUSH_TIMEOUT_SECONDS,
)
Publishes the numeric series to Langfuse as numeric scores for dashboard rendering.
Mirrors LangfuseBackend: publish
calls auth_check first and raises BackendError on an unreachable or rejecting
backend before writing, and every score is a synchronous REST create, so a score the server
refuses raises instead of vanishing. The local sink (if provided) is written first and is
authoritative; the Langfuse leg is additive. Client injected for tests; from_credentials
builds the real MIT SDK client.
Source code in src/calibrated_agent_evaluation/backends/dashboard.py
publish ¶
Publish the numeric series to Langfuse (and the local sink first, if configured).
Source code in src/calibrated_agent_evaluation/backends/dashboard.py
from_credentials
classmethod
¶
from_credentials(
public_key,
secret_key,
host,
*,
timeout_seconds=5,
local=None,
client_factory=None,
)
Build against the real MIT SDK client (or an injected factory in tests).
timeout_seconds bounds every request including the auth_check probe (the SDK accepts
timeout in seconds, verified langfuse==3.15.0), so an unreachable backend fails fast.
Source code in src/calibrated_agent_evaluation/backends/dashboard.py
LocalDashboardSink ¶
The authoritative file sink: numeric series -> CSV+JSON, round-trippable.
Offline, dependency-free, identical in both worlds. Writes both files under directory;
load reads the JSON back field-for-field. Any filesystem/parse failure is wrapped into
RunStoreError (no raw OSError/JSONDecodeError escapes -- "no silent failure").
Source code in src/calibrated_agent_evaluation/backends/dashboard.py
publish ¶
Write the numeric series to dashboard_series.{csv,json} under the directory.
Source code in src/calibrated_agent_evaluation/backends/dashboard.py
load ¶
Read the numeric series back field-for-field (the round-trip).
Source code in src/calibrated_agent_evaluation/backends/dashboard.py
LangfuseBackend ¶
A TraceStoreBackend over an unmodified Langfuse v3 deployment.
Writes block until the Langfuse client confirms its queue drained, so a returned write is
durable. Build one with from_credentials; the constructor takes an injected client.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
client
|
LangfuseLike
|
A Langfuse v3 client, or a stand-in with the same surface. |
required |
telemetry_disabled
|
bool
|
Recorded for provenance; the client never sends usage telemetry. |
True
|
flush_timeout_seconds
|
float
|
How long a write waits for the client's queue to drain before it
fails with |
20.0
|
export_tap
|
ExportTap | None
|
Counts the client's failed span exports, so a trace write whose export
failed raises instead of returning. |
None
|
Source code in src/calibrated_agent_evaluation/backends/langfuse.py
from_credentials
classmethod
¶
from_credentials(
public_key,
secret_key,
host,
*,
timeout_seconds=5,
flush_timeout_seconds=20.0,
client_factory=None,
)
Construct against the OSS build with telemetry disabled.
Telemetry-off is enforced in two real places, NOT by the client alone: (a) the OSS
server reads TELEMETRY_ENABLED (set in docker/langfuse/docker-compose.yml), and
(b) this adapter restricts itself to the MIT/OSS public surface; the EE_GATED_METHODS
denylist (asserted in tests) is the client-side guarantee that no enterprise-licensed
feature is invoked. The v3 Python client does NOT read TELEMETRY_ENABLED (it only
reads LANGFUSE_TRACING_ENABLED / the tracing_enabled kwarg, which gate our own
span writes; flipping those would break put_trace, so we must not). We still set the
server-recognized TELEMETRY_ENABLED=false in-process while the client is built; a
harmless belt-and-suspenders for a co-located server that reads the process env; and
restore the prior value after, so constructing a backend has no lasting global side effect.
client_factory is an injection seam for tests; in production it imports the real MIT
SDK. The real langfuse.Langfuse client structurally satisfies LangfuseLike
at the call sites the adapter uses (verified surface).
timeout_seconds bounds every API request; including the synchronous reachability
probe (auth_check); so an unreachable backend fails fast instead of hanging. The
real SDK accepts timeout (seconds; verified langfuse==3.15.0) and defaults it to 5;
we surface it as an explicit, tunable parameter rather than rely on that default.
flush_timeout_seconds separately bounds the blocking flush() after a write; the
SDK's flush does an unbounded queue.join() (verified langfuse==3.15.0) that hangs
forever if the ingestion queue wedges, so _flush_bounded caps it and raises a typed
BackendError on timeout rather than blocking the caller indefinitely.
Source code in src/calibrated_agent_evaluation/backends/langfuse.py
check_reachable ¶
Check that Langfuse can be reached and accepts these credentials, writing nothing.
It is the synchronous probe every write runs first: the client's auth_check, bounded
by its request timeout. Run it before costly work whose results go here, so an unreachable
host or refused keys stop the work before it starts.
Raises:
| Type | Description |
|---|---|
BackendError
|
Langfuse can't be reached, or it refuses the credentials. |
Source code in src/calibrated_agent_evaluation/backends/langfuse.py
put_trace ¶
Write trace and its span tree to Langfuse, parents before children.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
trace
|
Trace
|
The normalized trace to store. |
required |
Raises:
| Type | Description |
|---|---|
SchemaError
|
The trace fails validation, or a span names a parent that isn't in the trace; nothing is written. |
BackendError
|
Langfuse is unreachable, rejects the write, or doesn't confirm it within
|
Source code in src/calibrated_agent_evaluation/backends/langfuse.py
put_score ¶
Write label to Langfuse as a score on its trace (and span, if it names one).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
label
|
Label
|
The label to store. |
required |
Raises:
| Type | Description |
|---|---|
SchemaError
|
The label fails validation; nothing is written. |
BackendError
|
Langfuse is unreachable, rejects the write, or doesn't confirm it. |
Source code in src/calibrated_agent_evaluation/backends/langfuse.py
retract_score ¶
Retract label by appending a tombstone score (see TraceStoreBackend).
get_scores then hides every put of this label stamped before the tombstone, including
one still in Langfuse's ingestion queue; a later put of the same label is visible again.
There is deliberately no read-first existence check: score ingestion is asynchronous
(verified live, a score takes seconds to become readable), so an undo pressed right after
a verdict would find nothing and refuse to retract a label that is about to appear.
Source code in src/calibrated_agent_evaluation/backends/langfuse.py
get_trace ¶
Read the trace Calibrated Agent Evaluation wrote under trace_id back from Langfuse.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
trace_id
|
str
|
The Calibrated Agent Evaluation trace id (not the Langfuse id derived from it). |
required |
Returns:
| Type | Description |
|---|---|
Trace
|
The trace, rebuilt from the envelope stored with it. |
Raises:
| Type | Description |
|---|---|
TraceNotFoundError
|
Langfuse has no trace for that id. |
BackendError
|
Langfuse can't be read, or the stored trace can't be rebuilt. |
Source code in src/calibrated_agent_evaluation/backends/langfuse.py
query ¶
Return every stored trace matching all given filters, reading every result page.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
frame
|
Frame | None
|
Keep only traces in this sampling frame. |
None
|
cell
|
dict[str, str] | None
|
Keep only traces whose taxonomy cell has all of these key/value pairs. |
None
|
ownership
|
Ownership | None
|
Keep only owned or only probed traces. |
None
|
Returns:
| Type | Description |
|---|---|
list[Trace]
|
The matching traces. |
Raises:
| Type | Description |
|---|---|
BackendError
|
Langfuse can't be read, or a listed trace can't be rebuilt. Use
|
Source code in src/calibrated_agent_evaluation/backends/langfuse.py
scan ¶
query, with every trace Calibrated Agent Evaluation did not write skipped and named.
A project shared with another client holds traces without the
calibrated_agent_evaluation_trace envelope, and query raises on the first one. A
pipeline stage reads with scan instead: the library's traces come back as query
returns them, and each foreign one is a SkippedTrace whose reason says how to bring
it in (calibrated-agent-evaluation ingest --from-langfuse, which maps it through the
path and its gates) or that it already was, under the same id. Skipped, not mapped on the
fly: a stage would otherwise score a trace that has passed none of the ingest gates, under
span ids no stored label can name. A Calibrated Agent Evaluation trace that fails to
reconstruct still raises, since that is a corrupt copy of our own data, not someone else's.
Source code in src/calibrated_agent_evaluation/backends/langfuse.py
pull ¶
Yield every trace in the project, including ones this library did not write.
query reads only the library's own traces: it reconstructs from the
calibrated_agent_evaluation_trace envelope and raises on a trace without one, so a
single foreign trace aborted the whole read. pull returns each trace in whichever form
it can: a Calibrated Agent Evaluation-authored trace reconstructed losslessly as before, a
foreign one mapped to OTLP-JSON
(langfuse_foreign) for the ingest path to
normalize, and a trace that fails either way as a per-trace error.
The listing is read to exhaustion before anything is yielded, so an unreachable or
rejecting server raises BackendError before the caller persists a single trace.
Re-putting a trace leaves every earlier put's observations server-side; the
Calibrated Agent Evaluation-authored path keeps only the latest put's spans, as
get_trace does.
Source code in src/calibrated_agent_evaluation/backends/langfuse.py
project ¶
The (id, name) of the project these credentials address.
Langfuse API keys are project-scoped and the public API has no project selector: the
project is whichever one the key pair belongs to (verified: api.projects.get() returns
exactly that project for a project key, langfuse==3.15.0 against server 3.202.1). So the
caller can check that it is reading the project it meant, but cannot choose another.
Source code in src/calibrated_agent_evaluation/backends/langfuse.py
get_scores ¶
Return the labels stored for a trace, oldest first, retracted and duplicates removed.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
trace_id
|
str
|
The trace whose labels to read. |
required |
span_id
|
str | None
|
Narrow the result to labels on this span. |
None
|
Returns:
| Type | Description |
|---|---|
list[Label]
|
The labels; an empty list when the trace has none. |
Raises:
| Type | Description |
|---|---|
BackendError
|
Langfuse can't be read, or a stored score isn't a Calibrated Agent Evaluation label. |
Source code in src/calibrated_agent_evaluation/backends/langfuse.py
scan_traces ¶
Read every matching trace from backend, reporting the ones it leaves out.
A backend that can hold foreign traces (ScanningBackend, e.g. LangfuseBackend)
scans; any other returns everything query does and skips nothing.
Source code in src/calibrated_agent_evaluation/backends/base.py
backend_session ¶
get_backend with the lifetime bound to the block.
Closes only what it created. A configured backend was opened by the caller and is left
open; closing someone else's handle is the mirror-image bug of leaking your own.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
configured
|
TraceStoreBackend | None
|
A backend the caller already built; yielded and left open. |
None
|
local_root
|
str | Path
|
The directory of the default local store. |
DEFAULT_LOCAL_ROOT
|
Yields:
| Type | Description |
|---|---|
TraceStoreBackend
|
The backend to use inside the block. |
Source code in src/calibrated_agent_evaluation/backends/factory.py
get_backend ¶
Return the backend to use.
If configured is given it is returned unchanged (no silent fallback to local).
Otherwise a LocalBackend rooted at local_root is returned. Constructing
the LocalBackend against an unwritable root raises (the filesystem error surfaces)
rather than silently no-op'ing.
The caller owns the returned backend's lifetime. A default LocalBackend holds an open
sqlite3 connection, so a caller that neither uses it as a context manager nor calls
close() leaks one per call. Use backend_session where the scope is a block; this
function stays available for the callers that hand the backend onward.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
configured
|
TraceStoreBackend | None
|
A backend the caller already built; returned as is. |
None
|
local_root
|
str | Path
|
The directory of the default local store. |
DEFAULT_LOCAL_ROOT
|
Returns:
| Type | Description |
|---|---|
TraceStoreBackend
|
|