Read from Langfuse¶
If your team already sends agent traces to Langfuse, Calibrated Agent
Evaluation can read them straight from the project. It works with an unmodified Langfuse v3
deployment, self-hosted or not, and needs the langfuse extra.
uv sync --inexact --extra langfuse
source .venv/bin/activate # puts `calibrated-agent-evaluation` on your PATH
export LANGFUSE_HOST="https://langfuse.your-company.internal" # default: http://localhost:3000
export LANGFUSE_PUBLIC_KEY="pk-lf-..."
export LANGFUSE_SECRET_KEY="sk-lf-..."
calibrated-agent-evaluation ingest --from-langfuse "My Project"
That reads every trace in the project, normalizes each one and writes the result to the local
store (./.calibrated-agent-evaluation). The traces in Langfuse are not changed.
Without the keys, the command stops before contacting anything:
$ calibrated-agent-evaluation ingest --from-langfuse "My Project"
calibrated-agent-evaluation ingest: the langfuse backend needs LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY in the environment (see https://awslabs.github.io/calibrated-agent-evaluation/integrate/langfuse/)
Why the project name is a check¶
Langfuse keys belong to one project, and its API cannot read another. So --from-langfuse takes
the project name (or id) as a check rather than a choice: if it does not match the project the
keys belong to, the command stops and names the project the keys do belong to. That catches a key
pasted from the wrong project before a single trace is read.
What it reads¶
Traces your application wrote to Langfuse, with the Langfuse SDK or with an OpenTelemetry exporter pointed at Langfuse's OTLP endpoint, are read like a trace file:
- each Langfuse observation becomes a span, and the OpenTelemetry attributes it recorded are kept;
- without those attributes, an
agentobservation is an agent step, atoolobservation a tool call and agenerationan LLM call; - a trace with no
agentobservation treats its outermost observation as the agent.
The same checks as file ingest apply. A trace whose tool calls cannot be attached to an agent is refused, and a trace that cannot be read at all is reported as skipped while the rest of the project still comes in. The command exits 1 if anything was refused or skipped.
Traces the library itself stored in Langfuse come back exactly as they were written.
Use Langfuse as the store¶
--backend langfuse makes Langfuse the store every command reads and writes, instead of the local
directory. It is a global flag, so it goes before the subcommand:
calibrated-agent-evaluation --backend langfuse ingest exports/ # ingest files into Langfuse
calibrated-agent-evaluation --backend langfuse run --ci # gate over the verdicts stored in Langfuse
An unreachable host fails at once with an error that names the host, before any command reads an empty result from it. Langfuse writes are asynchronous; a command waits for each write to become readable before the next command depends on it.
From Python:
# needs the langfuse extra
from calibrated_agent_evaluation.backends import LangfuseBackend
with LangfuseBackend.from_credentials(
public_key="pk-lf-...",
secret_key="sk-lf-...",
host="https://langfuse.your-company.internal",
) as backend:
trace = backend.get_trace("ORD-1001")
Try it against a local Langfuse¶
The repository ships a stock Langfuse v3 stack for development under docker/langfuse/, with
telemetry off and a provisioned dev project:
just langfuse-up # web UI on http://localhost:3000
export LANGFUSE_PUBLIC_KEY=pk-lf-calibrated-agent-evaluation-dev LANGFUSE_SECRET_KEY=sk-lf-calibrated-agent-evaluation-dev
calibrated-agent-evaluation --backend langfuse ingest fixtures/raw/strands
The dev keys in docker/langfuse/.env.example are throwaway values for a local instance only.
Next¶
- Ingest trace files for exports that never reach Langfuse.
- Read and label traces for what to do with the traces once they are in.