calibrated-agent-evaluation ingest¶
normalize trace dumps (OTLP / Inspect .eval / JSONL / parquet) into the store; refuses lossy ones
usage: calibrated-agent-evaluation ingest [-h] [--from-langfuse PROJECT]
[--format {auto,strands,otel,inspect,jsonl,parquet,hf}]
[--convention {auto,gen_ai/latest,gen_ai/legacy,gen_ai/messages,openllmetry,openinference}]
[--dry-run] [--require-owned]
[--frame {discovery,production-representative}]
[--overwrite] [--allow-partial-tools] [--sample N]
[--seed SEED] [--cell KEY=VALUE] [--source {agentcore}]
[--runtime RUNTIME] [--region REGION]
[--profile PROFILE] [--omni] [--since SINCE]
[--until UNTIL] [--rekey-sessions] [--with-scores]
[--backend {local,langfuse}] [--root PATH] [--explain]
[paths ...]
Normalize trace files, directories (walked recursively) or a Langfuse project into the store.
--backend selects the destination and goes before the subcommand: calibrated-agent-evaluation
--backend langfuse ingest exports/. --sample N --seed S ingests a seeded random sample of the
population and tags it production-representative, which is what lets a headline rate be computed
over it.
positional arguments:
paths trace files or directories to ingest
options:
-h, --help show this help message and exit
--from-langfuse PROJECT
pull every trace from a Langfuse project instead of files, including
traces Calibrated Agent Evaluation did not write; PROJECT (name or id)
must be the project the LANGFUSE_* keys belong to, since Langfuse keys are
project-scoped
--format {auto,strands,otel,inspect,jsonl,parquet,hf}
source format (default: auto-detect from the file shape); jsonl / parquet
/ hf read a dataset, one trace per record, and hf takes HuggingFace repo
ids as the paths
--convention {auto,gen_ai/latest,gen_ai/legacy,gen_ai/messages,openllmetry,openinference}
the OTel GenAI convention to read OTLP sources with (default: detect it);
for a trace whose signals match more than one
--dry-run report what would be ingested and refused; write nothing to the backend
--require-owned refuse a trace that degrades to outcome-only (ownership='probed'), for a
corpus that will be analyzed step by step
--frame {discovery,production-representative}
sampling frame for traces whose spans carry no
calibrated_agent_evaluation.frame attribute (default: discovery;
production-representative is the frame that authorizes a headline rate, so
it is only ever set on purpose)
--overwrite replace a stored trace with the same id but different content (default:
refuse, since its labels would be re-attached to the new content)
--allow-partial-tools
ingest a trace that recovered only some of its source tool calls,
reporting the shortfall (default: refuse it)
--sample N ingest a seeded random sample of N traces instead of every one, tagged
production-representative; needs --seed, and records the seed and the
population size
--seed SEED the seed for --sample; the same seed and population always draw the same
sample
--cell KEY=VALUE with --sample --from-langfuse, draw from the traces whose taxonomy cell
has KEY=VALUE (repeatable); one cell at a time is a stratified sample
--source {agentcore} pull from a managed source instead of files/Langfuse; 'agentcore' reads
Amazon Bedrock AgentCore telemetry into the --backend store (needs the
[agentcore] extra)
--runtime RUNTIME the AgentCore Runtime's OpenTelemetry service.name, which for the default
endpoint is <agent>.DEFAULT, not the runtime id (required with --source
agentcore)
--region REGION AWS region (default: the profile's)
--profile PROFILE AWS profile (default: AWS_PROFILE)
--omni read spans via CloudWatch Omni SQL instead of Logs Insights (needs the
extra)
--since SINCE start of the read window as ISO-8601 (required with --source agentcore)
--until UNTIL end of the read window as ISO-8601 (required with --source agentcore)
--rekey-sessions collapse a mixed-session AgentCore trace to the runtime session id
(default: refuse it)
--with-scores also read AgentCore evaluator results as advisory labels
--backend {local,langfuse}
trace-store backend (default: local, offline)
--root PATH local root for the trace store and the stage records (default:
$CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, else ./.calibrated-agent-
evaluation; quickstart defaults to a temp dir)
--explain after the plain summary, print the statistics behind it (bounds, methods,
thresholds, record paths); CALIBRATED_AGENT_EVALUATION_EXPLAIN=1 turns it
on for every command