Skip to content

calibrated-agent-evaluation ingest

normalize trace dumps (OTLP / Inspect .eval / JSONL / parquet) into the store; refuses lossy ones

usage: calibrated-agent-evaluation ingest [-h] [--from-langfuse PROJECT]
                                          [--format {auto,strands,otel,inspect,jsonl,parquet,hf}]
                                          [--convention {auto,gen_ai/latest,gen_ai/legacy,gen_ai/messages,openllmetry,openinference}]
                                          [--dry-run] [--require-owned]
                                          [--frame {discovery,production-representative}]
                                          [--overwrite] [--allow-partial-tools] [--sample N]
                                          [--seed SEED] [--cell KEY=VALUE] [--source {agentcore}]
                                          [--runtime RUNTIME] [--region REGION]
                                          [--profile PROFILE] [--omni] [--since SINCE]
                                          [--until UNTIL] [--rekey-sessions] [--with-scores]
                                          [--backend {local,langfuse}] [--root PATH] [--explain]
                                          [paths ...]

Normalize trace files, directories (walked recursively) or a Langfuse project into the store.
--backend selects the destination and goes before the subcommand: calibrated-agent-evaluation
--backend langfuse ingest exports/. --sample N --seed S ingests a seeded random sample of the
population and tags it production-representative, which is what lets a headline rate be computed
over it.

positional arguments:
  paths                 trace files or directories to ingest

options:
  -h, --help            show this help message and exit
  --from-langfuse PROJECT
                        pull every trace from a Langfuse project instead of files, including
                        traces Calibrated Agent Evaluation did not write; PROJECT (name or id)
                        must be the project the LANGFUSE_* keys belong to, since Langfuse keys are
                        project-scoped
  --format {auto,strands,otel,inspect,jsonl,parquet,hf}
                        source format (default: auto-detect from the file shape); jsonl / parquet
                        / hf read a dataset, one trace per record, and hf takes HuggingFace repo
                        ids as the paths
  --convention {auto,gen_ai/latest,gen_ai/legacy,gen_ai/messages,openllmetry,openinference}
                        the OTel GenAI convention to read OTLP sources with (default: detect it);
                        for a trace whose signals match more than one
  --dry-run             report what would be ingested and refused; write nothing to the backend
  --require-owned       refuse a trace that degrades to outcome-only (ownership='probed'), for a
                        corpus that will be analyzed step by step
  --frame {discovery,production-representative}
                        sampling frame for traces whose spans carry no
                        calibrated_agent_evaluation.frame attribute (default: discovery;
                        production-representative is the frame that authorizes a headline rate, so
                        it is only ever set on purpose)
  --overwrite           replace a stored trace with the same id but different content (default:
                        refuse, since its labels would be re-attached to the new content)
  --allow-partial-tools
                        ingest a trace that recovered only some of its source tool calls,
                        reporting the shortfall (default: refuse it)
  --sample N            ingest a seeded random sample of N traces instead of every one, tagged
                        production-representative; needs --seed, and records the seed and the
                        population size
  --seed SEED           the seed for --sample; the same seed and population always draw the same
                        sample
  --cell KEY=VALUE      with --sample --from-langfuse, draw from the traces whose taxonomy cell
                        has KEY=VALUE (repeatable); one cell at a time is a stratified sample
  --source {agentcore}  pull from a managed source instead of files/Langfuse; 'agentcore' reads
                        Amazon Bedrock AgentCore telemetry into the --backend store (needs the
                        [agentcore] extra)
  --runtime RUNTIME     the AgentCore Runtime's OpenTelemetry service.name, which for the default
                        endpoint is <agent>.DEFAULT, not the runtime id (required with --source
                        agentcore)
  --region REGION       AWS region (default: the profile's)
  --profile PROFILE     AWS profile (default: AWS_PROFILE)
  --omni                read spans via CloudWatch Omni SQL instead of Logs Insights (needs the
                        extra)
  --since SINCE         start of the read window as ISO-8601 (required with --source agentcore)
  --until UNTIL         end of the read window as ISO-8601 (required with --source agentcore)
  --rekey-sessions      collapse a mixed-session AgentCore trace to the runtime session id
                        (default: refuse it)
  --with-scores         also read AgentCore evaluator results as advisory labels
  --backend {local,langfuse}
                        trace-store backend (default: local, offline)
  --root PATH           local root for the trace store and the stage records (default:
                        $CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, else ./.calibrated-agent-
                        evaluation; quickstart defaults to a temp dir)
  --explain             after the plain summary, print the statistics behind it (bounds, methods,
                        thresholds, record paths); CALIBRATED_AGENT_EVALUATION_EXPLAIN=1 turns it
                        on for every command