Skip to content

Troubleshooting

When a command can't give you an answer, it exits 1 and says what to collect or change. A usage mistake exits 2. A Python traceback is a bug; please report it. Add --explain (or set CALIBRATED_AGENT_EVALUATION_EXPLAIN=1) to see the numbers and record paths behind a result. In the library, errors derive from calibrated_agent_evaluation.schema.CalibratedAgentEvaluationDataError; the CLI's derive from calibrated_agent_evaluation.cli.errors.CliError.

AgentCore messages are on Bedrock AgentCore.

Messages you may see and what to do

Message What triggered it What to do
Judge not certified On your labels, the judge's cautious catch rate for failures, or for passes, is below 0.80 Label more traces of the class that is short (the message says how many), or improve the judge
Few-shot pool used as a test set The traces you checked the judge on are the examples in its prompt Check it on a separate set
Labels already used to test a judge This set already decided one judge for this mode Label a fresh set of traces the judge hasn't been checked on
Rate measured on the calibration set The rate sample is the set the judge was checked on Score a random sample of production traffic
Rate sample too small Fewer than 150 traces in the rate sample Score more production traces (the message says how many)
Not enough labels for a corrected rate The judge is certified, but on too few labels for its accuracy to give a reliable range Label more, or plan with --rate-safe next time
Judge too weak to correct The judge's catch rates for failures and passes add up to less than 1.5 Use the range the report still prints, or improve the judge
Flag rate outside what the judge can produce The judge flagged fewer traces than its own false-alarm rate, or more than it catches Check the rate sample; the range is still reported
Judge biased The judge changes its verdict when something irrelevant about the input changes, such as a longer answer Fix the judge prompt, then check it again on fresh labels

Methods gives the rule behind each threshold.

Installation and extras

You see Why Do
an extra is not installed the command needs a third-party package run the command it prints; calibrated-agent-evaluation doctor lists them all (Install)
calibrated-agent-evaluation capture needs the OTel SDK and OTLP encoder the strands extra carries them, for any agent install the strands extra (OTLP-JSON ingest needs no extra)
This needs the [agentcore] extra an AgentCore command or module was used without its extra run the printed command
No AWS credentials were found for this command calibrated-agent-evaluation doctor --agentcore, calibrated-agent-evaluation dataset publish or pull found no usable profile or keys set AWS_PROFILE, or pass --profile (AgentCore)
Output
$ calibrated-agent-evaluation run    # without the stats extra
calibrated-agent-evaluation run: this stage needs an engine that isn't installed: uv sync --inexact --extra stats


$ calibrated-agent-evaluation label --serve --rater me    # without the ui extra
calibrated-agent-evaluation label: the review surface needs the UI extra; uv sync --inexact --extra ui


$ calibrated-agent-evaluation --backend langfuse ingest fixtures/raw/strands    # without the langfuse extra
calibrated-agent-evaluation ingest: the langfuse backend needs the optional extra: uv sync --inexact --extra langfuse


$ calibrated-agent-evaluation doctor --agentcore --region us-west-2    # without the agentcore extra
calibrated-agent-evaluation doctor: This needs the [agentcore] extra, which isn't installed.
Next: uv sync --inexact --extra agentcore
(Run with --explain for the details.)


calibrated-agent-evaluation doctor: No AWS credentials were found for this command.
Next: set AWS_PROFILE to a profile that can reach the account, or pass --profile.
(Run with --explain for the details.)

Getting traces in

You see Why Do
cannot load agent '...' the current directory isn't on the import path PYTHONPATH=. calibrated-agent-evaluation capture ..., or install your agent's package
captured 0 spans the agent emitted no OpenTelemetry spans enable instrumentation; for another process, calibrated-agent-evaluation ingest its OTLP-JSON (Capture)
the store already holds trace '...' a re-run capture keeps the stored trace new --trace-id, omit it for the unique OTel id, or --overwrite
[REFUSED] ... resolve to trace id ... two files in a batch share a trace id rename one so the name prefixes differ, or set calibrated_agent_evaluation.trace_id
[skipped] ... could not detect no attributes the ingest recognizes pass --convention, or add GenAI attributes (gallery)
degraded to ownership='probed' input and output but no agent or tool steps stored outcome-only (still labelable) unless --require-owned
Output
$ calibrated-agent-evaluation capture --agent orderbot:run_once
calibrated-agent-evaluation capture: cannot load agent 'orderbot:run_once': No module named 'orderbot'


$ calibrated-agent-evaluation capture --agent os:getcwd
calibrated-agent-evaluation capture: captured 0 spans from 'os:getcwd': the agent emitted no OpenTelemetry spans (is it instrumented? did it exit before a span closed?), so nothing was stored

The third command stops on the missing agent and tool steps:

$ calibrated-agent-evaluation capture --agent orderbot:run_once --service orderbot --dry-run
calibrated-agent-evaluation capture: 1 would ingest, 0 refused, 0 skipped
  [ok] <tmp>/calibrated-agent-evaluation-capture.otlp.json → <otel-trace-id> (gen_ai/latest, 1 span(s), 1 tool call(s), owned)
(exit 0)

$ calibrated-agent-evaluation capture --agent orderbot:run_once --service orderbot --trace-id ORD-A1001
calibrated-agent-evaluation capture: 1 ingested, 0 refused, 0 skipped
  [ok] <tmp>/ORD-A1001.otlp.json → ORD-A1001 (gen_ai/latest, 1 span(s), 1 tool call(s), owned)
(exit 0)

$ calibrated-agent-evaluation capture --agent orderbot:run_once --service orderbot --trace-id ORD-A1001
calibrated-agent-evaluation capture: the store already holds trace 'ORD-A1001' and a capture cannot be re-run to recover what would be replaced; pass a different --trace-id, or --overwrite to replace it
(exit 1)

$ calibrated-agent-evaluation capture --agent os:getcwd
calibrated-agent-evaluation capture: captured 0 spans from 'os:getcwd': the agent emitted no OpenTelemetry spans (is it instrumented? did it exit before a span closed?), so nothing was stored
(exit 1)



$ calibrated-agent-evaluation ingest --dry-run fixtures/raw/otel
trace ORD-DEGRADED degraded to ownership='probed': only a terminal input/output envelope was instrumented (no per-step tool spans and no per-step gen_ai message events)
trace degraded_v2 degraded to ownership='probed': no gen_ai.input/output.messages content and no tool calls recovered from the gen_ai/messages session
calibrated-agent-evaluation ingest: 7 would ingest (2 probed), 2 refused, 1 skipped
  [ok] fixtures/raw/otel/ORD-1001.attrs.otlp.json → 411234653c440f74f9c7bd885663e509 (gen_ai/latest, 1 span(s), 3 tool call(s), owned)
  [ok] fixtures/raw/otel/ORD-1001.build2.otlp.json → 180fed21d2542a2cee4e99fc1f050396 (gen_ai/latest, 1 span(s), 3 tool call(s), owned)
  [ok] fixtures/raw/otel/ORD-1001.events.otlp.json → ad66fcf93d899366e10b923c366a42cb (gen_ai/latest, 1 span(s), 3 tool call(s), owned)
  [ok] fixtures/raw/otel/ORD-1002.events.otlp.json → ORD-1002 (gen_ai/latest, 1 span(s), 3 tool call(s), owned)
  [ok] fixtures/raw/otel/ORD-DEGRADED.otlp.json → ORD-DEGRADED (gen_ai/latest, 0 span(s), 0 tool call(s), probed)
  [ok] fixtures/raw/otel/degraded_v2.otlp.json → degraded_v2 (gen_ai/messages, 0 span(s), 0 tool call(s), probed)
  [ok] fixtures/raw/otel/messages/swebench_tool_call.otlp.json → swebench_tool_call (gen_ai/messages, 1 span(s), 1 tool call(s), owned)
  [REFUSED] fixtures/raw/otel/openinference/weather_tool_call.otlp.json: 2 sources in this batch resolve to trace id 'f026f026f026f026f026f026f026f026' ('fixtures/raw/otel/openinference/weather_tool_call.otlp.json', 'fixtures/raw/otel/openllmetry/weather_tool_call.otlp.json') and the store is keyed on it, so persisting them would overwrite one with the other; rename them so their first dotted filename segments differ, or tag each with a distinct calibrated_agent_evaluation.trace_id span attribute
  [REFUSED] fixtures/raw/otel/openllmetry/weather_tool_call.otlp.json: 2 sources in this batch resolve to trace id 'f026f026f026f026f026f026f026f026' ('fixtures/raw/otel/openinference/weather_tool_call.otlp.json', 'fixtures/raw/otel/openllmetry/weather_tool_call.otlp.json') and the store is keyed on it, so persisting them would overwrite one with the other; rename them so their first dotted filename segments differ, or tag each with a distinct calibrated_agent_evaluation.trace_id span attribute
  [skipped] fixtures/raw/otel/unknown/unrecognized.otlp.json: could not detect an OTel GenAI convention: no gen_ai.input/output.messages, openinference.span.kind/llm.*_messages, gen_ai.prompt/completion.*, invoke_agent/execute_tool, or legacy gen_ai.* message-event signal found. Declare `semconv=` on the source (`calibrated-agent-evaluation ingest --convention <name>`) to force one, or register a custom convention.


$ calibrated-agent-evaluation ingest --require-owned fixtures/raw/otel/ORD-DEGRADED.otlp.json
trace ORD-DEGRADED degraded to ownership='probed': only a terminal input/output envelope was instrumented (no per-step tool spans and no per-step gen_ai message events)
calibrated-agent-evaluation ingest: 0 ingested, 1 refused, 0 skipped
  [REFUSED] fixtures/raw/otel/ORD-DEGRADED.otlp.json: trace 'ORD-DEGRADED' degraded to ownership='probed' and --require-owned was set; it carries terminal input/output only, with no per-step spans, so step-level attribution and per-agent rates have nothing to read; re-capture with tool/agent instrumentation, or drop --require-owned to accept outcome-only data

The trace store

You see Why Do
the langfuse backend needs LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY no credentials export both and LANGFUSE_HOST (per project)
Langfuse ... failed: ConnectError nothing answers at LANGFUSE_HOST just langfuse-up for the local stack; calibrated-agent-evaluation doctor checks reachability
StaleReadError async writes unreadable after CALIBRATED_AGENT_EVALUATION_READ_WAIT_SECONDS (30) rerun later, or raise the wait
no trace with trace_id '...' wrong store; the default is $CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, then ./.calibrated-agent-evaluation pass --root or run from the right directory
SchemaVersionError on calibrated-agent-evaluation load the dump was written by another schema version, and records are not converted between versions load with the version that wrote the dump (Versioning)
holds dump files but no dump-manifest.json on load (IncompleteDumpError) the dump was interrupted or is still being written, since the manifest is written last dump the source store again
differs from DIR/dump-manifest.json or is missing on load (DumpMismatchError) a file was edited after the dump, or another dump overwrote part of this one dump the source store again; to commit a dump, keep git from rewriting its line endings (Configure the CI job)
file(s) its dump-manifest.json does not list on load (UnlistedDumpFileError) a .csv or .json file in the directory was not written by that dump remove the files it names, or dump into an empty directory
Output
$ calibrated-agent-evaluation ingest --from-langfuse "My Project"
calibrated-agent-evaluation ingest: the langfuse backend needs LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY in the environment (see https://awslabs.github.io/calibrated-agent-evaluation/integrate/langfuse/)


$ LANGFUSE_HOST=http://127.0.0.1:9 calibrated-agent-evaluation --backend langfuse ingest fixtures/raw/strands/CLM-4471.otlp.json
calibrated-agent-evaluation ingest: Langfuse get_trace('CLM-4471') failed: ConnectError: [Errno N] Connection refused
(exit 1)


$ calibrated-agent-evaluation run --trace NOPE
calibrated-agent-evaluation run: no trace with trace_id 'NOPE' in local store

The dump here was edited to look older:

$ calibrated-agent-evaluation --root restored load dump
calibrated-agent-evaluation load: dump schema_version '0' != running SCHEMA_VERSION '1'; refusing to load (no coercion)
(exit 1)

Calibration and judging

You see Why Do
no calibration, no score a judge scores only once it is certified and has passed its bias check in this store (library: UncalibratedScoreError) calibrate, bias, then judge (Put a judge on probation)
not enough labels to trust this judge too few labels to tell label the per-class count it names
labels already used to test a judge for this mode a set tests a mode once, whatever the result or judge file name; overlapping sets count too label a fresh set (Watch production)
bias check failed the judge was swayed by a change that should not matter more often than its threshold allows revise the prompt, then check it again on fresh labels
no recorded verdict (cassette miss) the prompt was never recorded, and replay does not call the live model record these traces with --cassette-mode record (it calls the live model), or narrow with --cell
model failed after 3 attempts live Bedrock call without a valid AWS session fix credentials, or pass --cassette
Output
$ calibrated-agent-evaluation judge --judge fixtures/judge-files/orderbot-refund-resolved.json
calibrated-agent-evaluation judge: no calibration record for judge 'orderbot-refund-resolved' under $CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT/records/calibration: no calibration, no score. Run `calibrated-agent-evaluation calibrate --judge fixtures/judge-files/orderbot-refund-resolved.json` first.


$ calibrated-agent-evaluation calibrate --judge fixtures/judge-files/orderbot-refund-resolved.json --cassette fixtures/raw/cassettes/judge_orderbot_resolved.cassette.json --cell feature=refund_handling
calibrated-agent-evaluation calibrate: Not enough labels yet to trust this judge (checked on 2 gold label(s)).
Next: label 54 more FAIL and 54 more PASS traces, then run `calibrated-agent-evaluation calibrate` again.
Rater agreement on the gold was not measured (the gold has one rater); have a second rater label the same gold (at least 130 of its traces) so the gold itself is checked.
Run with --explain for the numbers behind this.


$ calibrated-agent-evaluation --root store calibrate --judge store/quickstart/judge-v1.json --cassette store/quickstart/judge.cassette.json --rate-safe --cell label_round=1
calibrated-agent-evaluation calibrate: These labels were already used to test a judge for this failure mode (it certified that judge), and a set of labels tests a judge only once.
Next: label fresh traces for this judge: re-testing on labels a judge was already checked on hides overfitting.
(Run with --explain for the details.)
(exit 1)



$ calibrated-agent-evaluation --root store bias --judge store/quickstart/judge-v1.json --probes store/quickstart/bias-probes-1 --thresholds store/quickstart/bias-thresholds.json --cassette store/quickstart/judge.cassette.json --explain
calibrated-agent-evaluation bias: Bias check failed: the judge changes its verdict when answers get longer and goes along with claims asserted in the conversation.
Next: revise the judge prompt to address this, then run `calibrated-agent-evaluation calibrate` and `calibrated-agent-evaluation bias` on the new version (with fresh labels).

bias battery for judge 'orderbot-refund-policy' (bedrock/global.anthropic.claude-haiku-4-5-20251001-v1:0@v1): BIASED, must-pass bias exceeded: length, sycophancy
  each bias is how much more often the judge's verdicts disagree across the biasing dimension than across a placebo that varies nothing, gated on its one-sided 95% upper bound against the judge model's threshold; a bias with a direction is also gated on how much more often its verdicts move toward the favored one than the placebo's do
  compassion_fade (advisory): 1 of 12 trial(s) disagreed, 2 of 12 placebo(s) did; excess 0.000 (95% upper bound 0.133), within the 0.35 threshold
  length (must-pass): 3 of 12 trial(s) disagreed, 0 of 12 placebo(s) did; excess 0.250 (95% upper bound 0.487); toward the favored verdict 1 of 12 trial(s) moved, 0 placebo(s) did, excess 0.083 (95% upper bound 0.301, OVER 0.20), OVER the 0.35 threshold
  self_preference (must-pass): 0 of 12 trial(s) disagreed, 0 of 12 placebo(s) did; excess 0.000 (95% upper bound 0.184), within the 0.35 threshold
  sycophancy (must-pass): 2 of 12 trial(s) disagreed, 0 of 12 placebo(s) did; excess 0.167 (95% upper bound 0.399); toward the favored verdict 2 of 12 trial(s) moved, 0 placebo(s) did, excess 0.167 (95% upper bound 0.399, OVER 0.20), OVER the 0.35 threshold
recorded <id> in store/records/bias
(exit 1)



$ calibrated-agent-evaluation calibrate --judge fixtures/judge-files/orderbot-refund-resolved.json --cassette fixtures/raw/cassettes/judge_orderbot_resolved.cassette.json
calibrated-agent-evaluation calibrate: judge 'orderbot-refund-resolved' on trace 'CLM-4473': no recorded verdict (cassette miss); judge cassette judge_orderbot_resolved.cassette.json: no recorded verdict for fingerprint 509742225a4f7106 (REPLAY mode does not fall through to a live call)


$ calibrated-agent-evaluation calibrate --judge fixtures/judge-files/orderbot-refund-resolved.json --cell feature=refund_handling
calibrated-agent-evaluation calibrate: judge 'orderbot-refund-resolved' on trace 'ORD-1001': model failed after 3 attempts; judge model 'bedrock/global.anthropic.claude-sonnet-4-6' failed to generate a structured verdict: The config profile (calibrated-agent-evaluation-docs-no-such-profile) could not be found

Rates and gating

You see Why Do
rate not estimated (too few scored traces) under 150 randomly sampled traces, the range is wrong too often calibrated-agent-evaluation judge --frame production-representative on more
rate not estimated (judge needs more labels) certified, but on too few labels for a rate label what --explain names, or use --rate-safe
no judge verdicts with a failure mode ... to gate there are no verdicts of the source you asked for in scope calibrated-agent-evaluation judge, or --verdicts human / scorer
usage error, exit 2 bad flag or argument calibrated-agent-evaluation <command> --help, CLI reference

In both rate-not-estimated cases the judge still scores and the gate still sets the exit code.

Output
$ calibrated-agent-evaluation --root store judge --judge store/quickstart/judge-v2.json --trace Q-30000 --cassette store/quickstart/judge.cassette.json
Wrote 1 verdict(s) for trusted_broken_policy_tool on 1 trace(s): 1 FAIL, 0 PASS.
Next: turn these into a failure rate with `calibrated-agent-evaluation --root store run --judge store/quickstart/judge-v2.json`.
Run with --explain for the numbers behind this.
(exit 0)

$ calibrated-agent-evaluation --root store run --judge store/quickstart/judge-v2.json --ci --explain
calibrated-agent-evaluation run: CI gate failed: 1 of 1 failure mode(s) don't clear the bar yet: trusted_broken_policy_tool.
  (The bar: at least 80% of each mode's traces must pass, even at the cautious end of what this many traces can show.)
trusted_broken_policy_tool failure rate: not estimated (too few scored traces; next, score 149 more production trace(s) with `calibrated-agent-evaluation judge`).
Next: fix the failing modes, or label more traces where a mode has too few to show its pass rate.

CI gate over 1 judge verdicts on 848 stored trace(s), threshold 0.80 on each mode's pass-rate 95% lower bound: FAILED, 1 of 1 mode(s) did not: trusted_broken_policy_tool
  trusted_broken_policy_tool: 1 FAIL of 1, pass-rate 95% lower bound 0.000 vs threshold 0.80: FAILED
trend (1 mode(s), Rogan-Gladen corrected fail rate):
  trusted_broken_policy_tool: corrected REFUSED (rate sample too small: the rate sample has 1 trace(s); a corrected rate needs at least 150, so score 149 more production-representative trace(s) and rerun. Below that the interval was measured to cover the true rate less often than it claims.), 95% CI [0.000, 1.000] (rogan_gladen_jeffreys_posterior_percentile, n=1, calibration calib-orderbot-refund-policy-gold-60968d-807cc1ffd3ff); raw 1.000, 95% CI [0.207, 1.000] (wilson)
recorded run run-<id> in store/records/run
  Known limit: below 150 scored traces, or when a class shows about 2 to 3 judge errors, the rate's range was measured to contain the true rate less often than 95% of the time.
(exit 1)



$ calibrated-agent-evaluation run --ci
calibrated-agent-evaluation run: no judge verdicts with a failure mode on 6 stored trace(s) to gate (--verdicts judge); score the traces with `calibrated-agent-evaluation judge` first, or gate another source with --verdicts scorer or --verdicts human


$ calibrated-agent-evaluation run --verdicts bogus
usage: calibrated-agent-evaluation run [-h] [--trace TRACE_ID] [--ci] [--threshold THRESHOLD]
                                       [--verdicts {judge,scorer,human}] [--judge FILE]
                                       [--profile PROFILE] [--region REGION] [--seed SEED]
                                       [--n-resamples N_RESAMPLES] [--publish {local,langfuse}]
                                       [--drift-baseline STORE]
                                       [--drift-feature {input-chars,turns,tool-calls}]
                                       [--fail-on-drift] [--atoms FILE]
                                       [--backend {local,langfuse}] [--root PATH] [--explain]
calibrated-agent-evaluation run: error: argument --verdicts: invalid choice: 'bogus' (choose from judge, scorer, human)