Troubleshooting¶
When a command can't give you an answer, it exits 1 and says what to collect or change. A usage
mistake exits 2. A Python traceback is a bug; please
report it. Add --explain (or set
CALIBRATED_AGENT_EVALUATION_EXPLAIN=1) to see the numbers and record paths behind a result.
In the library, errors derive from calibrated_agent_evaluation.schema.CalibratedAgentEvaluationDataError;
the CLI's derive from calibrated_agent_evaluation.cli.errors.CliError.
AgentCore messages are on Bedrock AgentCore.
Messages you may see and what to do¶
| Message | What triggered it | What to do |
|---|---|---|
| Judge not certified | On your labels, the judge's cautious catch rate for failures, or for passes, is below 0.80 | Label more traces of the class that is short (the message says how many), or improve the judge |
| Few-shot pool used as a test set | The traces you checked the judge on are the examples in its prompt | Check it on a separate set |
| Labels already used to test a judge | This set already decided one judge for this mode | Label a fresh set of traces the judge hasn't been checked on |
| Rate measured on the calibration set | The rate sample is the set the judge was checked on | Score a random sample of production traffic |
| Rate sample too small | Fewer than 150 traces in the rate sample | Score more production traces (the message says how many) |
| Not enough labels for a corrected rate | The judge is certified, but on too few labels for its accuracy to give a reliable range | Label more, or plan with --rate-safe next time |
| Judge too weak to correct | The judge's catch rates for failures and passes add up to less than 1.5 | Use the range the report still prints, or improve the judge |
| Flag rate outside what the judge can produce | The judge flagged fewer traces than its own false-alarm rate, or more than it catches | Check the rate sample; the range is still reported |
| Judge biased | The judge changes its verdict when something irrelevant about the input changes, such as a longer answer | Fix the judge prompt, then check it again on fresh labels |
Methods gives the rule behind each threshold.
Installation and extras¶
| You see | Why | Do |
|---|---|---|
| an extra is not installed | the command needs a third-party package | run the command it prints; calibrated-agent-evaluation doctor lists them all (Install) |
calibrated-agent-evaluation capture needs the OTel SDK and OTLP encoder |
the strands extra carries them, for any agent |
install the strands extra (OTLP-JSON ingest needs no extra) |
This needs the [agentcore] extra |
an AgentCore command or module was used without its extra | run the printed command |
No AWS credentials were found for this command |
calibrated-agent-evaluation doctor --agentcore, calibrated-agent-evaluation dataset publish or pull found no usable profile or keys |
set AWS_PROFILE, or pass --profile (AgentCore) |
Output
$ calibrated-agent-evaluation run # without the stats extra
calibrated-agent-evaluation run: this stage needs an engine that isn't installed: uv sync --inexact --extra stats
$ calibrated-agent-evaluation label --serve --rater me # without the ui extra
calibrated-agent-evaluation label: the review surface needs the UI extra; uv sync --inexact --extra ui
$ calibrated-agent-evaluation --backend langfuse ingest fixtures/raw/strands # without the langfuse extra
calibrated-agent-evaluation ingest: the langfuse backend needs the optional extra: uv sync --inexact --extra langfuse
$ calibrated-agent-evaluation doctor --agentcore --region us-west-2 # without the agentcore extra
calibrated-agent-evaluation doctor: This needs the [agentcore] extra, which isn't installed.
Next: uv sync --inexact --extra agentcore
(Run with --explain for the details.)
calibrated-agent-evaluation doctor: No AWS credentials were found for this command.
Next: set AWS_PROFILE to a profile that can reach the account, or pass --profile.
(Run with --explain for the details.)
Getting traces in¶
| You see | Why | Do |
|---|---|---|
cannot load agent '...' |
the current directory isn't on the import path | PYTHONPATH=. calibrated-agent-evaluation capture ..., or install your agent's package |
captured 0 spans |
the agent emitted no OpenTelemetry spans | enable instrumentation; for another process, calibrated-agent-evaluation ingest its OTLP-JSON (Capture) |
the store already holds trace '...' |
a re-run capture keeps the stored trace | new --trace-id, omit it for the unique OTel id, or --overwrite |
[REFUSED] ... resolve to trace id ... |
two files in a batch share a trace id | rename one so the name prefixes differ, or set calibrated_agent_evaluation.trace_id |
[skipped] ... could not detect |
no attributes the ingest recognizes | pass --convention, or add GenAI attributes (gallery) |
degraded to ownership='probed' |
input and output but no agent or tool steps | stored outcome-only (still labelable) unless --require-owned |
Output
$ calibrated-agent-evaluation capture --agent orderbot:run_once
calibrated-agent-evaluation capture: cannot load agent 'orderbot:run_once': No module named 'orderbot'
$ calibrated-agent-evaluation capture --agent os:getcwd
calibrated-agent-evaluation capture: captured 0 spans from 'os:getcwd': the agent emitted no OpenTelemetry spans (is it instrumented? did it exit before a span closed?), so nothing was stored
The third command stops on the missing agent and tool steps:
$ calibrated-agent-evaluation capture --agent orderbot:run_once --service orderbot --dry-run
calibrated-agent-evaluation capture: 1 would ingest, 0 refused, 0 skipped
[ok] <tmp>/calibrated-agent-evaluation-capture.otlp.json → <otel-trace-id> (gen_ai/latest, 1 span(s), 1 tool call(s), owned)
(exit 0)
$ calibrated-agent-evaluation capture --agent orderbot:run_once --service orderbot --trace-id ORD-A1001
calibrated-agent-evaluation capture: 1 ingested, 0 refused, 0 skipped
[ok] <tmp>/ORD-A1001.otlp.json → ORD-A1001 (gen_ai/latest, 1 span(s), 1 tool call(s), owned)
(exit 0)
$ calibrated-agent-evaluation capture --agent orderbot:run_once --service orderbot --trace-id ORD-A1001
calibrated-agent-evaluation capture: the store already holds trace 'ORD-A1001' and a capture cannot be re-run to recover what would be replaced; pass a different --trace-id, or --overwrite to replace it
(exit 1)
$ calibrated-agent-evaluation capture --agent os:getcwd
calibrated-agent-evaluation capture: captured 0 spans from 'os:getcwd': the agent emitted no OpenTelemetry spans (is it instrumented? did it exit before a span closed?), so nothing was stored
(exit 1)
$ calibrated-agent-evaluation ingest --dry-run fixtures/raw/otel
trace ORD-DEGRADED degraded to ownership='probed': only a terminal input/output envelope was instrumented (no per-step tool spans and no per-step gen_ai message events)
trace degraded_v2 degraded to ownership='probed': no gen_ai.input/output.messages content and no tool calls recovered from the gen_ai/messages session
calibrated-agent-evaluation ingest: 7 would ingest (2 probed), 2 refused, 1 skipped
[ok] fixtures/raw/otel/ORD-1001.attrs.otlp.json → 411234653c440f74f9c7bd885663e509 (gen_ai/latest, 1 span(s), 3 tool call(s), owned)
[ok] fixtures/raw/otel/ORD-1001.build2.otlp.json → 180fed21d2542a2cee4e99fc1f050396 (gen_ai/latest, 1 span(s), 3 tool call(s), owned)
[ok] fixtures/raw/otel/ORD-1001.events.otlp.json → ad66fcf93d899366e10b923c366a42cb (gen_ai/latest, 1 span(s), 3 tool call(s), owned)
[ok] fixtures/raw/otel/ORD-1002.events.otlp.json → ORD-1002 (gen_ai/latest, 1 span(s), 3 tool call(s), owned)
[ok] fixtures/raw/otel/ORD-DEGRADED.otlp.json → ORD-DEGRADED (gen_ai/latest, 0 span(s), 0 tool call(s), probed)
[ok] fixtures/raw/otel/degraded_v2.otlp.json → degraded_v2 (gen_ai/messages, 0 span(s), 0 tool call(s), probed)
[ok] fixtures/raw/otel/messages/swebench_tool_call.otlp.json → swebench_tool_call (gen_ai/messages, 1 span(s), 1 tool call(s), owned)
[REFUSED] fixtures/raw/otel/openinference/weather_tool_call.otlp.json: 2 sources in this batch resolve to trace id 'f026f026f026f026f026f026f026f026' ('fixtures/raw/otel/openinference/weather_tool_call.otlp.json', 'fixtures/raw/otel/openllmetry/weather_tool_call.otlp.json') and the store is keyed on it, so persisting them would overwrite one with the other; rename them so their first dotted filename segments differ, or tag each with a distinct calibrated_agent_evaluation.trace_id span attribute
[REFUSED] fixtures/raw/otel/openllmetry/weather_tool_call.otlp.json: 2 sources in this batch resolve to trace id 'f026f026f026f026f026f026f026f026' ('fixtures/raw/otel/openinference/weather_tool_call.otlp.json', 'fixtures/raw/otel/openllmetry/weather_tool_call.otlp.json') and the store is keyed on it, so persisting them would overwrite one with the other; rename them so their first dotted filename segments differ, or tag each with a distinct calibrated_agent_evaluation.trace_id span attribute
[skipped] fixtures/raw/otel/unknown/unrecognized.otlp.json: could not detect an OTel GenAI convention: no gen_ai.input/output.messages, openinference.span.kind/llm.*_messages, gen_ai.prompt/completion.*, invoke_agent/execute_tool, or legacy gen_ai.* message-event signal found. Declare `semconv=` on the source (`calibrated-agent-evaluation ingest --convention <name>`) to force one, or register a custom convention.
$ calibrated-agent-evaluation ingest --require-owned fixtures/raw/otel/ORD-DEGRADED.otlp.json
trace ORD-DEGRADED degraded to ownership='probed': only a terminal input/output envelope was instrumented (no per-step tool spans and no per-step gen_ai message events)
calibrated-agent-evaluation ingest: 0 ingested, 1 refused, 0 skipped
[REFUSED] fixtures/raw/otel/ORD-DEGRADED.otlp.json: trace 'ORD-DEGRADED' degraded to ownership='probed' and --require-owned was set; it carries terminal input/output only, with no per-step spans, so step-level attribution and per-agent rates have nothing to read; re-capture with tool/agent instrumentation, or drop --require-owned to accept outcome-only data
The trace store¶
| You see | Why | Do |
|---|---|---|
the langfuse backend needs LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY |
no credentials | export both and LANGFUSE_HOST (per project) |
Langfuse ... failed: ConnectError |
nothing answers at LANGFUSE_HOST |
just langfuse-up for the local stack; calibrated-agent-evaluation doctor checks reachability |
StaleReadError |
async writes unreadable after CALIBRATED_AGENT_EVALUATION_READ_WAIT_SECONDS (30) |
rerun later, or raise the wait |
no trace with trace_id '...' |
wrong store; the default is $CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, then ./.calibrated-agent-evaluation |
pass --root or run from the right directory |
SchemaVersionError on calibrated-agent-evaluation load |
the dump was written by another schema version, and records are not converted between versions | load with the version that wrote the dump (Versioning) |
holds dump files but no dump-manifest.json on load (IncompleteDumpError) |
the dump was interrupted or is still being written, since the manifest is written last | dump the source store again |
differs from DIR/dump-manifest.json or is missing on load (DumpMismatchError) |
a file was edited after the dump, or another dump overwrote part of this one | dump the source store again; to commit a dump, keep git from rewriting its line endings (Configure the CI job) |
file(s) its dump-manifest.json does not list on load (UnlistedDumpFileError) |
a .csv or .json file in the directory was not written by that dump |
remove the files it names, or dump into an empty directory |
Output
$ calibrated-agent-evaluation ingest --from-langfuse "My Project"
calibrated-agent-evaluation ingest: the langfuse backend needs LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY in the environment (see https://awslabs.github.io/calibrated-agent-evaluation/integrate/langfuse/)
$ LANGFUSE_HOST=http://127.0.0.1:9 calibrated-agent-evaluation --backend langfuse ingest fixtures/raw/strands/CLM-4471.otlp.json
calibrated-agent-evaluation ingest: Langfuse get_trace('CLM-4471') failed: ConnectError: [Errno N] Connection refused
(exit 1)
$ calibrated-agent-evaluation run --trace NOPE
calibrated-agent-evaluation run: no trace with trace_id 'NOPE' in local store
The dump here was edited to look older:
Calibration and judging¶
| You see | Why | Do |
|---|---|---|
no calibration, no score |
a judge scores only once it is certified and has passed its bias check in this store (library: UncalibratedScoreError) |
calibrate, bias, then judge (Put a judge on probation) |
| not enough labels to trust this judge | too few labels to tell | label the per-class count it names |
| labels already used to test a judge for this mode | a set tests a mode once, whatever the result or judge file name; overlapping sets count too | label a fresh set (Watch production) |
| bias check failed | the judge was swayed by a change that should not matter more often than its threshold allows | revise the prompt, then check it again on fresh labels |
no recorded verdict (cassette miss) |
the prompt was never recorded, and replay does not call the live model | record these traces with --cassette-mode record (it calls the live model), or narrow with --cell |
model failed after 3 attempts |
live Bedrock call without a valid AWS session | fix credentials, or pass --cassette |
Output
$ calibrated-agent-evaluation judge --judge fixtures/judge-files/orderbot-refund-resolved.json
calibrated-agent-evaluation judge: no calibration record for judge 'orderbot-refund-resolved' under $CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT/records/calibration: no calibration, no score. Run `calibrated-agent-evaluation calibrate --judge fixtures/judge-files/orderbot-refund-resolved.json` first.
$ calibrated-agent-evaluation calibrate --judge fixtures/judge-files/orderbot-refund-resolved.json --cassette fixtures/raw/cassettes/judge_orderbot_resolved.cassette.json --cell feature=refund_handling
calibrated-agent-evaluation calibrate: Not enough labels yet to trust this judge (checked on 2 gold label(s)).
Next: label 54 more FAIL and 54 more PASS traces, then run `calibrated-agent-evaluation calibrate` again.
Rater agreement on the gold was not measured (the gold has one rater); have a second rater label the same gold (at least 130 of its traces) so the gold itself is checked.
Run with --explain for the numbers behind this.
$ calibrated-agent-evaluation --root store calibrate --judge store/quickstart/judge-v1.json --cassette store/quickstart/judge.cassette.json --rate-safe --cell label_round=1
calibrated-agent-evaluation calibrate: These labels were already used to test a judge for this failure mode (it certified that judge), and a set of labels tests a judge only once.
Next: label fresh traces for this judge: re-testing on labels a judge was already checked on hides overfitting.
(Run with --explain for the details.)
(exit 1)
$ calibrated-agent-evaluation --root store bias --judge store/quickstart/judge-v1.json --probes store/quickstart/bias-probes-1 --thresholds store/quickstart/bias-thresholds.json --cassette store/quickstart/judge.cassette.json --explain
calibrated-agent-evaluation bias: Bias check failed: the judge changes its verdict when answers get longer and goes along with claims asserted in the conversation.
Next: revise the judge prompt to address this, then run `calibrated-agent-evaluation calibrate` and `calibrated-agent-evaluation bias` on the new version (with fresh labels).
bias battery for judge 'orderbot-refund-policy' (bedrock/global.anthropic.claude-haiku-4-5-20251001-v1:0@v1): BIASED, must-pass bias exceeded: length, sycophancy
each bias is how much more often the judge's verdicts disagree across the biasing dimension than across a placebo that varies nothing, gated on its one-sided 95% upper bound against the judge model's threshold; a bias with a direction is also gated on how much more often its verdicts move toward the favored one than the placebo's do
compassion_fade (advisory): 1 of 12 trial(s) disagreed, 2 of 12 placebo(s) did; excess 0.000 (95% upper bound 0.133), within the 0.35 threshold
length (must-pass): 3 of 12 trial(s) disagreed, 0 of 12 placebo(s) did; excess 0.250 (95% upper bound 0.487); toward the favored verdict 1 of 12 trial(s) moved, 0 placebo(s) did, excess 0.083 (95% upper bound 0.301, OVER 0.20), OVER the 0.35 threshold
self_preference (must-pass): 0 of 12 trial(s) disagreed, 0 of 12 placebo(s) did; excess 0.000 (95% upper bound 0.184), within the 0.35 threshold
sycophancy (must-pass): 2 of 12 trial(s) disagreed, 0 of 12 placebo(s) did; excess 0.167 (95% upper bound 0.399); toward the favored verdict 2 of 12 trial(s) moved, 0 placebo(s) did, excess 0.167 (95% upper bound 0.399, OVER 0.20), OVER the 0.35 threshold
recorded <id> in store/records/bias
(exit 1)
$ calibrated-agent-evaluation calibrate --judge fixtures/judge-files/orderbot-refund-resolved.json --cassette fixtures/raw/cassettes/judge_orderbot_resolved.cassette.json
calibrated-agent-evaluation calibrate: judge 'orderbot-refund-resolved' on trace 'CLM-4473': no recorded verdict (cassette miss); judge cassette judge_orderbot_resolved.cassette.json: no recorded verdict for fingerprint 509742225a4f7106 (REPLAY mode does not fall through to a live call)
$ calibrated-agent-evaluation calibrate --judge fixtures/judge-files/orderbot-refund-resolved.json --cell feature=refund_handling
calibrated-agent-evaluation calibrate: judge 'orderbot-refund-resolved' on trace 'ORD-1001': model failed after 3 attempts; judge model 'bedrock/global.anthropic.claude-sonnet-4-6' failed to generate a structured verdict: The config profile (calibrated-agent-evaluation-docs-no-such-profile) could not be found
Rates and gating¶
| You see | Why | Do |
|---|---|---|
| rate not estimated (too few scored traces) | under 150 randomly sampled traces, the range is wrong too often | calibrated-agent-evaluation judge --frame production-representative on more |
| rate not estimated (judge needs more labels) | certified, but on too few labels for a rate | label what --explain names, or use --rate-safe |
no judge verdicts with a failure mode ... to gate |
there are no verdicts of the source you asked for in scope | calibrated-agent-evaluation judge, or --verdicts human / scorer |
| usage error, exit 2 | bad flag or argument | calibrated-agent-evaluation <command> --help, CLI reference |
In both rate-not-estimated cases the judge still scores and the gate still sets the exit code.
Output
$ calibrated-agent-evaluation --root store judge --judge store/quickstart/judge-v2.json --trace Q-30000 --cassette store/quickstart/judge.cassette.json
Wrote 1 verdict(s) for trusted_broken_policy_tool on 1 trace(s): 1 FAIL, 0 PASS.
Next: turn these into a failure rate with `calibrated-agent-evaluation --root store run --judge store/quickstart/judge-v2.json`.
Run with --explain for the numbers behind this.
(exit 0)
$ calibrated-agent-evaluation --root store run --judge store/quickstart/judge-v2.json --ci --explain
calibrated-agent-evaluation run: CI gate failed: 1 of 1 failure mode(s) don't clear the bar yet: trusted_broken_policy_tool.
(The bar: at least 80% of each mode's traces must pass, even at the cautious end of what this many traces can show.)
trusted_broken_policy_tool failure rate: not estimated (too few scored traces; next, score 149 more production trace(s) with `calibrated-agent-evaluation judge`).
Next: fix the failing modes, or label more traces where a mode has too few to show its pass rate.
CI gate over 1 judge verdicts on 848 stored trace(s), threshold 0.80 on each mode's pass-rate 95% lower bound: FAILED, 1 of 1 mode(s) did not: trusted_broken_policy_tool
trusted_broken_policy_tool: 1 FAIL of 1, pass-rate 95% lower bound 0.000 vs threshold 0.80: FAILED
trend (1 mode(s), Rogan-Gladen corrected fail rate):
trusted_broken_policy_tool: corrected REFUSED (rate sample too small: the rate sample has 1 trace(s); a corrected rate needs at least 150, so score 149 more production-representative trace(s) and rerun. Below that the interval was measured to cover the true rate less often than it claims.), 95% CI [0.000, 1.000] (rogan_gladen_jeffreys_posterior_percentile, n=1, calibration calib-orderbot-refund-policy-gold-60968d-807cc1ffd3ff); raw 1.000, 95% CI [0.207, 1.000] (wilson)
recorded run run-<id> in store/records/run
Known limit: below 150 scored traces, or when a class shows about 2 to 3 judge errors, the rate's range was measured to contain the true rate less often than 95% of the time.
(exit 1)
$ calibrated-agent-evaluation run --ci
calibrated-agent-evaluation run: no judge verdicts with a failure mode on 6 stored trace(s) to gate (--verdicts judge); score the traces with `calibrated-agent-evaluation judge` first, or gate another source with --verdicts scorer or --verdicts human
$ calibrated-agent-evaluation run --verdicts bogus
usage: calibrated-agent-evaluation run [-h] [--trace TRACE_ID] [--ci] [--threshold THRESHOLD]
[--verdicts {judge,scorer,human}] [--judge FILE]
[--profile PROFILE] [--region REGION] [--seed SEED]
[--n-resamples N_RESAMPLES] [--publish {local,langfuse}]
[--drift-baseline STORE]
[--drift-feature {input-chars,turns,tool-calls}]
[--fail-on-drift] [--atoms FILE]
[--backend {local,langfuse}] [--root PATH] [--explain]
calibrated-agent-evaluation run: error: argument --verdicts: invalid choice: 'bogus' (choose from judge, scorer, human)