Skip to content

Calibrated Agent Evaluation

Check your LLM judge before you trust its numbers.

Most teams grade their AI agent with another model, an LLM judge, and ship on what it says. This toolkit has people on your team label a sample of runs, measures the judge against those labels, and lets the judge score the runs that gate your CI build only after it passes. Then it tells you how often your agent really fails.

A judge is a new hire on review duty

You would not let a new hire's code reviews gate production on their first day. You would read a few dozen of their calls next to your own first, see where they go easy and where they nitpick, and sign them off once they agree with you often enough. An LLM judge needs the same treatment, and most teams skip it.

A judge's calls compared with a person's on a sample, then cleared to review alone A judge's calls compared with a person's on a sample, then cleared to review alone

This toolkit runs that probation. Your team labels a sample of the agent's runs, the judge grades the same runs, and the two are compared. A judge that misses too many real failures, or flags too many good runs, cannot score anything, so its verdicts never reach the build's gate. Changing the judge's model or prompt starts its probation over.

What a judge's number hides

1,000 sample traces, marked by what the judge said and what was true 1,000 sample traces, marked by what the judge said and what was true

The judge flagged 14.8% of 1,000 traces

Calibrated Agent Evaluation reports 6.0% likely between 0.0% and 12.6%

The true rate is 6.0% known here because the sample is synthetic

The judge is right on 90% of failures and 90% of passes. On a sample where few traces fail, the 10% of passes it gets wrong outnumber the real failures, so its raw rate is more than double the truth. The toolkit measures the judge's mistakes against your labels and takes them back out.

The program behind this picture
Judge certified: it caught 90.0% of the failures and cleared 90.0% of the passes.
It flagged 148 of 1,000 production traces: 14.8%.
Failure rate: 6.0% (likely between 0.0% and 12.6%).
The true rate in this synthetic sample: 6.0%.

On another sample it flagged 50 of 1,000: 5.0%.
That is fewer than the judge's false alarms alone would produce,
so there is no honest single number.
Failure rate: likely between 0.0% and 0.5%.

The sample is synthetic, so the true rate is known. The picture and this output are rebuilt from docs/site/examples/rate_correction.py on every docs build.

Four ways a judge's number misleads a team

  • It misses a failure

    The judge passes runs where the agent issues a refund the policy doesn't allow. The dashboard is green, and the money is gone. Step 4

  • It flags failures that aren't there

    The judge wrongly flags a few good replies in every hundred. When real failures are rare, those outnumber them, and the team spends a sprint fixing replies that were fine. Step 4

  • A generic score hides the real problem

    "Helpfulness" goes up release after release while the refund-policy mistakes stay where they were, because nobody named them. Step 3

  • A fix only works on the test set

    The failure rate drops on the traces the team tuned against. In production, the same failures show up under a different name. Step 6

How it works

1 Read your traces

Bring the recorded runs of your agent from OpenTelemetry, Langfuse, Inspect, Strands or a file, and read them before deciding what to measure.

2 Label and name failures

People on your team mark runs pass or fail in a review UI, and the toolkit groups the failures into named failure modes.

3 Put a judge on probation

The judge grades the labeled runs. It may score new runs only after it agrees with your team often enough and passes a bias check: padding a reply, writing in the judge's own style, or a customer insisting they are right must not change its verdict.

4 Gate and report

A certified judge scores new runs. The CI gate passes or fails each failure mode, and the report gives the failure rate with the judge's mistakes taken out.

The walkthrough follows one support team through every step.

Try it

No API keys and no network. The sample corpus ships with the package:

pip install calibrated-agent-evaluation
calibrated-agent-evaluation quickstart
What it prints
$ calibrated-agent-evaluation quickstart
calibrated-agent-evaluation quickstart: a first error-analysis pass on the bundled sample corpus

Seeded 6 agent traces + 9 gold labels into an offline local store
  ($CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT)

First-pass result: 5 of 6 traces have a labeled failure.
Failure modes found:
    - trusted_broken_policy_tool: 2
    - dropped_material_fact: 1
    - misread_amount: 1
    - vacancy_threshold_margin_mishandled: 1

Those counts come from the gold labels that ship with the corpus: nothing has been
judged yet. Next steps:
  - calibrated-agent-evaluation doctor
    see which optional extras are installed
  - uv sync --inexact --extra ui, then
    calibrated-agent-evaluation --root $CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT label --serve --rater YOUR_ID
    review traces in the UI
  - uv sync --inexact --extra stats
    judge calibration and corrected failure rates
  - calibrated-agent-evaluation --root $CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT cluster
    check failure-mode saturation and coverage
  - calibrated-agent-evaluation --root $CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT run --verdicts human --ci
    the CI gate over the gold labels
  - calibrated-agent-evaluation calibrate, calibrated-agent-evaluation bias, calibrated-agent-evaluation judge
    certify a judge, then score with it
  - calibrated-agent-evaluation quickstart --certify
    a set big enough to certify one, offline

Install lists the optional extras.

Works with what you have

It reads traces from OpenTelemetry GenAI spans, Langfuse, OpenInference, OpenLLMetry, Inspect AI, Strands Agents, Bedrock AgentCore and plain datasets. It can also sample an agent you only reach over HTTP. Connect your agent.


Package and CLI: calibrated-agent-evaluation · import name: calibrated_agent_evaluation · License: Apache-2.0 · Source: awslabs/calibrated-agent-evaluation