Use the Claude Code skills¶
The repository is a Claude Code plugin with five skills, so a coding agent can run the loop with you: review your current eval setup, help you find failure modes, draft a judge and take it through the checks. Every step of the walkthrough shows the skill next to the CLI command in a "With Claude Code" tab.
Each skill works by running calibrated-agent-evaluation commands. A skill does not write labels
for you and does not compute any number itself; it reports what the command printed.
Install¶
Add the repository as a plugin marketplace from inside Claude Code, then install the plugin:
/plugin marketplace add awslabs/calibrated-agent-evaluation
/plugin install calibrated-agent-evaluation@calibrated-agent-evaluation
From a local checkout, pass the checkout path to /plugin marketplace add instead, or load the
plugin for one session with claude --plugin-dir <path-to-checkout>.
The skills call the CLI, so install the package in the project you are evaluating as well (see
Install). Claude Code names the skills calibrated-agent-evaluation:<skill>.
For Kiro, copy each skills/<name>/ directory from the repository into your project's
.kiro/skills/ or into ~/.kiro/skills/.
The five skills¶
| Skill | Ask it to | It runs |
|---|---|---|
audit-eval-pipeline |
Review how you evaluate today: unchecked judges gating CI, 1 to 5 scores, raw judge counts reported as rates, "10/10 passed" with no range, fixes accepted on the traces they were tuned on | Reads only, then recommends the command or skill that fixes each finding |
run-error-analysis |
Find and name your agent's failure modes from real traces, before any metric | ingest (files), capture, probe, define, generate, cluster; prints the label --serve command for you to run |
build-binary-judge |
Turn one named failure mode into a judge file: one pass/fail question, critique before verdict | Writes the judge file and checks it with judge --check |
validate-judge |
Find out whether a judge can be trusted to score, gate CI or report a corrected rate | calibrate, bias, judge, run |
build-annotation-ui |
Get the review app ready so a person can label traces with hotkeys | checks the store and the ui extra, then prints the label --serve command for you to run |
A test in the repository checks every command and flag the skills mention against the shipped CLI, so a renamed flag fails the build.
A first session¶
Starting from traces you already have:
- "Audit how we evaluate the refunds agent."
audit-eval-pipelinereads your eval code and lists what to fix. - "Ingest last week's traces and help me find the failure modes."
run-error-analysisingests your exports, prints the command that opens the review app, and runsclusteronce you have labeled a sample. - "Write a judge for the trusted_broken_policy_tool failure mode."
build-binary-judgedrafts the judge file. - "Can we trust it?"
validate-judgerunscalibrateandbiasand tells you what they decided, including how many more labels you need if the judge isn't ready.