Configure the CI job¶
calibrated-agent-evaluation run --ci exits 0 when every failure mode passes and 1 when one does
not, naming it. Any CI system can use the exit code. Why each mode is gated the way it is, and how
many traces a mode needs, is on Gate your pull requests.
calibrated-agent-evaluation --root .calibrated-agent-evaluation quickstart
calibrated-agent-evaluation run --verdicts human --ci
On the six-trace sample corpus, with the human labels as verdicts, every mode fails, because two or three traces per mode, one of them failing, are too few to pass:
$ calibrated-agent-evaluation run --verdicts human --ci
calibrated-agent-evaluation run: CI gate failed: 4 of 4 failure mode(s) don't clear the bar yet: intake/dropped_material_fact, orderbot/trusted_broken_policy_tool, policy-checker/trusted_broken_policy_tool, vacancy_threshold_margin_mishandled.
(The bar: at least 80% of each mode's traces must pass, even at the cautious end of what this many traces can show.)
No failure rate estimated: pass --judge FILE to correct a judge's verdicts.
Next: fix the failing modes, or label more traces where a mode has too few to show its pass rate.
Run with --explain for the numbers behind this.
The job¶
%%{init: {"themeVariables": {"edgeLabelBackground": "transparent"}}}%%
flowchart TD
pr([PR opened]) --> load[Load the committed evaluation set]
load --> add["Add the PR traces"]
add --> src{Which verdicts?}
src -->|human labels or code scorers| gate
src -->|LLM judge| trust{"judge admits it?<br/>certified and bias-checked"}
trust -->|no: no verdicts written| refuse
trust -->|yes| score["Score: recorded replies or live"]
score --> gate["Gate: run --ci"]
gate -->|all modes clear| pass(["exit 0: pass"])
gate -->|a mode falls short| fail(["exit 1: fail"])
gate -->|no verdicts| refuse(["exit 1: refusal"])
Install the package in the job¶
Install from PyPI with the stats extra and pin the version you tested against.
name: eval-gate
on: [pull_request]
jobs:
evaluate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v5
- run: uv tool install "calibrated-agent-evaluation[stats]==0.1.0"
- run: calibrated-agent-evaluation load eval-set
- run: calibrated-agent-evaluation run --ci --verdicts human
Slow start in a crowded CI image
Pydantic scans installed packages for plugins on first validation. If the job uses none, set
PYDANTIC_DISABLE_PLUGINS=__all__.
Seed the job's store¶
calibrated-agent-evaluation dump writes the traces, labels and judge records to a directory you
commit, and calibrated-agent-evaluation load restores it as the job's first step
(docs/site/examples/integrate/ci_seed_and_gate.py):
# once, on your machine: build the evaluation set and dump it for the repo
$ calibrated-agent-evaluation --root eval-store quickstart
(exit 0)
$ calibrated-agent-evaluation --root eval-store dump eval-set
dumped the artifact set to eval-set (CSV+JSON); no records
(exit 0)
# in the CI job: restore the set into a fresh store and gate
$ calibrated-agent-evaluation --root .calibrated-agent-evaluation load eval-set
loaded the artifact set from eval-set into the backend; no records
(exit 0)
$ calibrated-agent-evaluation --root .calibrated-agent-evaluation run --verdicts human --ci
calibrated-agent-evaluation run: CI gate failed: 4 of 4 failure mode(s) don't clear the bar yet: intake/dropped_material_fact, orderbot/trusted_broken_policy_tool, policy-checker/trusted_broken_policy_tool, vacancy_threshold_margin_mishandled.
(The bar: at least 80% of each mode's traces must pass, even at the cautious end of what this many traces can show.)
No failure rate estimated: pass --judge FILE to correct a judge's verdicts.
Next: fix the failing modes, or label more traces where a mode has too few to show its pass rate.
Run with --explain for the numbers behind this.
(exit 1)
load checks every file against the sha256 recorded in dump-manifest.json, so commit the
directory byte for byte. The CSV files end their lines with CRLF, which git rewrites under
core.autocrlf or a text=auto rule; add a .gitattributes line such as eval-set/** -text.
Then bring in the PR's traces as a random sample of the traffic you want a verdict on. The gate
reads only traces tagged production-representative, and ingest tags untagged traces discovery
by default, so pass the frame on purpose: draw a seeded sample with --sample N --seed S, which
tags the drawn traces, or pass --frame production-representative for a directory that already is
one. Traces from generate --run are tagged discovery and do not count toward the gate. Score the
traces with judge --judge FILE, then run the gate:
calibrated-agent-evaluation load eval-set
calibrated-agent-evaluation ingest pr-traces/ --sample 200 --seed 7
calibrated-agent-evaluation judge --judge judge.json --cassette judge-cassette.json
calibrated-agent-evaluation run --ci --judge judge.json
Or point --backend langfuse at a project that holds the set.
A job that scores with a judge but should not carry your labeled set can load the judge alone: the judge file plus the calibration and bias records that admit it.
$ calibrated-agent-evaluation dump kit --judges-only --judge demo/quickstart/judge-v2.json
dumped 1 judge(s) ('orderbot-refund-policy') to kit: each judge file with the calibration record and bias battery that admit it, and no traces or labels. Next: `calibrated-agent-evaluation load kit` into the store that will score with them.
Run with --explain for the numbers behind this.
$ calibrated-agent-evaluation --root ci load kit
loaded 1 judge(s) from kit: ci/judges/judge-v2.json; records: 1 new calibration, 1 new bias. Next: `calibrated-agent-evaluation judge --judge ci/judges/judge-v2.json` scores the traces in this store.
Choose the verdicts¶
| Flag | Reads | Model calls |
|---|---|---|
--verdicts judge (default) |
the judge verdicts in scope: each must come from a judge whose certifying calibration and CLEAR bias battery are in the store, or the run stops | replayed or live, below |
--verdicts scorer |
code-scorer verdicts from calibrated-agent-evaluation score |
none |
--verdicts human |
human labels | none |
If the source you name has no verdicts in scope, the run stops, says so, and names the other
sources. --judge FILE adds each judged mode's corrected failure rate to the report next
to the gate decision. That rate is read only from randomly sampled traces the judge was not checked
against; the report says how many it left out, and --explain lists them.
Under --ci, code-scorer verdicts from a generic text-similarity or quality metric (ROUGE, BLEU,
METEOR, BERTScore, helpfulness) are refused as gate input. An LLM judge's verdicts are gated
whatever its prompt asks. --atoms FILE names which dimensions are
objective; a gated mode must be one of them.
Code scorers¶
calibrated-agent-evaluation score writes code-scorer verdicts. The compare-to-expected scorers,
exact, levenshtein, numeric, json-diff and valid-json (the scorers extra), compare each
trace's output with the expected value a person recorded on a FAIL for the mode, or one from
--expected FILE (JSONL of trace_id and expected):
$ calibrated-agent-evaluation score --scorer exact --mode wrong_reply --key answer --expected expected-replies.jsonl --explain
Scored 2 trace(s) for wrong_reply with the exact scorer: 1 FAIL, 1 PASS.
4 trace(s) in scope had no expected value to compare with and were not scored; --explain names them.
Next: gate these verdicts with `calibrated-agent-evaluation run --verdicts scorer` (they are code verdicts, not a judge's, so no calibration is involved).
wrote 2 code verdict(s) as rater 'scorer:exact@d40a58a2' (parameters {"key": "answer", "scorer": "exact"}) for mode 'wrong_reply' on 6 stored trace(s)
expected values from docs/site/examples/guides/expected-replies.jsonl
not scored: 4 trace(s) with no expected value for this mode: 'CLM-4471', 'CLM-4473', 'CLM-4477', 'ORD-1002'
Two more scorers run the output instead of comparing its text. code runs the trace's code against
the --expected tests in a sandbox: local by default, which is best-effort and not a security
boundary, or the AgentCore Code Interpreter with --sandbox agentcore, which you should use for
untrusted model output. sql runs the trace's query and the --expected query against a seeded
SQLite database and compares the result sets. Each run with different parameters is a separate
rater in the store.
Judge model calls¶
- Replay, for a fixed regression set: record the judge's replies once with
--cassette PATH --cassette-mode recordoncalibrate,biasorjudge, commit the cassette, and pass--cassette PATHin the job. Replay is the default mode and needs no credentials; a trace the cassette never recorded fails. Record mode always calls the live model and overwrites replies already in the cassette. - Live, for new PR traces: the
inspectextra, and credentials from the CI secret store. For Bedrock, use an assumed IAM role, not long-lived keys.
Exit codes¶
| Code | Meaning |
|---|---|
| 0 | The run finished. With --ci, the gate also passed. |
| 1 | With --ci, a mode failed the gate. Also any refusal, and a failed --publish. |
| 2 | A usage error: an unknown flag or value. |
| 3 | With --fail-on-drift, input drift or a stale judge was flagged and the gate did not fail. |
A failing gate exits 1 even when drift is flagged too, so a job that treats any non-zero code as a failure needs no special case. A job that wants drift to warn without blocking can allow exit 3.
Compare with the last release¶
Dump the store when you cut a release, load the dump into its own root, and pass it as the baseline:
$ calibrated-agent-evaluation run \
--judge demo/quickstart/judge-v2.json \
--drift-baseline baseline \
--drift-feature input-chars \
--fail-on-drift
CI gate failed: 1 of 1 failure mode(s) don't clear the bar yet: trusted_broken_policy_tool.
(The bar: at least 80% of each mode's traces must pass, even at the cautious end of what this many traces can show.)
trusted_broken_policy_tool failure rate: 20% (likely between 14% and 26%).
Next: fix the failing modes, or label more traces where a mode has too few to show its pass rate.
(Without --ci this run is informational and exits 0.)
Input drift on input-chars: none found (200 current traces against 200 in the baseline).
(The mix of taxonomy cells moved by 0.000 on a 0 to 1 scale; shown for reference, it never raises a flag.)
Run with --explain for the numbers behind this.
--drift-feature is required with --drift-baseline and takes input-chars, turns or
tool-calls. Drift never changes the gate decision. When drift is flagged, each judge given with
--judge also gets a staleness warning, and --fail-on-drift turns either into exit 3. With fewer
than 50 traces on a side, the run says drift was not measured. Check for drift
explains how to read the measures.
Publish and export¶
--publish local writes the run's corrected rates and drift measures to
<root>/dashboard/<run_id>/ as CSV and JSON. --publish langfuse writes the same files and also
sends each value as a score on a trace named for the run, in the project the LANGFUSE_* variables
name, whatever --backend is. The run is recorded before the publish starts. If the publish fails,
the job exits 1 and prints Run recorded as <id>; publish failed: ....
calibrated-agent-evaluation results --out FILE writes every stored estimate (bounds, rates, drift,
and each refusal with its reason) to one CSV, a row each:
$ calibrated-agent-evaluation results --out results.csv
Wrote 14 estimate(s) from 6 record(s) (2 calibration, 1 run, 2 bias, 1 judge) to results.csv.
Run with --explain for the numbers behind this.
See also the run reference.