Skip to content

Configure the CI job

calibrated-agent-evaluation run --ci exits 0 when every failure mode passes and 1 when one does not, naming it. Any CI system can use the exit code. Why each mode is gated the way it is, and how many traces a mode needs, is on Gate your pull requests.

calibrated-agent-evaluation --root .calibrated-agent-evaluation quickstart
calibrated-agent-evaluation run --verdicts human --ci

On the six-trace sample corpus, with the human labels as verdicts, every mode fails, because two or three traces per mode, one of them failing, are too few to pass:

$ calibrated-agent-evaluation run --verdicts human --ci
calibrated-agent-evaluation run: CI gate failed: 4 of 4 failure mode(s) don't clear the bar yet: intake/dropped_material_fact, orderbot/trusted_broken_policy_tool, policy-checker/trusted_broken_policy_tool, vacancy_threshold_margin_mishandled.
  (The bar: at least 80% of each mode's traces must pass, even at the cautious end of what this many traces can show.)
No failure rate estimated: pass --judge FILE to correct a judge's verdicts.
Next: fix the failing modes, or label more traces where a mode has too few to show its pass rate.
Run with --explain for the numbers behind this.

The job

%%{init: {"themeVariables": {"edgeLabelBackground": "transparent"}}}%%
flowchart TD
    pr([PR opened]) --> load[Load the committed evaluation set]
    load --> add["Add the PR traces"]
    add --> src{Which verdicts?}
    src -->|human labels or code scorers| gate
    src -->|LLM judge| trust{"judge admits it?<br/>certified and bias-checked"}
    trust -->|no: no verdicts written| refuse
    trust -->|yes| score["Score: recorded replies or live"]
    score --> gate["Gate: run --ci"]
    gate -->|all modes clear| pass(["exit 0: pass"])
    gate -->|a mode falls short| fail(["exit 1: fail"])
    gate -->|no verdicts| refuse(["exit 1: refusal"])

Install the package in the job

Install from PyPI with the stats extra and pin the version you tested against.

name: eval-gate
on: [pull_request]

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v5
      - run: uv tool install "calibrated-agent-evaluation[stats]==0.1.0"
      - run: calibrated-agent-evaluation load eval-set
      - run: calibrated-agent-evaluation run --ci --verdicts human
eval-gate:
  image: ghcr.io/astral-sh/uv:python3.13-bookworm
  script:
    - uv tool install "calibrated-agent-evaluation[stats]==0.1.0"
    - export PATH="$(uv tool dir --bin):$PATH"
    - calibrated-agent-evaluation load eval-set
    - calibrated-agent-evaluation run --ci --verdicts human
Slow start in a crowded CI image

Pydantic scans installed packages for plugins on first validation. If the job uses none, set PYDANTIC_DISABLE_PLUGINS=__all__.

Seed the job's store

calibrated-agent-evaluation dump writes the traces, labels and judge records to a directory you commit, and calibrated-agent-evaluation load restores it as the job's first step (docs/site/examples/integrate/ci_seed_and_gate.py):

# once, on your machine: build the evaluation set and dump it for the repo

$ calibrated-agent-evaluation --root eval-store quickstart
(exit 0)

$ calibrated-agent-evaluation --root eval-store dump eval-set
dumped the artifact set to eval-set (CSV+JSON); no records
(exit 0)

# in the CI job: restore the set into a fresh store and gate

$ calibrated-agent-evaluation --root .calibrated-agent-evaluation load eval-set
loaded the artifact set from eval-set into the backend; no records
(exit 0)

$ calibrated-agent-evaluation --root .calibrated-agent-evaluation run --verdicts human --ci
calibrated-agent-evaluation run: CI gate failed: 4 of 4 failure mode(s) don't clear the bar yet: intake/dropped_material_fact, orderbot/trusted_broken_policy_tool, policy-checker/trusted_broken_policy_tool, vacancy_threshold_margin_mishandled.
  (The bar: at least 80% of each mode's traces must pass, even at the cautious end of what this many traces can show.)
No failure rate estimated: pass --judge FILE to correct a judge's verdicts.
Next: fix the failing modes, or label more traces where a mode has too few to show its pass rate.
Run with --explain for the numbers behind this.
(exit 1)

load checks every file against the sha256 recorded in dump-manifest.json, so commit the directory byte for byte. The CSV files end their lines with CRLF, which git rewrites under core.autocrlf or a text=auto rule; add a .gitattributes line such as eval-set/** -text.

Then bring in the PR's traces as a random sample of the traffic you want a verdict on. The gate reads only traces tagged production-representative, and ingest tags untagged traces discovery by default, so pass the frame on purpose: draw a seeded sample with --sample N --seed S, which tags the drawn traces, or pass --frame production-representative for a directory that already is one. Traces from generate --run are tagged discovery and do not count toward the gate. Score the traces with judge --judge FILE, then run the gate:

calibrated-agent-evaluation load eval-set
calibrated-agent-evaluation ingest pr-traces/ --sample 200 --seed 7
calibrated-agent-evaluation judge --judge judge.json --cassette judge-cassette.json
calibrated-agent-evaluation run --ci --judge judge.json

Or point --backend langfuse at a project that holds the set.

A job that scores with a judge but should not carry your labeled set can load the judge alone: the judge file plus the calibration and bias records that admit it.

$ calibrated-agent-evaluation dump kit --judges-only --judge demo/quickstart/judge-v2.json
dumped 1 judge(s) ('orderbot-refund-policy') to kit: each judge file with the calibration record and bias battery that admit it, and no traces or labels. Next: `calibrated-agent-evaluation load kit` into the store that will score with them.
Run with --explain for the numbers behind this.
$ calibrated-agent-evaluation --root ci load kit
loaded 1 judge(s) from kit: ci/judges/judge-v2.json; records: 1 new calibration, 1 new bias. Next: `calibrated-agent-evaluation judge --judge ci/judges/judge-v2.json` scores the traces in this store.

Choose the verdicts

Flag Reads Model calls
--verdicts judge (default) the judge verdicts in scope: each must come from a judge whose certifying calibration and CLEAR bias battery are in the store, or the run stops replayed or live, below
--verdicts scorer code-scorer verdicts from calibrated-agent-evaluation score none
--verdicts human human labels none

If the source you name has no verdicts in scope, the run stops, says so, and names the other sources. --judge FILE adds each judged mode's corrected failure rate to the report next to the gate decision. That rate is read only from randomly sampled traces the judge was not checked against; the report says how many it left out, and --explain lists them.

Under --ci, code-scorer verdicts from a generic text-similarity or quality metric (ROUGE, BLEU, METEOR, BERTScore, helpfulness) are refused as gate input. An LLM judge's verdicts are gated whatever its prompt asks. --atoms FILE names which dimensions are objective; a gated mode must be one of them.

Code scorers

calibrated-agent-evaluation score writes code-scorer verdicts. The compare-to-expected scorers, exact, levenshtein, numeric, json-diff and valid-json (the scorers extra), compare each trace's output with the expected value a person recorded on a FAIL for the mode, or one from --expected FILE (JSONL of trace_id and expected):

$ calibrated-agent-evaluation score --scorer exact --mode wrong_reply --key answer --expected expected-replies.jsonl --explain
Scored 2 trace(s) for wrong_reply with the exact scorer: 1 FAIL, 1 PASS.
4 trace(s) in scope had no expected value to compare with and were not scored; --explain names them.
Next: gate these verdicts with `calibrated-agent-evaluation run --verdicts scorer` (they are code verdicts, not a judge's, so no calibration is involved).

wrote 2 code verdict(s) as rater 'scorer:exact@d40a58a2' (parameters {"key": "answer", "scorer": "exact"}) for mode 'wrong_reply' on 6 stored trace(s)
  expected values from docs/site/examples/guides/expected-replies.jsonl
  not scored: 4 trace(s) with no expected value for this mode: 'CLM-4471', 'CLM-4473', 'CLM-4477', 'ORD-1002'

Two more scorers run the output instead of comparing its text. code runs the trace's code against the --expected tests in a sandbox: local by default, which is best-effort and not a security boundary, or the AgentCore Code Interpreter with --sandbox agentcore, which you should use for untrusted model output. sql runs the trace's query and the --expected query against a seeded SQLite database and compares the result sets. Each run with different parameters is a separate rater in the store.

Judge model calls

  • Replay, for a fixed regression set: record the judge's replies once with --cassette PATH --cassette-mode record on calibrate, bias or judge, commit the cassette, and pass --cassette PATH in the job. Replay is the default mode and needs no credentials; a trace the cassette never recorded fails. Record mode always calls the live model and overwrites replies already in the cassette.
  • Live, for new PR traces: the inspect extra, and credentials from the CI secret store. For Bedrock, use an assumed IAM role, not long-lived keys.

Exit codes

Code Meaning
0 The run finished. With --ci, the gate also passed.
1 With --ci, a mode failed the gate. Also any refusal, and a failed --publish.
2 A usage error: an unknown flag or value.
3 With --fail-on-drift, input drift or a stale judge was flagged and the gate did not fail.

A failing gate exits 1 even when drift is flagged too, so a job that treats any non-zero code as a failure needs no special case. A job that wants drift to warn without blocking can allow exit 3.

Compare with the last release

Dump the store when you cut a release, load the dump into its own root, and pass it as the baseline:

$ calibrated-agent-evaluation run \
    --judge demo/quickstart/judge-v2.json \
    --drift-baseline baseline \
    --drift-feature input-chars \
    --fail-on-drift
CI gate failed: 1 of 1 failure mode(s) don't clear the bar yet: trusted_broken_policy_tool.
  (The bar: at least 80% of each mode's traces must pass, even at the cautious end of what this many traces can show.)
trusted_broken_policy_tool failure rate: 20% (likely between 14% and 26%).
Next: fix the failing modes, or label more traces where a mode has too few to show its pass rate.
(Without --ci this run is informational and exits 0.)
Input drift on input-chars: none found (200 current traces against 200 in the baseline).
  (The mix of taxonomy cells moved by 0.000 on a 0 to 1 scale; shown for reference, it never raises a flag.)
Run with --explain for the numbers behind this.

--drift-feature is required with --drift-baseline and takes input-chars, turns or tool-calls. Drift never changes the gate decision. When drift is flagged, each judge given with --judge also gets a staleness warning, and --fail-on-drift turns either into exit 3. With fewer than 50 traces on a side, the run says drift was not measured. Check for drift explains how to read the measures.

Publish and export

--publish local writes the run's corrected rates and drift measures to <root>/dashboard/<run_id>/ as CSV and JSON. --publish langfuse writes the same files and also sends each value as a score on a trace named for the run, in the project the LANGFUSE_* variables name, whatever --backend is. The run is recorded before the publish starts. If the publish fails, the job exits 1 and prints Run recorded as <id>; publish failed: ....

calibrated-agent-evaluation results --out FILE writes every stored estimate (bounds, rates, drift, and each refusal with its reason) to one CSV, a row each:

$ calibrated-agent-evaluation results --out results.csv
Wrote 14 estimate(s) from 6 record(s) (2 calibration, 1 run, 2 bias, 1 judge) to results.csv.
Run with --explain for the numbers behind this.

See also the run reference.