Check for drift¶
Find out whether production traffic still looks like the traces your judge was checked on.
A judge is checked on one set of traces. If production moves away from that set (longer
conversations, a new product line, more late returns), how often the judge is right may change too.
calibrated-agent-evaluation run --drift-baseline STORE --drift-feature input-chars|turns|tool-calls
runs this check on every run (see Configure the CI job).
This page calls the library directly, for an input the CLI does not offer as a --drift-feature.
Steps¶
Pick an input you can compute from every trace, collect it for the calibration traces
and for a production sample, and call calibrated_agent_evaluation.run.input_drift. This program does that for the
bundled judge kit, on two dimensions:
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
# SPDX-License-Identifier: Apache-2.0
"""Check whether production traffic still looks like the traces a judge was calibrated on.
Seeds the bundled judge kit, then compares two dimensions (the length of the customer's message,
and the days since delivery the policy tool reported) between the calibration traces and the
production sample with ``calibrated_agent_evaluation.run.input_drift``. Run from the repository root
with the ``stats`` extra::
uv run --extra stats python docs/site/examples/guides/drift_check.py
"""
from __future__ import annotations
import json
import subprocess
import sys
import tempfile
from pathlib import Path
from calibrated_agent_evaluation.backends.local import LocalBackend
from calibrated_agent_evaluation.run import input_drift
from calibrated_agent_evaluation.schema import Trace
CALIBRATED_AGENT_EVALUATION = str(Path(sys.executable).parent / "calibrated-agent-evaluation")
def message_length(trace: Trace) -> float:
return float(len(json.dumps(trace.input)))
def days_since_delivery(trace: Trace) -> float:
"""What the policy tool reported; the hunted calibration traces lean toward late returns."""
for span in trace.spans:
for call in span.tool_calls:
if call.name == "check_refund_policy" and isinstance(call.result, dict):
return float(call.result["days_since_delivery"])
raise ValueError(f"{trace.trace_id}: no check_refund_policy result")
DETAIL = "--detail" in sys.argv[1:]
def show(title: str, dimension: str, baseline: list[float], fresh: list[float]) -> None:
report = input_drift(baseline, fresh, dimension=dimension, seed=0)
if not DETAIL:
verdict = (
"drifted: the two samples differ by more than chance and by enough to matter"
if report.any_exceeded
else "no drift: the two samples look alike"
)
print(f"{dimension} ({report.n_baseline} vs {report.n_fresh} traces): {verdict}")
return
print(f"{title}, {dimension} ({report.n_baseline} vs {report.n_fresh} traces)")
for m in report.metrics:
if m.name == "jsd":
print(f" {m.name:<12} {m.value:.3f} p={m.p_value:.3f} reported only")
continue
bar = "none" if m.threshold is None else f"{m.threshold:.2f}"
flag = "DRIFT" if m.exceeded else "ok"
print(f" {m.name:<12} {m.value:.3f} p={m.p_value:.3f} bar={bar:<5} {flag}")
print()
with tempfile.TemporaryDirectory() as tmp:
store = Path(tmp) / "store"
subprocess.run( # noqa: S603 - fixed argv
[CALIBRATED_AGENT_EVALUATION, "--root", str(store), "quickstart", "--certify"],
capture_output=True,
check=True,
timeout=120,
)
with LocalBackend(store) as backend:
calibration = backend.query(frame="discovery")
production = backend.query(frame="production-representative")
for name, feature in (
("message length", message_length),
("days since delivery", days_since_delivery),
):
show(
"calibration traces vs production sample",
name,
[feature(t) for t in calibration],
[feature(t) for t in production],
)
message length (648 vs 200 traces): no drift: the two samples look alike
days since delivery (648 vs 200 traces): drifted: the two samples differ by more than chance and by enough to matter
The same check, with the numbers behind each verdict
Run the program with --detail to see each measure and the bar it must clear.
calibration traces vs production sample, message length (648 vs 200 traces)
psi 0.042 p=0.690 bar=0.25 ok
jsd 0.008 p=0.684 reported only
wasserstein 0.046 p=0.745 bar=0.10 ok
calibration traces vs production sample, days since delivery (648 vs 200 traces)
psi 0.523 p=0.001 bar=0.25 DRIFT
jsd 0.095 p=0.001 reported only
wasserstein 0.363 p=0.001 bar=0.10 DRIFT
Message length looks the same in both sets. Days since delivery does not: the calibration traces were picked to find failures, and late returns are a common failure, so they lean late. That is expected for a hand-picked calibration set, and it is why a judge's failure rate is measured on a separate random sample.
Reading the output¶
The first block is the answer for each input. The detail gives three measures of how far apart the
two samples are. Two of them can raise a flag: psi compares how the values spread across ten
bins, and wasserstein measures how far the values moved. A measure flags only when the difference
is larger than chance would produce and also large enough to matter (for wasserstein, a shift of
a tenth of the width of the middle half of the baseline's values), so a sample of thousands does not flag a trivial shift. The third,
jsd, reads the same bins as psi and is shown for reference only. Methods
names each test and its source.
To check whether the judge itself changed, rather than its inputs, run
calibrated-agent-evaluation calibrate --check against fresh labels (see
Watch production). In the library,
calibrated_agent_evaluation.run.judge_drift compares two calibration records on separate samples
and judge_drift_paired compares two runs on the same traces.
Pitfalls¶
A flag on a small sample can be noise¶
When nothing changed, the check still flags about 6% of the time: between 5.8% and 7.0% across five
kinds of data, each measured over 2,000 random splits of 100 against 100 traces (or 50 against 50).
The committed measurement is fixtures/expected/drift/null_calibration.txt. Confirm a flag on a
fresh sample before acting on it.
Small samples miss small shifts¶
In the same file, a shift of half a standard deviation between two samples of 100 is flagged about 90% of the time, and a shift of a fifth of a standard deviation about 24% of the time. At 1,000 per side the smaller shift is flagged 93% of the time. A shift below the bar is not flagged however large the sample.
Drift is not failure¶
A flagged dimension says the inputs changed. Check the judge against fresh human labels on the new traffic before trusting its rate.
Each side needs enough traces¶
With the default 10 bins, fewer than 50 values per side gets a "not measured" note, not a "no drift" answer.
See the calibrated_agent_evaluation.run reference for the full signatures.