Skip to content

Check for drift

Find out whether production traffic still looks like the traces your judge was checked on.

A judge is checked on one set of traces. If production moves away from that set (longer conversations, a new product line, more late returns), how often the judge is right may change too. calibrated-agent-evaluation run --drift-baseline STORE --drift-feature input-chars|turns|tool-calls runs this check on every run (see Configure the CI job). This page calls the library directly, for an input the CLI does not offer as a --drift-feature.

Steps

Pick an input you can compute from every trace, collect it for the calibration traces and for a production sample, and call calibrated_agent_evaluation.run.input_drift. This program does that for the bundled judge kit, on two dimensions:

# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
# SPDX-License-Identifier: Apache-2.0

"""Check whether production traffic still looks like the traces a judge was calibrated on.

Seeds the bundled judge kit, then compares two dimensions (the length of the customer's message,
and the days since delivery the policy tool reported) between the calibration traces and the
production sample with ``calibrated_agent_evaluation.run.input_drift``. Run from the repository root
with the ``stats`` extra::

    uv run --extra stats python docs/site/examples/guides/drift_check.py
"""

from __future__ import annotations

import json
import subprocess
import sys
import tempfile
from pathlib import Path

from calibrated_agent_evaluation.backends.local import LocalBackend
from calibrated_agent_evaluation.run import input_drift
from calibrated_agent_evaluation.schema import Trace

CALIBRATED_AGENT_EVALUATION = str(Path(sys.executable).parent / "calibrated-agent-evaluation")


def message_length(trace: Trace) -> float:
    return float(len(json.dumps(trace.input)))


def days_since_delivery(trace: Trace) -> float:
    """What the policy tool reported; the hunted calibration traces lean toward late returns."""
    for span in trace.spans:
        for call in span.tool_calls:
            if call.name == "check_refund_policy" and isinstance(call.result, dict):
                return float(call.result["days_since_delivery"])
    raise ValueError(f"{trace.trace_id}: no check_refund_policy result")


DETAIL = "--detail" in sys.argv[1:]


def show(title: str, dimension: str, baseline: list[float], fresh: list[float]) -> None:
    report = input_drift(baseline, fresh, dimension=dimension, seed=0)
    if not DETAIL:
        verdict = (
            "drifted: the two samples differ by more than chance and by enough to matter"
            if report.any_exceeded
            else "no drift: the two samples look alike"
        )
        print(f"{dimension} ({report.n_baseline} vs {report.n_fresh} traces): {verdict}")
        return
    print(f"{title}, {dimension} ({report.n_baseline} vs {report.n_fresh} traces)")
    for m in report.metrics:
        if m.name == "jsd":
            print(f"  {m.name:<12} {m.value:.3f}  p={m.p_value:.3f}  reported only")
            continue
        bar = "none" if m.threshold is None else f"{m.threshold:.2f}"
        flag = "DRIFT" if m.exceeded else "ok"
        print(f"  {m.name:<12} {m.value:.3f}  p={m.p_value:.3f}  bar={bar:<5} {flag}")
    print()


with tempfile.TemporaryDirectory() as tmp:
    store = Path(tmp) / "store"
    subprocess.run(  # noqa: S603 - fixed argv
        [CALIBRATED_AGENT_EVALUATION, "--root", str(store), "quickstart", "--certify"],
        capture_output=True,
        check=True,
        timeout=120,
    )
    with LocalBackend(store) as backend:
        calibration = backend.query(frame="discovery")
        production = backend.query(frame="production-representative")

for name, feature in (
    ("message length", message_length),
    ("days since delivery", days_since_delivery),
):
    show(
        "calibration traces vs production sample",
        name,
        [feature(t) for t in calibration],
        [feature(t) for t in production],
    )
message length (648 vs 200 traces): no drift: the two samples look alike
days since delivery (648 vs 200 traces): drifted: the two samples differ by more than chance and by enough to matter
The same check, with the numbers behind each verdict

Run the program with --detail to see each measure and the bar it must clear.

calibration traces vs production sample, message length (648 vs 200 traces)
  psi          0.042  p=0.690  bar=0.25  ok
  jsd          0.008  p=0.684  reported only
  wasserstein  0.046  p=0.745  bar=0.10  ok

calibration traces vs production sample, days since delivery (648 vs 200 traces)
  psi          0.523  p=0.001  bar=0.25  DRIFT
  jsd          0.095  p=0.001  reported only
  wasserstein  0.363  p=0.001  bar=0.10  DRIFT

Message length looks the same in both sets. Days since delivery does not: the calibration traces were picked to find failures, and late returns are a common failure, so they lean late. That is expected for a hand-picked calibration set, and it is why a judge's failure rate is measured on a separate random sample.

Reading the output

The first block is the answer for each input. The detail gives three measures of how far apart the two samples are. Two of them can raise a flag: psi compares how the values spread across ten bins, and wasserstein measures how far the values moved. A measure flags only when the difference is larger than chance would produce and also large enough to matter (for wasserstein, a shift of a tenth of the width of the middle half of the baseline's values), so a sample of thousands does not flag a trivial shift. The third, jsd, reads the same bins as psi and is shown for reference only. Methods names each test and its source.

To check whether the judge itself changed, rather than its inputs, run calibrated-agent-evaluation calibrate --check against fresh labels (see Watch production). In the library, calibrated_agent_evaluation.run.judge_drift compares two calibration records on separate samples and judge_drift_paired compares two runs on the same traces.

Pitfalls

A flag on a small sample can be noise

When nothing changed, the check still flags about 6% of the time: between 5.8% and 7.0% across five kinds of data, each measured over 2,000 random splits of 100 against 100 traces (or 50 against 50). The committed measurement is fixtures/expected/drift/null_calibration.txt. Confirm a flag on a fresh sample before acting on it.

Small samples miss small shifts

In the same file, a shift of half a standard deviation between two samples of 100 is flagged about 90% of the time, and a shift of a fifth of a standard deviation about 24% of the time. At 1,000 per side the smaller shift is flagged 93% of the time. A shift below the bar is not flagged however large the sample.

Drift is not failure

A flagged dimension says the inputs changed. Check the judge against fresh human labels on the new traffic before trusting its rate.

Each side needs enough traces

With the default 10 bins, fewer than 50 values per side gets a "not measured" note, not a "no drift" answer.

See the calibrated_agent_evaluation.run reference for the full signatures.