calibrated_agent_evaluation.run¶
calibrated_agent_evaluation.run ¶
The run / CI-gate stage -- the run step of probe -> ... -> run -> improve.
The point where the flat per-trace Labels that score_traces and
run_judge already produced become a CI pass/fail gate, a per-failure-mode trend report
(prevalence-corrected rates with CIs, never a single aggregate accuracy number), a passive
staleness warning, and drift detection (PSI/JSD/Wasserstein + judge ΔTPR/ΔTNR). It covers
multiplicity control and confidence-treatment everywhere.
Pure methodology plane: this package imports no vendor SDK type (a static test guards it). It
reuses the stats primitives (multiplicity/corrected_rate/wilson_lower), enforce_ci_gate,
the scorer/judge outputs, assert_production_frame, and the drift block in
drift -- it re-implements none of them. The only vendor
import lives in the confined backends/dashboard.py adapter. The library gate returns a typed
RunGateDecision and a persisted RunRecord; the CLI wires
calibrated-agent-evaluation run --ci|--cadence onto it and maps the decision to a process exit
code.
The record types, errors, store and per-mode aggregation load eagerly and need nothing beyond the
base install. The gate, trend, drift, staleness and multiplicity functions need the
stats extra, so they load lazily; backends/dashboard.py imports the report types under the
langfuse extra alone.
RunError ¶
Bases: CalibratedAgentEvaluationDataError
Base class for every run-stage refusal.
RunGateError ¶
Bases: RunError
The CI gate refuses to produce a pass/fail decision.
Raised when an exploration-lane scorer is presented as a gate, or when the gate is
asked to run over an empty input (no traces / no verdicts) rather than returning a vacuous
passed=True.
RunStoreError ¶
Bases: RunError
A run-record / dashboard-sink file read or write failed.
Wraps the underlying OSError / json.JSONDecodeError / csv failure so a
filesystem/parse error surfaces inside the typed family rather than as a raw stdlib exception
escaping the store's public surface ("wrap external errors at the boundary"). A stored
schema_version != the running RUN_SCHEMA_VERSION also refuses here (no coercion).
JudgeCheckRecord
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
One re-check of a certified judge on fresh gold.
against_calib_id is the certifying record the drift is measured against. tpr and
tnr are the judge's rates on the fresh gold with their 95% lower bounds; drift is the
change from the certifying record, each with its interval. drifted is true when either
change's interval lies wholly below zero. needed_fail and needed_pass are the
per-class sizes check_size asked for.
Config:
frozen:Trueextra:forbid
Fields:
-
check_id(str) -
judge_id(str) -
model(str) -
prompt_version(str) -
judge_fingerprint(str | None) -
against_calib_id(str) -
check_set_id(str) -
n_pos(int) -
n_neg(int) -
tpr(Bound) -
tnr(Bound) -
drift(JudgeDriftReport) -
drifted(bool) -
needed_fail(int) -
needed_pass(int) -
agreement_alpha(Interval | None) -
trace_ids(tuple[str, ...]) -
schema_version(str) -
provenance(ProvenanceStamp)
DiscoveryEnqueue
dataclass
¶
A CI-failing trace routed into the labeling queue tagged as discovery.
frame is always "discovery": a trace the CI gate caught is non-random augmentation (not
a random draw), so it grows the discovery set but must never enter production-frame estimation
(the assert_production_frame guard enforces that downstream).
DriftMetric
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
One input-drift statistic, its permutation p-value, and whether it was flagged.
value is the statistic as method computes it (PSI over baseline-quantile bins, JSD
over the same bins, Wasserstein divided by the pooled IQR). p_value is its permutation
test against the same-distribution null. For PSI and Wasserstein, exceeded is True only
when that p-value survives the BY-FDR correction across those two AND, where threshold is
set, value is above it; threshold=None means significance alone decides. JSD is
reported only: its threshold is None and exceeded is always False. Records written
before JSD left the gate may carry a JSD flag from the older three-metric family.
Config:
frozen:Trueextra:forbid
Fields:
InputDriftReport
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
Input-distribution drift: PSI + JSD + Wasserstein over one input dimension.
multiplicity is the BY-FDR family the gated metrics' p-values were corrected in (PSI and
Wasserstein, in that order; JSD is reported outside it), and
n_permutations / seed say how those p-values were drawn, so the flags can be recomputed
from the record. n_baseline / n_fresh are the two sample sizes.
Config:
frozen:Trueextra:forbid
Fields:
-
metrics(tuple[DriftMetric, ...]) -
dimension(str) -
multiplicity(MultiplicityResult | None) -
n_permutations(int | None) -
seed(int | None) -
n_baseline(int | None) -
n_fresh(int | None)
any_exceeded
property
¶
True when any metric crossed its threshold (the staleness input).
JudgeDriftReport
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
Judge drift vs a frozen baseline: ΔTPR / ΔTNR, each with a CI.
Each delta is an Interval (point = the
delta, lo/hi the CI) so the confidence treatment travels with the number. There is no
Δκ: TPR and TNR describe a binary judge completely, and κ on a hunted calibration set also moves
with the hunt's class mix, which says nothing about the judge.
Config:
frozen:Trueextra:forbid
Fields:
ModeCounts
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
Per-mode confusion counts aggregated from the flat verdict/scorer Labels.
n_judged_fail is the judge/scorer FAIL count over the n_total items of this mode; the
gate turns it into a pass-rate lower bound and the trend turns it into a corrected rate.
calib_id names the CalibrationRecord that
corrects this mode (its own judge).
Config:
frozen:Trueextra:forbid
Fields:
ModeGate
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
One gated mode: its verdict counts, its pass-rate 95% lower bound and the bar it must clear.
lower_bound is the one-sided Wilson bound on (n_total - n_fail) / n_total; passed
is lower_bound >= threshold. threshold is this mode's own bar (a per-mode override, or
the gate's default).
Config:
frozen:Trueextra:forbid
Fields:
ModeKey
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
A failure-mode key: failure_type alone, or (agent, failure_type) for a crew.
agent is None for single-agent and workflow-level modes.
Config:
frozen:Trueextra:forbid
Fields:
-
failure_type(str) -
agent(str | None)
MultiplicityResult
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
A corrected family's method + comparison count m + reject vector.
m is stated per run. family names the correction site ("drift_flags",
"improve_guard", ...); method is Holm (a hard-reject guardrail) or fdr_by (exploratory
flags).
Config:
frozen:Trueextra:forbid
Fields:
RunDecision
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
A single tiered run decision -- a staleness warning or an exploratory flag.
inputs is validated non-empty at construction: a recorded decision with no inputs is refused
rather than silently accepted (the EnforcementDecision precedent). Staleness decisions
are always warning-tier and never flip the gate.
Config:
frozen:Trueextra:forbid
Fields:
Validators:
-
_check_invariants
RunGateDecision
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
The hard CI pass/fail verdict.
passed is True iff every gated mode's pass-rate 95% lower bound cleared threshold;
failing_modes names the modes that did not and n_modes counts the modes gated. modes
holds each gated mode's counts and bound, in the order the gate checked them. Requiring
every mode to pass is an intersection-union test, so the false-pass rate is capped at the
per-mode alpha without a correction across modes, and none is recorded. decisions carries
attached non-gating warnings (e.g. staleness): they are recorded but never flip passed.
Config:
frozen:Trueextra:forbid
Fields:
RunRecord
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
The persisted, version-pinned audit artifact.
Records the run mode, the gate outcome, how many modes it gated, which failed and each mode's
counts and lower bound, the judge/calibration ids
and scorer lanes used, the per-mode corrected rates, the drift, every corrected family + its
m, the tiered decisions, and the inputs that produced the decision. inputs is validated
non-empty. Round-trips through store; schema_version
refuses on mismatch (no coercion).
Config:
frozen:Trueextra:forbid
Fields:
-
run_id(str) -
mode(RunMode) -
gate_passed(bool | None) -
judge_ids(tuple[str, ...]) -
calib_ids(tuple[str, ...]) -
scorer_lanes(tuple[str, ...]) -
trend(tuple[TrendPoint, ...]) -
input_drift(InputDriftReport | None) -
judge_drift(tuple[JudgeDriftReport, ...]) -
multiplicity(tuple[MultiplicityResult, ...]) -
decisions(tuple[RunDecision, ...]) -
inputs(dict[str, str]) -
gate_n_modes(int | None) -
gate_failing_modes(tuple[ModeKey, ...]) -
gate_modes(tuple[ModeGate, ...]) -
schema_version(str) -
provenance(ProvenanceStamp)
Validators:
-
_check_invariants
RunReport
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
The full cadence-run rollup -- the vendor-neutral artifact the dashboard sink publishes.
Holds the per-mode trend, the input/judge drift, every corrected
family's multiplicity result, and the tiered decisions (staleness + flags). It carries
no confidence-treatment-free headline number: the only rates it holds are the
per-TrendPoint corrected-rate intervals.
Config:
frozen:Trueextra:forbid
Fields:
-
run_id(str) -
mode(RunMode) -
trend(tuple[TrendPoint, ...]) -
input_drift(InputDriftReport | None) -
judge_drift(tuple[JudgeDriftReport, ...]) -
multiplicity(tuple[MultiplicityResult, ...]) -
decisions(tuple[RunDecision, ...]) -
gate(RunGateDecision | None)
TrendPoint
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
One per failure mode: the Rogan-Gladen corrected rate + CI, beside the raw rate.
corrected_rate is the shared Interval: a
refused correction (J<0.5 or non-identifiable) carries point=None + refusal_reason and
the mode is still present with its widening interval -- never dropped, never a fabricated point.
raw_rate is the judge's uncorrected FAIL fraction on the same rate sample with its
two-sided 95% Wilson interval: what a reader would conclude without calibration. It is
reported so the correction's effect is visible, never as a substitute for corrected_rate;
it stays populated when the correction is refused.
Config:
frozen:Trueextra:forbid
Fields:
DriftThresholds
dataclass
¶
CiGateRequest
dataclass
¶
CiGateRequest(
labels,
gold,
judges=(),
scorers=(),
threshold=DEFAULT_PASSRATE_THRESHOLD,
thresholds=dict(),
run_id="run",
decisions=(),
)
The inputs to one CI-gate run.
labels are the per-trace verdict/scorer results; gold maps each (trace_id, span_id)
to its failure mode, or is a function giving each label's mode (see
aggregate_by_mode; use the function when several modes'
judges score one trace). judges + calib_records are the (judge, its CalibrationRecord)
pairs whose certification is checked before gating; scorers are the code scorers whose
lane is checked. threshold is the default pass-rate lower-bound bar; thresholds
overrides it per mode. run_id labels the resulting decision/record.
aggregate_by_mode ¶
Reduce verdict/scorer labels to per-ModeKey ModeCounts.
gold maps each (trace_id, span_id) to the failure mode it belongs to (the human-ratified
assignment), or is a function giving each label's mode. A map holds one mode per target, so
when several judges for different modes score one trace, pass the function: a map would put
every verdict on the target under whichever mode it holds. A label with no mode (absent from
the map, or None from the function) is skipped -- never silently folded into a wrong mode.
FAIL is the positive class, so n_judged_fail is the count of outcome == "FAIL"
verdicts per mode.
calib_by_mode optionally names the CalibrationRecord id that corrects each mode (its own
judge); recorded on the counts so the trend stage can look up the right record.
Returns an empty dict for an empty label set (the gate turns that into a refusal).
Source code in src/calibrated_agent_evaluation/run/aggregate.py
check_size ¶
The fresh gold per class (FAIL, PASS) a re-check of record's judge needs.
For each class, the smallest n at which a judge showing threshold (the count rounded
up, so the shown rate is never below it) is flagged by the drift interval against the record,
at n and every larger size up to max_n. At the planned size a judge that truly fell
to the threshold is flagged about half the time, and one that fell further more often.
Raises:
| Type | Description |
|---|---|
InfeasiblePlanError
|
No size up to |
Source code in src/calibrated_agent_evaluation/run/judge_check.py
drift_flagged ¶
Whether either rate's change has an interval wholly below zero.
dump_judge_checks ¶
Write records to judge_check_record.{csv,json} under path.
Raises:
| Type | Description |
|---|---|
RunStoreError
|
A bad |
Source code in src/calibrated_agent_evaluation/run/judge_check.py
load_judge_checks ¶
Read judge check records back from path; a missing file is an empty store.
Raises:
| Type | Description |
|---|---|
RunStoreError
|
A version mismatch or any read, parse or validation failure. |
Source code in src/calibrated_agent_evaluation/run/judge_check.py
run_record_of ¶
Build the persisted RunRecord from a RunReport.
gate_passed comes from the report's gate (None for a cadence run with no gate), and the
gate threshold that produced it is stamped on provenance.thresholds["gate"] so the record
keeps it. calib_ids are the trend points' calibration ids, in order, without repeats.
seed / n_resamples are what the trend's corrected rates were computed with; pass the
values given to trend_report. The gate's attached
decisions are recorded alongside the report's own.
Source code in src/calibrated_agent_evaluation/run/report.py
dump_run_records ¶
Write records to run_record.{csv,json} under path.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
records
|
Sequence[RunRecord]
|
The run records to persist. |
required |
path
|
str | Path
|
The directory to write into. |
required |
fmt
|
str
|
Which file a later |
'csv'
|
Raises:
| Type | Description |
|---|---|
RunStoreError
|
A bad |
Source code in src/calibrated_agent_evaluation/run/store.py
load_run_records ¶
Read run records back from path; a missing file is an empty store.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str | Path
|
The directory |
required |
fmt
|
str
|
|
'csv'
|
Returns:
| Type | Description |
|---|---|
list[RunRecord]
|
The stored records. |
Raises:
| Type | Description |
|---|---|
RunStoreError
|
A schema-version mismatch, or any read, parse or validation failure. |
Source code in src/calibrated_agent_evaluation/run/store.py
input_drift ¶
input_drift(
baseline,
fresh,
*,
dimension="input",
thresholds=None,
jsd_baseline=None,
jsd_fresh=None,
bins=DEFAULT_DRIFT_BINS,
n_permutations=DEFAULT_N_PERMUTATIONS,
seed=0,
)
Report PSI + JSD + Wasserstein input drift between baseline and fresh.
baseline/fresh are raw-value samples. PSI bins them on the baseline's quantiles;
Wasserstein is divided by the pooled IQR. JSD reads jsd_baseline/jsd_fresh when supplied
(per-category counts, permuted as category labels); absent those it uses the PSI bins. Each
metric gets a permutation p-value (n_permutations relabellings, seed). PSI and
Wasserstein are corrected together with BY-FDR at q = 0.10 via
flag_slices (family "drift_flags", m = 2), and
either is exceeded when it survives that correction and clears its severity bar. JSD is
reported with its p-value but sits outside the family and is never exceeded. Empty inputs,
or fewer than 5 * bins values per side, raise StatisticsInputError -- never a
spurious zero-drift.
Source code in src/calibrated_agent_evaluation/run/drift_report.py
102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 | |
judge_drift ¶
Report ΔTPR / ΔTNR between two calibration records on independent samples, each with a CI.
Reconstructs the TPR/TNR counts from each record (round(tpr.point * n_pos) etc.) and takes
the difference-of-proportions CI for two independent samples
(stats.drift.delta_proportion_ci). A judge whose TPR/TNR is unchanged yields deltas
straddling 0; a degraded judge yields a delta whose CI sits below 0 -- the signal that the judge
needs re-calibration. before/after are the two frozen calibration records the caller
supplies (the trickle that produces them in production is a later active-cadence feature).
The independent interval is wrong when the two records share items, because the same item
judged twice makes the comparison paired. Sharing is refused with
SampleMisuseError when the records name the same hunted set or any member trace id in
common; use judge_drift_paired with both runs' verdicts instead. A record without
member trace ids (built outside certify) is taken
to be on its own sample.
Source code in src/calibrated_agent_evaluation/run/drift_report.py
judge_drift_paired ¶
Report ΔTPR / ΔTNR between two runs of a judge over the same items, each with a CI.
before and after are (human gold, judge verdict) pairs for the same items, matched on
(trace_id, span_id). Within each gold class, an item judged correctly by one run and not the
other is a discordant pair; the change in TPR (over gold FAIL items) and TNR (over gold PASS
items) gets Newcombe's (1998) paired hybrid-score interval, which uses that pairing instead of
treating the two runs as independent samples.
Raises SampleMisuseError when the two runs do not cover exactly the same items or an
item's gold label differs between them, and StatisticsInputError when a gold class is
empty.
Source code in src/calibrated_agent_evaluation/run/drift_report.py
flag_slices ¶
Flag exploratory cross-slice signals via Benjamini-Yekutieli FDR at q = 0.10.
pvals are the per-slice p-values (one per watched slice); family names the flag family
("drift_flags", "regression_flags", ...). Returns a
MultiplicityResult whose reject[i] is True
for a slice that survives the BY correction (a genuine flag) and whose m == the number of
slices watched (stated per run). Delegates to the multiplicity primitive via
apply_multiplicity; fdr_bh is not offered here -- correlated slices forbid it.
Source code in src/calibrated_agent_evaluation/run/flags.py
discovery_traces ¶
Return the CI-failing traces as discovery-tagged labeling-queue entries.
A trace whose verdict is FAIL is CI-augmentation, not a random draw: it is enqueued tagged
frame="discovery" (the caller does the enqueue; the gate supplies the frame-tagged set). The
frame tag is what keeps it out of production-frame rate estimation (enforced by the
assert_production_frame guard on the trend side). A passing trace is not enqueued.
Source code in src/calibrated_agent_evaluation/run/gate.py
gate_mode_counts ¶
Gate per-mode counts on each mode's pass-rate 95% lower bound.
The gating step of run_ci_gate, for a caller that has already reduced its verdicts to
one observation per trace and mode (for example human labels, where one mode-free PASS is a
pass for every mode). Admits nothing: judge and scorer admission is run_ci_gate's.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
counts
|
Mapping[ModeKey, ModeCounts]
|
Each mode's verdict total and FAIL count. |
required |
threshold
|
float
|
The default bar every mode's lower bound must clear. |
DEFAULT_PASSRATE_THRESHOLD
|
thresholds
|
Mapping[ModeKey, float] | None
|
Per-mode overrides of |
None
|
decisions
|
Sequence[RunDecision]
|
Non-gating decisions to attach (recorded, never flip |
()
|
Returns:
| Type | Description |
|---|---|
RunGateDecision
|
Whether every mode passed, which failed, and each mode's counts and bound. |
Raises:
| Type | Description |
|---|---|
RunGateError
|
|
Source code in src/calibrated_agent_evaluation/run/gate.py
run_ci_gate ¶
Run the CI gate and return a RunGateDecision.
Refuses an empty input rather than passing vacuously. Admits inputs, aggregates, and gates each
mode on its pass-rate 95% lower bound. passed is True iff every gated mode clears its
threshold (an intersection-union test, so no correction across modes); failing_modes names
those that do not and n_modes counts the modes gated. Attached decisions (e.g. a
staleness warning) are recorded but never flip passed.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
request
|
CiGateRequest
|
The verdicts, thresholds and options for the gate. |
required |
Returns:
| Type | Description |
|---|---|
RunGateDecision
|
Whether every mode passed, which failed, and any attached decisions. |
Raises:
| Type | Description |
|---|---|
RunGateError
|
The input is empty, or a scorer is not allowed to gate. |
UncalibratedScoreError
|
A judge verdict comes from a judge with no certifying record. |
Source code in src/calibrated_agent_evaluation/run/gate.py
apply_multiplicity ¶
Correct pvals for multiplicity and package the method + m + reject vector.
family names the correction site ("drift_flags", "improve_guard", ...). method is
"holm" (a guardrail family where one rejection blocks the change) or "fdr_by"
(exploratory dependence-aware FDR); fdr_bh and any other method are refused by the
underlying primitive (StatisticsInputError), because slices are built from overlapping
traces and BH assumes an independence they do not have. m == len(pvals) is recorded so
the run is auditable.
Source code in src/calibrated_agent_evaluation/run/multi.py
check_staleness ¶
Return a warning-tier staleness RunDecision, or None.
Warns when EITHER trigger fires:
- version drift -- the record no longer certifies
judge(its(model, prompt_version)changed, or it is not CERTIFIED), read viacertifies; or - input drift --
input_driftis supplied and any of its metrics crossed its threshold.
Returns None when the version matches AND drift is sub-threshold (or absent). The decision
is always warning tier -- staleness never hard-refuses (it is passive; a caller must not let
flip a gate). Never raises.
Source code in src/calibrated_agent_evaluation/run/staleness.py
assert_production_rate_frame ¶
Refuse a discovery-frame row from entering a headline (production) rate.
Reuses assert_production_frame: a CI-augmented
(discovery) trace is non-random augmentation, so it grows the discovery set but must never be
counted toward a production-representative headline rate (raises FrameContaminationError).
The run stage does not re-implement the guard -- it wires its rate path through the existing
one.
Source code in src/calibrated_agent_evaluation/run/trend.py
trend_report ¶
trend_report(
counts,
calib_by_mode,
rate_samples,
*,
seed=0,
n_resamples=DEFAULT_N_RESAMPLES,
rate_trace_ids,
rate_cluster_ids=None,
)
One TrendPoint per mode: the corrected rate + CI, keyed per failure mode.
counts is the per-mode aggregation (keyed by
ModeKey); calib_by_mode gives each mode its
own CalibrationRecord; rate_samples gives each mode the judge verdicts (1=FAIL, 0=PASS) over
its SEPARATE production-representative sample
that corrected_rate corrects. A mode absent from calib_by_mode or rate_samples is
skipped (it cannot be corrected without its own judge + rate sample) rather than mis-reported.
rate_trace_ids gives, per mode, the trace id of each rate-sample verdict, which
corrected_rate checks against the calibration gold; a mode without them is refused there.
rate_cluster_ids gives, per mode, the trace id of each rate-sample verdict when a trace
contributes several, so that mode's interval accounts for the clustering.
A mode absent from it is taken as independent verdicts.
A refused correction (J < 0.5, non-identifiable, a rate sample below the minimum, or a record
whose gold is below the coverage floor) surfaces as an Interval with point=None +
refusal_reason; the mode is still emitted, so the gate and the other modes still report.
The report contains no single blended aggregate accuracy scalar:
the only rates are the per-mode corrected-rate intervals.
Source code in src/calibrated_agent_evaluation/run/trend.py
38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 | |