calibrated_agent_evaluation.sampling¶
calibrated_agent_evaluation.sampling ¶
Frame-assembly engine.
Turns a taxonomy grid into two dataset frames: - Discovery frame -- pairwise covering array (every 2-way interaction at least 1x). - Production-representative frame -- proportional stratified sampling.
Public API re-exports from submodules.
DeffWarning ¶
Bases: UserWarning
Non-fatal design-effect warning when Deff exceeds a threshold.
FrameContaminationError ¶
Bases: CalibratedAgentEvaluationDataError
Attempt to use a discovery frame where production-representative is required.
SamplingConfigError ¶
Bases: CalibratedAgentEvaluationDataError
Invalid grid or distribution input.
FrameResult
dataclass
¶
FrameRow
dataclass
¶
TaxonomyGrid
dataclass
¶
Input grid: dimension name → list of possible values.
TrafficDistribution
dataclass
¶
Declared stratum weights for production-representative sampling.
discovery_frame ¶
Produce a covering-array discovery frame from a taxonomy grid.
Uses allpairspy's AllPairs as the base generator and supplements with
additional rows to guarantee full t-way interaction coverage.
Source code in src/calibrated_agent_evaluation/sampling/discovery.py
production_frame ¶
production_frame(
grid,
distribution,
n,
*,
oversample=None,
deff_threshold=DEFAULT_DEFF_THRESHOLD,
)
Produce a proportional stratified sample.
The strata are the values of one grid dimension. Rows are allocated in proportion to the declared distribution, at least one per stratum, and every row carries the design weight of its stratum's realised allocation, so the weighted shares match the declared distribution even where rounding, the 1-row minimum or oversample moved the row counts. When any weight differs from 1 the Kish design effect is reported, with its warning threshold forwarded via deff_threshold.
Raises:
| Type | Description |
|---|---|
SamplingConfigError
|
The grid or distribution is invalid, a stratum key names no value or
a value in two dimensions, the keys come from more than one dimension, or |
Source code in src/calibrated_agent_evaluation/sampling/production.py
kish_deff ¶
Compute Kish design effect and effective sample size.
Deff = 1 + CV²(weights) = (sum(w²) * n) / (sum(w))² n_eff = n / Deff
Emits DeffWarning when deff exceeds threshold.
Source code in src/calibrated_agent_evaluation/sampling/stats.py
assert_production_frame ¶
Raise if any row has frame != 'production-representative'.
Source code in src/calibrated_agent_evaluation/sampling/validate.py
validate_distribution ¶
Raise SamplingConfigError unless weights are finite, non-negative and sum to 1 ±0.01.
Source code in src/calibrated_agent_evaluation/sampling/validate.py
validate_grid ¶
Raise on < 2 dims or any dim with < 2 values.