calibrated_agent_evaluation.synth¶
calibrated_agent_evaluation.synth ¶
Input synthesis & generation engine: the Generate stage.
Turns a taxonomy frame into natural-language inputs and (optionally) routes them through
the real agent to produce frame-tagged, taxonomy-attributable Traces. Two-step generation
(tuple → NL) with dedup/diversity guards; inputs-only (never fabricates an output or a label).
The vendor SDK (strands_evals) is confined to borrow
and never imported at package-import time (the [synth] extra is opt-in).
No synth artifact is persisted through the dump/load registry: the engine emits in-memory
SynthInput objects (consumed immediately) and Trace objects (the schema layer already
owns Trace codecs). So there is no new codec to register here.
DEFAULT_RISK_LEXICON
module-attribute
¶
DEFAULT_RISK_LEXICON = frozenset(
{
"legal",
"medical",
"health",
"financial",
"finance",
"high-stakes",
"high_stakes",
"low-resource",
"low_resource",
"underrepresented",
"safety",
"clinical",
}
)
DiversityFloorWarning ¶
Bases: UserWarning
Non-fatal per-cell diversity short-fall warning.
Emitted when a cell cannot meet its distinct-input floor after the configured regeneration budget; surfaced explicitly rather than silently under-delivering (conventions: "no silent truncation").
RepresentativenessWarning ¶
Bases: UserWarning
Non-fatal: the guards removed every input of a production-representative cell.
Dedup or the diversity floor left a cell of a production-representative frame with no input,
so no weighting can restore the declared distribution. The inputs are retagged discovery
(never read as headline rates) and this warning names the emptied cells.
RiskZoneWarning ¶
Bases: UserWarning
Non-fatal high-stakes-domain warning.
Emitted (warn-and-allow) when a taxonomy value names a risk-zone domain. Synthetic inputs are allowed everywhere; the "block synthetic labels" half is structural; the engine has no label-writing path.
SynthError ¶
Bases: CalibratedAgentEvaluationDataError
A hard synthesis refusal.
Raised when: outputs are requested without a real executor (outputs are
never fabricated); the executor run fails; a borrowed strands_evals call fails;
multi-turn is requested without the [synth] extra; or a rendered input
is empty. The message explains the cause; the chained __cause__ preserves
the original traceback.
Embedder ¶
Renderer ¶
Bases: Protocol
Injectable tuple-to-NL renderer: the render step (caller-provided; no vendor type).
Mirrors the Embedder / Prefill injection pattern: the engine BUILDS the
two-step pipeline + guards; the caller supplies the concrete NL renderer (a template renderer,
an LLM renderer, ...). The engine ships only deterministic test fakes.
SynthInput
dataclass
¶
One synthesized natural-language input for a taxonomy cell.
text is the rendered NL input (non-empty). cell/frame/weight are
propagated unchanged from the originating FrameRow so every synthesized artifact is
frame-tagged and taxonomy-attributable. turns is populated only for multi-turn
generation; None for single-shot.
TaskBuilder ¶
Bases: Protocol
Injectable task-assembly seam (caller-provided; no vendor type in synth/).
Turns rendered inputs into a framework task the ExecutorBackend accepts. The return type
is Any because the task type is the executor's vendor concern (an Inspect Task, a
Strands StrandsTask), NOT synth's; keeping synth/ framework-agnostic and vendor-free.
The caller supplies the framework-specific builder.
Optional extension: build_all(inputs) -> list[Any]. generate_and_run calls it when
present (via getattr, which is why it is documented here rather than declared above;
declaring it would make every existing builder non-conforming). It returns one task per executor
call, so the batching decision belongs to the builder: an Inspect Task already holds one
Sample per input and returns a single task, while a StrandsTask carries exactly one
input and returns N. A builder whose framework is one-input-per-task and which omits it would
evaluate only the first input (a defect that once shipped), so the count guard in
_verify_frame_propagation refuses that outcome whatever the builder does.
generate_and_run ¶
generate_and_run(
frame,
renderer,
executor,
task_builder,
*,
multi_turn=False,
turn_task=None,
actor_profile_for=None,
simulator_cls=None,
max_turns=6,
conversation_hooks=None,
**generate_kwargs,
)
Render inputs and route them through the real agent, returning frame-tagged Traces.
Single-shot: builds a task via task_builder and runs it through executor; the returned
traces are validated by the executor and their frame/taxonomy_cell are verified to match
the originating rows; a mismatch is refused (never silently accepted). The output is
always the executor's, never the renderer's.
Multi-turn (multi_turn=True): each input drives a simulated conversation. The borrowed
ActorSimulator writes the user turns; every agent turn is a fresh executor run over the
conversation so far, built by turn_task(input, history) (the task type is the executor's
concern, so the builder is injected the way task_builder is). Each input is stored as one
conversation trace that keeps every turn's spans, and its frame and taxonomy cell are checked
like any other. conversation_hooks plugs in an executor that holds a session per
conversation (its turn task and end hook) and another simulated user.
Raises SynthError if executor is None (outputs are never fabricated),
if the executor raises, or on any borrowed-simulator failure.
Source code in src/calibrated_agent_evaluation/synth/generate.py
dedup ¶
Remove near-duplicate inputs, keeping one representative per group.
Two inputs are near-duplicates when their embedding cosine distance is < threshold. The
first occurrence in input order is the kept representative. threshold <= 0 keeps every
input (no pair is strictly below a non-positive threshold), so dedup is opt-out by config.
Source code in src/calibrated_agent_evaluation/synth/guards.py
enforce_diversity_floor ¶
enforce_diversity_floor(
inputs,
embedder,
renderer,
*,
floor,
threshold=DEFAULT_DEDUP_THRESHOLD,
max_regen=5,
)
Guarantee floor mutually-distinct inputs per taxonomy cell.
For each cell, keeps its deduped inputs and, while it holds fewer than floor distinct
inputs, re-renders that cell up to max_regen times to add new distinct inputs. If the
floor still cannot be met, emits a DiversityFloorWarning naming the cell and the
achieved count (never silently under-delivers). floor <= 1 is a no-op guard.
Source code in src/calibrated_agent_evaluation/synth/guards.py
scan_risk_zones ¶
Scan a frame's taxonomy cells for risk-zone domains.
Returns a mapping matched-domain -> list of cells that named it. If any match is found,
emits a single RiskZoneWarning naming the matched domain(s). An empty lexicon
disables scanning entirely (the caller opted out). Never blocks: inputs are
warn-and-allow; the warning is the whole behavior.