Skip to content

calibrated_agent_evaluation.synth

calibrated_agent_evaluation.synth

Input synthesis & generation engine: the Generate stage.

Turns a taxonomy frame into natural-language inputs and (optionally) routes them through the real agent to produce frame-tagged, taxonomy-attributable Traces. Two-step generation (tuple → NL) with dedup/diversity guards; inputs-only (never fabricates an output or a label). The vendor SDK (strands_evals) is confined to borrow and never imported at package-import time (the [synth] extra is opt-in).

No synth artifact is persisted through the dump/load registry: the engine emits in-memory SynthInput objects (consumed immediately) and Trace objects (the schema layer already owns Trace codecs). So there is no new codec to register here.

DEFAULT_DEDUP_THRESHOLD module-attribute

DEFAULT_DEDUP_THRESHOLD = 0.05

DEFAULT_RISK_LEXICON module-attribute

DEFAULT_RISK_LEXICON = frozenset(
    {
        "legal",
        "medical",
        "health",
        "financial",
        "finance",
        "high-stakes",
        "high_stakes",
        "low-resource",
        "low_resource",
        "underrepresented",
        "safety",
        "clinical",
    }
)

DiversityFloorWarning

Bases: UserWarning

Non-fatal per-cell diversity short-fall warning.

Emitted when a cell cannot meet its distinct-input floor after the configured regeneration budget; surfaced explicitly rather than silently under-delivering (conventions: "no silent truncation").

RepresentativenessWarning

Bases: UserWarning

Non-fatal: the guards removed every input of a production-representative cell.

Dedup or the diversity floor left a cell of a production-representative frame with no input, so no weighting can restore the declared distribution. The inputs are retagged discovery (never read as headline rates) and this warning names the emptied cells.

RiskZoneWarning

Bases: UserWarning

Non-fatal high-stakes-domain warning.

Emitted (warn-and-allow) when a taxonomy value names a risk-zone domain. Synthetic inputs are allowed everywhere; the "block synthetic labels" half is structural; the engine has no label-writing path.

SynthError

Bases: CalibratedAgentEvaluationDataError

A hard synthesis refusal.

Raised when: outputs are requested without a real executor (outputs are never fabricated); the executor run fails; a borrowed strands_evals call fails; multi-turn is requested without the [synth] extra; or a rendered input is empty. The message explains the cause; the chained __cause__ preserves the original traceback.

Embedder

Bases: Protocol

Injectable embedding interface.

embed

embed(texts)

Convert texts to embedding vectors.

Source code in src/calibrated_agent_evaluation/cluster/embed.py
def embed(self, texts: list[str]) -> list[list[float]]:
    """Convert texts to embedding vectors."""
    ...

Renderer

Bases: Protocol

Injectable tuple-to-NL renderer: the render step (caller-provided; no vendor type).

Mirrors the Embedder / Prefill injection pattern: the engine BUILDS the two-step pipeline + guards; the caller supplies the concrete NL renderer (a template renderer, an LLM renderer, ...). The engine ships only deterministic test fakes.

render

render(cell)

Render a single taxonomy cell into a natural-language input.

Source code in src/calibrated_agent_evaluation/synth/types.py
def render(self, cell: dict[str, str]) -> str:
    """Render a single taxonomy cell into a natural-language input."""
    ...

SynthInput dataclass

SynthInput(text, cell, frame, weight=1.0, turns=None)

One synthesized natural-language input for a taxonomy cell.

text is the rendered NL input (non-empty). cell/frame/weight are propagated unchanged from the originating FrameRow so every synthesized artifact is frame-tagged and taxonomy-attributable. turns is populated only for multi-turn generation; None for single-shot.

text instance-attribute

text

cell instance-attribute

cell

frame instance-attribute

frame

weight class-attribute instance-attribute

weight = 1.0

turns class-attribute instance-attribute

turns = None

TaskBuilder

Bases: Protocol

Injectable task-assembly seam (caller-provided; no vendor type in synth/).

Turns rendered inputs into a framework task the ExecutorBackend accepts. The return type is Any because the task type is the executor's vendor concern (an Inspect Task, a Strands StrandsTask), NOT synth's; keeping synth/ framework-agnostic and vendor-free. The caller supplies the framework-specific builder.

Optional extension: build_all(inputs) -> list[Any]. generate_and_run calls it when present (via getattr, which is why it is documented here rather than declared above; declaring it would make every existing builder non-conforming). It returns one task per executor call, so the batching decision belongs to the builder: an Inspect Task already holds one Sample per input and returns a single task, while a StrandsTask carries exactly one input and returns N. A builder whose framework is one-input-per-task and which omits it would evaluate only the first input (a defect that once shipped), so the count guard in _verify_frame_propagation refuses that outcome whatever the builder does.

build

build(inputs)

Assemble a framework task from the synthesized inputs.

Source code in src/calibrated_agent_evaluation/synth/types.py
def build(self, inputs: list[SynthInput]) -> Any:
    """Assemble a framework task from the synthesized inputs."""
    ...

generate_and_run

generate_and_run(
    frame,
    renderer,
    executor,
    task_builder,
    *,
    multi_turn=False,
    turn_task=None,
    actor_profile_for=None,
    simulator_cls=None,
    max_turns=6,
    conversation_hooks=None,
    **generate_kwargs,
)

Render inputs and route them through the real agent, returning frame-tagged Traces.

Single-shot: builds a task via task_builder and runs it through executor; the returned traces are validated by the executor and their frame/taxonomy_cell are verified to match the originating rows; a mismatch is refused (never silently accepted). The output is always the executor's, never the renderer's.

Multi-turn (multi_turn=True): each input drives a simulated conversation. The borrowed ActorSimulator writes the user turns; every agent turn is a fresh executor run over the conversation so far, built by turn_task(input, history) (the task type is the executor's concern, so the builder is injected the way task_builder is). Each input is stored as one conversation trace that keeps every turn's spans, and its frame and taxonomy cell are checked like any other. conversation_hooks plugs in an executor that holds a session per conversation (its turn task and end hook) and another simulated user.

Raises SynthError if executor is None (outputs are never fabricated), if the executor raises, or on any borrowed-simulator failure.

Source code in src/calibrated_agent_evaluation/synth/generate.py
def generate_and_run(
    frame: FrameResult,
    renderer: Renderer,
    executor: _Executor | None,
    task_builder: TaskBuilder | None,
    *,
    multi_turn: bool = False,
    turn_task: Callable[[SynthInput, list[dict[str, Any]]], Any] | None = None,
    actor_profile_for: Callable[[SynthInput], Any] | None = None,
    simulator_cls: Callable[..., Any] | None = None,
    max_turns: int = 6,
    conversation_hooks: ConversationHooks | None = None,
    **generate_kwargs: Any,
) -> list[Trace]:
    """Render inputs and route them through the real agent, returning frame-tagged Traces.

    Single-shot: builds a task via ``task_builder`` and runs it through ``executor``; the returned
    traces are validated by the executor and their ``frame``/``taxonomy_cell`` are verified to match
    the originating rows; a mismatch is refused (never silently accepted). The output is
    always the executor's, never the renderer's.

    Multi-turn (``multi_turn=True``): each input drives a simulated conversation. The borrowed
    ``ActorSimulator`` writes the user turns; every agent turn is a fresh executor run over the
    conversation so far, built by ``turn_task(input, history)`` (the task type is the executor's
    concern, so the builder is injected the way ``task_builder`` is). Each input is stored as one
    conversation trace that keeps every turn's spans, and its frame and taxonomy cell are checked
    like any other. ``conversation_hooks`` plugs in an executor that holds a session per
    conversation (its turn task and end hook) and another simulated user.

    Raises :class:`SynthError` if ``executor`` is ``None`` (outputs are never fabricated),
    if the executor raises, or on any borrowed-simulator failure.
    """
    if executor is None:
        raise SynthError(
            "agent outputs require a real ExecutorBackend; Calibrated Agent Evaluation never "
            "fabricates outputs. Pass an executor, or use generate() for the inputs-only tier."
        )

    inputs = generate(frame, renderer, **generate_kwargs)

    if multi_turn:
        return _run_multi_turn(
            inputs,
            executor,
            turn_task=turn_task,
            actor_profile_for=actor_profile_for,
            simulator_cls=simulator_cls,
            max_turns=max_turns,
            hooks=conversation_hooks or ConversationHooks(),
        )

    if task_builder is None:
        raise SynthError("single-shot route-through requires a task_builder")

    # `build_all` lets the builder say how many executor calls its framework needs: one batched
    # Inspect Task holding N Samples, or N StrandsTasks each holding one input. Before this,
    # `generate_and_run` called `build(inputs)` once and ran the single task it got back; so a
    # one-input-per-task builder evaluated `inputs[0]` and silently discarded the rest, while the
    # frame/cell checks passed because the one trace it did return was individually valid.
    build_all = cast(
        "Callable[[list[SynthInput]], list[Any]] | None",
        getattr(task_builder, "build_all", None),
    )
    tasks: list[Any] = list(build_all(inputs)) if build_all else [task_builder.build(inputs)]

    traces: list[Trace] = []
    for task in tasks:
        try:
            traces.extend(executor.run(task))
        except Exception as exc:
            raise SynthError(f"agent run failed: {type(exc).__name__}: {exc}") from exc

    _verify_frame_propagation(traces, inputs)
    return traces

dedup

dedup(
    inputs, embedder, *, threshold=DEFAULT_DEDUP_THRESHOLD
)

Remove near-duplicate inputs, keeping one representative per group.

Two inputs are near-duplicates when their embedding cosine distance is < threshold. The first occurrence in input order is the kept representative. threshold <= 0 keeps every input (no pair is strictly below a non-positive threshold), so dedup is opt-out by config.

Source code in src/calibrated_agent_evaluation/synth/guards.py
def dedup(
    inputs: list[SynthInput],
    embedder: Embedder,
    *,
    threshold: float = DEFAULT_DEDUP_THRESHOLD,
) -> list[SynthInput]:
    """Remove near-duplicate inputs, keeping one representative per group.

    Two inputs are near-duplicates when their embedding cosine distance is ``< threshold``. The
    first occurrence in input order is the kept representative. ``threshold <= 0`` keeps every
    input (no pair is strictly below a non-positive threshold), so dedup is opt-out by config.
    """
    if not inputs:
        return []
    vectors = embedder.embed([i.text for i in inputs])
    kept: list[SynthInput] = []
    kept_vectors: list[list[float]] = []
    for item, vec in zip(inputs, vectors, strict=True):
        if any(_cosine_distance(vec, kv) < threshold for kv in kept_vectors):
            continue
        kept.append(item)
        kept_vectors.append(vec)
    return kept

enforce_diversity_floor

enforce_diversity_floor(
    inputs,
    embedder,
    renderer,
    *,
    floor,
    threshold=DEFAULT_DEDUP_THRESHOLD,
    max_regen=5,
)

Guarantee floor mutually-distinct inputs per taxonomy cell.

For each cell, keeps its deduped inputs and, while it holds fewer than floor distinct inputs, re-renders that cell up to max_regen times to add new distinct inputs. If the floor still cannot be met, emits a DiversityFloorWarning naming the cell and the achieved count (never silently under-delivers). floor <= 1 is a no-op guard.

Source code in src/calibrated_agent_evaluation/synth/guards.py
def enforce_diversity_floor(
    inputs: list[SynthInput],
    embedder: Embedder,
    renderer: Renderer,
    *,
    floor: int,
    threshold: float = DEFAULT_DEDUP_THRESHOLD,
    max_regen: int = 5,
) -> list[SynthInput]:
    """Guarantee ``floor`` mutually-distinct inputs per taxonomy cell.

    For each cell, keeps its deduped inputs and, while it holds fewer than ``floor`` distinct
    inputs, re-renders that cell up to ``max_regen`` times to add new distinct inputs. If the
    floor still cannot be met, emits a :class:`DiversityFloorWarning` naming the cell and the
    achieved count (never silently under-delivers). ``floor <= 1`` is a no-op guard.
    """
    if floor <= 1:
        return inputs

    by_cell: dict[tuple[tuple[str, str], ...], list[SynthInput]] = {}
    order: list[tuple[tuple[str, str], ...]] = []
    for item in inputs:
        key = tuple(sorted(item.cell.items()))
        if key not in by_cell:
            by_cell[key] = []
            order.append(key)
        by_cell[key].append(item)

    result: list[SynthInput] = []
    for key in order:
        cell_inputs = dedup(by_cell[key], embedder, threshold=threshold)
        attempts = 0
        while len(cell_inputs) < floor and attempts < max_regen:
            attempts += 1
            cell = dict(key)
            candidate_text = renderer.render(cell)
            template = by_cell[key][0]
            candidate = SynthInput(
                text=candidate_text,
                cell=template.cell,
                frame=template.frame,
                weight=template.weight,
            )
            cell_inputs = dedup([*cell_inputs, candidate], embedder, threshold=threshold)
        if len(cell_inputs) < floor:
            warnings.warn(
                f"cell {dict(key)} yielded {len(cell_inputs)} distinct input(s); "
                f"diversity floor is {floor} (exhausted {max_regen} regeneration attempts)",
                DiversityFloorWarning,
                stacklevel=2,
            )
        result.extend(cell_inputs)
    return result

scan_risk_zones

scan_risk_zones(frame, *, lexicon=DEFAULT_RISK_LEXICON)

Scan a frame's taxonomy cells for risk-zone domains.

Returns a mapping matched-domain -> list of cells that named it. If any match is found, emits a single RiskZoneWarning naming the matched domain(s). An empty lexicon disables scanning entirely (the caller opted out). Never blocks: inputs are warn-and-allow; the warning is the whole behavior.

Source code in src/calibrated_agent_evaluation/synth/risk.py
def scan_risk_zones(
    frame: FrameResult,
    *,
    lexicon: frozenset[str] = DEFAULT_RISK_LEXICON,
) -> dict[str, list[dict[str, str]]]:
    """Scan a frame's taxonomy cells for risk-zone domains.

    Returns a mapping ``matched-domain -> list of cells that named it``. If any match is found,
    emits a single :class:`RiskZoneWarning` naming the matched domain(s). An empty ``lexicon``
    disables scanning entirely (the caller opted out). Never blocks: inputs are
    warn-and-allow; the warning is the whole behavior.
    """
    matches: dict[str, list[dict[str, str]]] = {}
    if not lexicon:
        return matches

    for row in frame.rows:
        for value in row.cell.values():
            token = value.casefold()
            for domain in lexicon:
                if domain.casefold() in token:
                    matches.setdefault(domain, [])
                    if row.cell not in matches[domain]:
                        matches[domain].append(row.cell)

    if matches:
        warnings.warn(
            f"synthetic inputs touch risk-zone domain(s) "
            f"{sorted(matches)}; synthetic inputs are allowed, but validate their distribution "
            f"against production before trusting any synthetic-derived rate",
            RiskZoneWarning,
            stacklevel=2,
        )
    return matches