Skip to content

calibrated_agent_evaluation.execute

calibrated_agent_evaluation.execute

Execution plane: run agents programmatically and get normalized Traces.

The public surface: ExecutorBackend (the protocol), InspectExecutor (Inspect-backed) and StrandsExecutor/StrandsTask (Strands-backed), ExecutionError (the typed error family), and the Cassette record/replay layer. The vendor SDKs are imported lazily so import calibrated_agent_evaluation.execute succeeds without the [inspect]/[strands] extras installed.

ExecutorBackend

Bases: Protocol

Run agents and return normalized, validated Traces.

The methodology plane programs against this protocol; it never sees Inspect or boto3 types.

run

run(task, *, log_dir=None)

Execute an agent task and return normalized Traces.

Raises ExecutionError on any failure (eval error, cassette miss, unreachable model, invalid trace from normalization). Never returns an unvalidated Trace.

Source code in src/calibrated_agent_evaluation/execute/base.py
def run(
    self,
    task: Any,
    *,
    log_dir: Path | None = None,
) -> list[Trace]:
    """Execute an agent task and return normalized Traces.

    Raises ``ExecutionError`` on any failure (eval error, cassette miss, unreachable model,
    invalid trace from normalization). Never returns an unvalidated Trace.
    """
    ...

Cassette

Cassette(path, mode=Mode.AUTO)

A JSON-file-backed record/replay store for one logical capture run.

Thread-safe: the in-memory index and the append are guarded by a lock. Safe across processes too: each append takes an advisory lock on a sidecar .<name>.lock file, merges in what other recorders wrote since, and rewrites the file atomically through a uniquely named temporary, so a crashed capture leaves a valid (if partial) cassette and two recorders on one file keep both their entries.

Source code in src/calibrated_agent_evaluation/execute/cassette.py
def __init__(self, path: str | Path, mode: Mode = Mode.AUTO) -> None:
    self._path = Path(path)
    self._mode = mode
    self._lock = Lock()
    self._entries: dict[str, _Entry] = {}
    self._load()

mode property

mode

Whether the cassette replays, records, or does both.

path property

path

The JSON file the cassette reads and writes.

lookup

lookup(fp, *, model=None, require_model=False)

Return a copy of the recorded response for fingerprint fp, or None if absent.

model is this run's model_stamp. An entry recorded under a different stamp raises CassetteModelMismatch. An entry recorded with no stamp is returned, unless require_model is set: then it raises CassetteModelMismatch too, since nothing shows it came from model.

Source code in src/calibrated_agent_evaluation/execute/cassette.py
def lookup(
    self, fp: str, *, model: dict[str, Any] | None = None, require_model: bool = False
) -> dict[str, Any] | None:
    """Return a copy of the recorded response for fingerprint ``fp``, or ``None`` if absent.

    ``model`` is this run's :func:`model_stamp`. An entry recorded under a different stamp
    raises :class:`CassetteModelMismatch`. An entry recorded with no stamp is returned, unless
    ``require_model`` is set: then it raises :class:`CassetteModelMismatch` too, since nothing
    shows it came from ``model``.
    """
    with self._lock:
        entry = self._entries.get(fp)
        if entry is None:
            return None
        if model is not None and entry.model is None and require_model:
            raise CassetteModelMismatch(
                f"cassette {self._path.name}: the recorded response for fingerprint {fp} has "
                f"no record of the model it came from, so it can't be credited to "
                f"{_describe(model)}. Re-record the cassette with this model"
            )
        if model is not None and entry.model is not None and entry.model != model:
            raise CassetteModelMismatch(
                f"cassette {self._path.name}: the recorded response for fingerprint {fp} came "
                f"from {_describe(entry.model)}, but this run uses {_describe(model)}; "
                "replaying it would credit one model's answer to another. Re-record the "
                "cassette with this model, or run with the model it was recorded with"
            )
        return json.loads(json.dumps(entry.response))

record

record(fp, request, response, *, model=None)

Record response for fp under model and write the cassette file.

Raises:

Type Description
CassetteError

The file could not be written. The cassette is left as it was, in memory and on file: the response is not recorded, and an entry it was replacing stays.

Source code in src/calibrated_agent_evaluation/execute/cassette.py
def record(
    self,
    fp: str,
    request: dict[str, Any],
    response: dict[str, Any],
    *,
    model: dict[str, Any] | None = None,
) -> None:
    """Record ``response`` for ``fp`` under ``model`` and write the cassette file.

    Raises:
        CassetteError: The file could not be written. The cassette is left as it was, in
            memory and on file: the response is not recorded, and an entry it was replacing
            stays.
    """
    with self._lock:
        self._record_and_flush(_Entry(fp, request, response, model))

play

play(
    request, live_call, *, model=None, require_model=False
)

Return the response for request, honoring the cassette mode.

live_call is a zero-arg callable that performs the real provider request and returns its response dict; it is invoked only in RECORD mode or on an AUTO miss. In REPLAY mode a miss raises CassetteMiss. model is the run's model_stamp, checked against a hit and recorded with a new entry; require_model refuses a hit with no stamp, as lookup does.

Source code in src/calibrated_agent_evaluation/execute/cassette.py
def play(
    self,
    request: dict[str, Any],
    live_call: Any,
    *,
    model: dict[str, Any] | None = None,
    require_model: bool = False,
) -> dict[str, Any]:
    """Return the response for ``request``, honoring the cassette mode.

    ``live_call`` is a zero-arg callable that performs the real provider request and returns its
    response dict; it is invoked only in RECORD mode or on an AUTO miss. In REPLAY mode a miss
    raises :class:`CassetteMiss`. ``model`` is the run's :func:`model_stamp`, checked against
    a hit and recorded with a new entry; ``require_model`` refuses a hit with no stamp, as
    :meth:`lookup` does.
    """
    fp = fingerprint(request)
    if self._mode is Mode.REPLAY:
        hit = self.lookup(fp, model=model, require_model=require_model)
        if hit is None:
            raise CassetteMiss(
                f"cassette {self._path.name}: no recorded response for fingerprint {fp} "
                f"(REPLAY mode does not fall through to a live call)"
            )
        logger.debug("cassette replay hit fp=%s", fp)
        return hit
    if self._mode is Mode.AUTO:
        hit = self.lookup(fp, model=model, require_model=require_model)
        if hit is not None:
            logger.debug("cassette auto hit fp=%s", fp)
            return hit
    logger.info("cassette recording live call fp=%s -> %s", fp, self._path.name)
    response = live_call()
    self.record(fp, request, response, model=model)
    return response

aplay async

aplay(
    request, live_call, *, model=None, require_model=False
)

Async twin of play; live_call is an async (awaitable) callable.

Source code in src/calibrated_agent_evaluation/execute/cassette.py
async def aplay(
    self,
    request: dict[str, Any],
    live_call: Any,
    *,
    model: dict[str, Any] | None = None,
    require_model: bool = False,
) -> dict[str, Any]:
    """Async twin of :meth:`play`; ``live_call`` is an async (awaitable) callable."""
    fp = fingerprint(request)
    if self._mode is Mode.REPLAY:
        hit = self.lookup(fp, model=model, require_model=require_model)
        if hit is None:
            raise CassetteMiss(
                f"cassette {self._path.name}: no recorded response for fingerprint {fp} "
                f"(REPLAY mode does not fall through to a live call)"
            )
        return hit
    if self._mode is Mode.AUTO:
        hit = self.lookup(fp, model=model, require_model=require_model)
        if hit is not None:
            return hit
    logger.info("cassette recording live call fp=%s -> %s", fp, self._path.name)
    response = await live_call()
    self.record(fp, request, response, model=model)
    return response

CassetteError

Bases: RuntimeError

A cassette could not be read or written (wraps the underlying OS/JSON error).

CassetteMiss

Bases: RuntimeError

Raised in REPLAY mode when a request fingerprint has no recorded response.

CassetteModelMismatch

Bases: CassetteError

A cassette entry was recorded with a different model or decoding config than this run's.

Replaying it would attribute one model's answers to another. Entries record the model that produced them; an entry with no record of its model (a cassette recorded before entries carried one) is replayed as before, unless the caller requires the record, which refuses it too.

Mode

Bases: StrEnum

How the cassette behaves for a given run.

RECORD class-attribute instance-attribute

RECORD = 'record'

REPLAY class-attribute instance-attribute

REPLAY = 'replay'

AUTO class-attribute instance-attribute

AUTO = 'auto'

ExecutionError

Bases: Exception

A typed error from the execution plane.

Wraps vendor/SDK exceptions via chaining (raise ExecutionError(...) from exc). The message carries diagnostic detail (file paths, eval status, fingerprint); the chained __cause__ preserves the original traceback for debugging.

JudgeConfigurationError

Bases: ExecutionError

The judge model was called in a way that cannot work, whatever the model does.

A subclass so an existing except ExecutionError boundary still catches it, but named so a caller that distinguishes transient from permanent can. judges/runtime._judge_one retries a bare ExecutionError as transient, so a permanent configuration fault raised as one burned the whole retry budget and then reported itself as "model failed after N attempts" -- sending the reader to look at the model instead of at the call site.

SandboxError

Bases: ExecutionError

Sandboxed execution could not be carried out.

Raised when the process could not start, a remote sandbox was unreachable, or its response was malformed. A subclass so an existing except ExecutionError boundary still catches it. This is an infrastructure failure, distinct from a program that ran and failed its tests (that is a PASS/FAIL verdict, not an error); the execution scorers re-wrap it as ScorerError at their boundary and never return a Label for it.

CassetteJudgeModel

CassetteJudgeModel(
    cassette,
    *,
    supports_schema=True,
    record_model=None,
    temperature=None,
    model_id=None,
)

Offline judge-model backend backed by a Cassette (test double + deterministic CI).

Mirrors InspectJudgeModel's surface so judges/ code is identical in both worlds. supports_response_schema is explicit (defaults to True) so a test can exercise either path; the recorded ModelOutput JSON is replayed by content fingerprint. A REPLAY miss raises CassetteMiss.

A repeat of an identical request (sample above 0) and a set temperature are part of the fingerprint, so a repeat replays its own recorded draw. Neither is in the fingerprint at its default, so cassettes recorded before either existed still replay.

model_id names the model this run judges with (it defaults to record_model). Entries are recorded under it, and replaying an entry recorded with a different model or temperature raises CassetteModelMismatch.

Source code in src/calibrated_agent_evaluation/execute/judge_model.py
def __init__(
    self,
    cassette: Cassette,
    *,
    supports_schema: bool = True,
    record_model: str | None = None,
    temperature: float | None = None,
    model_id: str | None = None,
) -> None:
    self._cassette = cassette
    self._supports_schema = supports_schema
    # Only needed in RECORD/AUTO mode to mint a real ModelOutput; unit tests use REPLAY.
    self._record_model = record_model
    self._temperature = temperature
    config = {} if temperature is None else {"temperature": temperature}
    self._model = model_stamp(model_id or record_model, config)

supports_response_schema

supports_response_schema()

Whether this backend honors a response schema.

Source code in src/calibrated_agent_evaluation/execute/judge_model.py
def supports_response_schema(self) -> bool:
    """Whether this backend honors a response schema."""
    return self._supports_schema

generate_structured

generate_structured(*, system, user, schema, sample=0)

Return the recorded structured reply for this request.

Source code in src/calibrated_agent_evaluation/execute/judge_model.py
def generate_structured(
    self,
    *,
    system: str,
    user: str,
    schema: dict[str, Any],
    sample: int = 0,
) -> StructuredReply:
    """Return the recorded structured reply for this request."""
    request: dict[str, Any] = {
        "system": system,
        "user": user,
        "schema": schema,
        "via": "response_schema" if self._supports_schema else "tool_call",
    }
    if sample:
        request["sample"] = sample
    if self._temperature is not None:
        request["temperature"] = self._temperature
    fp = fingerprint(request)
    # RECORD means re-record: a stored entry is not consulted, or re-recording a cassette would
    # silently keep every old verdict. A model mismatch on lookup propagates as its own type.
    hit = (
        None
        if self._cassette.mode is Mode.RECORD
        else self._cassette.lookup(fp, model=self._model)
    )
    if hit is not None:
        return StructuredReply(fields=hit["fields"], via=hit["via"])
    if self._cassette.mode is Mode.REPLAY:
        raise CassetteMiss(
            f"judge cassette {self._cassette.path.name}: no recorded verdict for fingerprint "
            f"{fp} (REPLAY mode does not fall through to a live call)"
        )
    # RECORD/AUTO miss: delegate to a real Inspect call, capture the normalized reply.
    if self._record_model is None:
        raise ExecutionError(
            "CassetteJudgeModel miss in RECORD/AUTO mode but no record_model was set; "
            "pass record_model= to capture a live verdict"
        )
    reply = InspectJudgeModel(
        self._record_model, temperature=self._temperature
    ).generate_structured(system=system, user=user, schema=schema)
    try:
        self._cassette.record(
            fp, request, {"fields": reply.fields, "via": reply.via}, model=self._model
        )
    except CassetteError as exc:
        # Permanent (an unwritable or corrupt cassette), so not retried as a model failure.
        raise JudgeConfigurationError(f"the live verdict could not be recorded: {exc}") from exc
    return reply

InspectJudgeModel

InspectJudgeModel(
    model, *, timeout_seconds=60.0, temperature=None
)

Real judge-model backend on Inspect model-runners.

Constructed with an Inspect model string (e.g. a "bedrock/…claude-sonnet-4-6" id). A model whose provider/name does not natively support response_schema is served through the forced-tool path; never free text.

temperature is sent only when set. A set temperature the provider did not send (Inspect drops it for Claude 4.7 and later on Bedrock) is refused as a JudgeConfigurationError, so a judge never runs at a temperature other than the one recorded for it.

Source code in src/calibrated_agent_evaluation/execute/judge_model.py
def __init__(
    self,
    model: str,
    *,
    timeout_seconds: float = 60.0,
    temperature: float | None = None,
) -> None:
    self._model_str = model
    self._timeout = timeout_seconds
    self._temperature = temperature

supports_response_schema

supports_response_schema()

True only for Bedrock Claude models (verified _providers/bedrock.py:642-657).

Bedrock honors response_schema for Claude via the Converse output_config.format, and warn-drops it for non-Claude Bedrock; SageMaker has no support. Rather than trust a sent schema was honored, we gate on the known-supporting family and route everything else through the forced-tool path. Conservative by design: a model we can't prove supports it uses the tool path, which is always structured.

Source code in src/calibrated_agent_evaluation/execute/judge_model.py
def supports_response_schema(self) -> bool:
    """True only for Bedrock Claude models (verified ``_providers/bedrock.py:642-657``).

    Bedrock honors ``response_schema`` for Claude via the Converse ``output_config.format``, and
    **warn-drops it for non-Claude Bedrock**; SageMaker has no support. Rather than trust a sent
    schema was honored, we gate on the known-supporting family and route everything else through
    the forced-tool path. Conservative by design: a model we can't prove supports it
    uses the tool path, which is always structured.
    """
    model = self._model_str.lower()
    provider, _, name = model.partition("/")
    return provider == "bedrock" and "claude" in name

generate_text

generate_text(*, system, user)

Return the model's plain-text completion for a (system, user) prompt.

The free-text sibling of generate_structured; used by the BedrockRenderer to render a taxonomy cell into a natural-language input. Wraps the verified Inspect model seam (get_model(model).generate(messages).completion, the exact call this class already makes internally). Raises ExecutionError on any model failure (wrapped from the vendor exception); never returns a fabricated or partial string.

Source code in src/calibrated_agent_evaluation/execute/judge_model.py
def generate_text(self, *, system: str, user: str) -> str:
    """Return the model's plain-text completion for a ``(system, user)`` prompt.

    The free-text sibling of :meth:`generate_structured`; used by the ``BedrockRenderer``
    to render a taxonomy cell into a natural-language input. Wraps the verified Inspect model
    seam (``get_model(model).generate(messages).completion``, the exact call this class already
    makes internally). Raises :class:`ExecutionError` on any model failure (wrapped from the
    vendor exception); never returns a fabricated or partial string.
    """
    try:
        return asyncio.run(self._agenerate_text(system=system, user=user))
    except (CassetteMiss, ExecutionError):
        raise
    except Exception as exc:
        raise ExecutionError(
            f"judge model {self._model_str!r} failed to generate text: {exc}"
        ) from exc

generate_structured

generate_structured(*, system, user, schema, sample=0)

Drive the async Inspect model call under a sync boundary; extract a structured reply.

sample needs nothing here: Inspect does not cache by default, so every call is a draw.

Refuses up front when an event loop is already running. asyncio.run raises RuntimeError there, which the broad except Exception below wrapped as ExecutionError -- and judges/runtime._judge_one treats any ExecutionError as transient and retries it. So calling a judge from inside an event loop burned max_retries attempts and then failed with "model failed after 4 attempts", naming a permanent config error as flakiness. JudgeConfigurationError is outside the retry family, so it surfaces on the first attempt.

Source code in src/calibrated_agent_evaluation/execute/judge_model.py
def generate_structured(
    self,
    *,
    system: str,
    user: str,
    schema: dict[str, Any],
    sample: int = 0,  # noqa: ARG002  # part of the port; see the note below
) -> StructuredReply:
    """Drive the async Inspect model call under a sync boundary; extract a structured reply.

    ``sample`` needs nothing here: Inspect does not cache by default, so every call is a draw.

    Refuses up front when an event loop is already running. `asyncio.run` raises `RuntimeError`
    there, which the broad `except Exception` below wrapped as `ExecutionError` -- and
    `judges/runtime._judge_one` treats any `ExecutionError` as transient and retries it. So
    calling a judge from inside an event loop burned `max_retries` attempts and then failed with
    "model failed after 4 attempts", naming a permanent config error as flakiness.
    `JudgeConfigurationError` is outside the retry family, so it surfaces on the first attempt.
    """
    try:
        asyncio.get_running_loop()
    except RuntimeError:
        pass  # no running loop: the sync boundary is safe to cross
    else:
        raise JudgeConfigurationError(
            f"judge model {self._model_str!r} was called from inside a running event loop; "
            "generate_structured() crosses a sync boundary with asyncio.run(), which cannot "
            "nest. Call it from a synchronous context, or await the async model directly"
        )
    try:
        return asyncio.run(self._agenerate(system=system, user=user, schema=schema))
    except (CassetteMiss, ExecutionError):
        raise
    except Exception as exc:
        raise ExecutionError(
            f"judge model {self._model_str!r} failed to generate a structured verdict: {exc}"
        ) from exc

JudgeModelBackend

Bases: Protocol

Call a judge model and return a forced-structured reply: the judge model seam.

judges/ programs against this; it never imports inspect_ai. Implementations raise ExecutionError on any model failure (the judges/ boundary re-wraps it as JudgeRuntimeError).

supports_response_schema

supports_response_schema()

Whether the selected model honors native response_schema.

Drives the path choice: True → native structured output; False → forced tool call.

Source code in src/calibrated_agent_evaluation/execute/judge_model.py
def supports_response_schema(self) -> bool:
    """Whether the selected model honors native ``response_schema``.

    Drives the path choice: ``True`` → native structured output; ``False`` → forced tool call.
    """
    ...

generate_structured

generate_structured(*, system, user, schema, sample=0)

Return a forced-structured reply for the (system, user) prompt against schema.

schema is a plain JSON-schema dict describing the verdict object (critique-before -verdict is guaranteed by the caller). sample numbers repeats of an identical request (0 is the first): a backend that answers by request content keeps repeats apart, so a repeat is a fresh draw and not the first reply played back. Raises ExecutionError on model failure or when no structured payload can be extracted (enforced with the typed error at the judges/ boundary).

Source code in src/calibrated_agent_evaluation/execute/judge_model.py
def generate_structured(
    self,
    *,
    system: str,
    user: str,
    schema: dict[str, Any],
    sample: int = 0,
) -> StructuredReply:
    """Return a forced-structured reply for the ``(system, user)`` prompt against ``schema``.

    ``schema`` is a plain JSON-schema dict describing the verdict object (critique-before
    -verdict is guaranteed by the caller). ``sample`` numbers repeats of an identical request
    (0 is the first): a backend that answers by request content keeps repeats apart, so a
    repeat is a fresh draw and not the first reply played back. Raises
    :class:`ExecutionError` on model failure or when no structured payload can be extracted
    (enforced with the typed error at the ``judges/`` boundary).
    """
    ...

StructuredReply dataclass

StructuredReply(fields, via)

A model's forced-structured verdict, normalized off any vendor type.

fields is the structured payload (e.g. {"critique": ..., "verdict": "pass"|"fail"}); via records which path produced it; "response_schema" (native structured output) or "tool_call" (the forced-tool fallback); for audit. judges/ validates fields against the verdict schema; this seam does not judge correctness, only that a structured payload exists.

fields instance-attribute

fields

via instance-attribute

via

LocalSubprocessSandbox

LocalSubprocessSandbox(
    *, max_memory_bytes=_DEFAULT_MAX_MEMORY_BYTES
)

The default, dependency-free sandbox: a timed, isolated-mode subprocess with a minimal env.

See the module docstring for the threat model: this is best-effort resource and accident isolation, not a security boundary. max_memory_bytes caps the child's address space where POSIX allows it.

Source code in src/calibrated_agent_evaluation/execute/sandbox.py
def __init__(self, *, max_memory_bytes: int = _DEFAULT_MAX_MEMORY_BYTES) -> None:
    self._max_memory_bytes = max_memory_bytes

run_code

run_code(code, *, language='python', timeout_s, files=None)

Run code as a Python program in a temp cwd seeded with files.

Raises SandboxError for an unsupported language, a files name that resolves outside the sandbox directory (absolute, or escaping through ..), or if the child cannot be started.

Source code in src/calibrated_agent_evaluation/execute/sandbox.py
def run_code(
    self,
    code: str,
    *,
    language: str = "python",
    timeout_s: float,
    files: dict[str, str] | None = None,
) -> SandboxResult:
    """Run ``code`` as a Python program in a temp cwd seeded with ``files``.

    Raises :class:`SandboxError` for an unsupported language, a ``files`` name that resolves
    outside the sandbox directory (absolute, or escaping through ``..``), or if the child
    cannot be started.
    """
    if language != "python":
        raise SandboxError(
            f"the local sandbox runs Python only, got language={language!r}; use the "
            "AgentCore Code Interpreter backend for other languages"
        )
    setter = _posix_limit_setter(timeout_s, self._max_memory_bytes)
    limits_applied = setter is not None
    program = _PREAMBLE + code

    with tempfile.TemporaryDirectory(prefix="calibrated_agent_evaluation_sandbox_") as tmp:
        cwd = Path(tmp).resolve()
        seeded = [
            (_resolve_inside(cwd, name), content) for name, content in (files or {}).items()
        ]
        for dest, content in seeded:
            dest.parent.mkdir(parents=True, exist_ok=True)
            dest.write_text(content, encoding="utf-8")
        main = cwd / "_calibrated_agent_evaluation_main.py"
        main.write_text(program, encoding="utf-8")
        out_path = cwd / "_calibrated_agent_evaluation_stdout"
        err_path = cwd / "_calibrated_agent_evaluation_stderr"
        started = time.monotonic()
        # Stream the child's output straight to files rather than PIPEs: communicate() would
        # buffer an unbounded amount in THIS process's memory, so a candidate stuck in a print
        # loop could exhaust the scorer's RAM long before the timeout fired. The files are read
        # back capped, and RLIMIT_FSIZE bounds what the child can write to them.
        try:
            with out_path.open("wb") as out_f, err_path.open("wb") as err_f:
                proc = subprocess.Popen(  # noqa: S603 - executing benchmark code is the feature
                    [sys.executable, "-I", str(main)],
                    cwd=str(cwd),
                    env=_child_env(cwd),
                    stdout=out_f,
                    stderr=err_f,
                    preexec_fn=setter,
                    # Own session/process group, so a timeout can kill grandchildren too, not
                    # just the direct child (PID == PGID on POSIX).
                    start_new_session=True,
                )
                try:
                    proc.wait(timeout=timeout_s)
                    timed_out = False
                except subprocess.TimeoutExpired:
                    _kill_process_group(proc)
                    proc.wait()
                    timed_out = True
        except OSError as exc:
            raise SandboxError(f"could not start the sandbox subprocess: {exc}") from exc
        return SandboxResult(
            stdout=_read_capped(out_path),
            stderr=_read_capped(err_path),
            exit_code=-1 if timed_out else proc.returncode,
            timed_out=timed_out,
            limits_applied=limits_applied,
            duration_s=time.monotonic() - started,
        )

SandboxBackend

Bases: Protocol

Run a snippet of code and return a normalized SandboxResult.

The methodology plane programs against this; it never imports a vendor SDK type. Implementations raise SandboxError when the execution could not be carried out (as opposed to a program that ran and exited non-zero, which is a normal SandboxResult).

run_code

run_code(code, *, language='python', timeout_s, files=None)

Execute code (seeded with files in its working directory) and return the result.

Raises SandboxError if the execution could not be carried out.

Source code in src/calibrated_agent_evaluation/execute/sandbox.py
def run_code(
    self,
    code: str,
    *,
    language: str = "python",
    timeout_s: float,
    files: dict[str, str] | None = None,
) -> SandboxResult:
    """Execute ``code`` (seeded with ``files`` in its working directory) and return the result.

    Raises :class:`SandboxError` if the execution could not be carried out.
    """
    ...

SandboxResult dataclass

SandboxResult(
    stdout,
    stderr,
    exit_code,
    timed_out,
    limits_applied,
    duration_s,
)

The normalized outcome of one sandboxed execution, free of any vendor type.

exit_code is 0 on clean completion; timed_out is True when the wall-clock limit fired (exit_code is then non-zero and undefined). limits_applied is False when the platform offered no POSIX resource limits, so a run on a limit-less host is auditable rather than silently unbounded.

stdout instance-attribute

stdout

stderr instance-attribute

stderr

exit_code instance-attribute

exit_code

timed_out instance-attribute

timed_out

limits_applied instance-attribute

limits_applied

duration_s instance-attribute

duration_s

BedrockEmbedder

BedrockEmbedder(
    *,
    model_id=_DEFAULT_MODEL_ID,
    region=_DEFAULT_REGION,
    timeout_seconds=_DEFAULT_TIMEOUT_SECONDS,
)

Concrete Embedder on Bedrock Titan v2 via boto3 invoke_model.

Construction is cheap (the boto3 client is built lazily on first embed, so importing this module needs no AWS session and importing calibrated_agent_evaluation.execute needs no boto3).

Source code in src/calibrated_agent_evaluation/execute/bedrock_embedder.py
def __init__(
    self,
    *,
    model_id: str = _DEFAULT_MODEL_ID,
    region: str = _DEFAULT_REGION,
    timeout_seconds: float = _DEFAULT_TIMEOUT_SECONDS,
) -> None:
    self._model_id = model_id
    self._region = region
    self._timeout_seconds = timeout_seconds
    self._client: Any = None

embed

embed(texts)

Return one embedding vector per input text, in order.

Raises ClusteringError if the backend is unreachable, a call fails, or the number of returned vectors does not equal len(texts); never a truncated/misaligned list.

Source code in src/calibrated_agent_evaluation/execute/bedrock_embedder.py
def embed(self, texts: list[str]) -> list[list[float]]:
    """Return one embedding vector per input text, in order.

    Raises :class:`ClusteringError` if the backend is unreachable, a call fails, or the number
    of returned vectors does not equal ``len(texts)``; never a truncated/misaligned list.
    """
    client = self._get_client()
    vectors: list[list[float]] = []
    for text in texts:
        vectors.append(self._embed_one(client, text))
    if len(vectors) != len(texts):
        # Defensive: the loop is 1:1 by construction, but assert the invariant explicitly so a
        # future batch-API change can never silently drop a vector.
        raise ClusteringError(
            f"embedder produced {len(vectors)} vectors for {len(texts)} inputs; "
            "refusing a misaligned result"
        )
    return vectors

BedrockRenderer

BedrockRenderer(model, *, timeout_seconds=60.0)

Concrete Renderer on Bedrock via the Inspect model seam.

model is an Inspect model string (e.g. "bedrock/global.anthropic.claude-sonnet-4-6"). Construction is cheap (no network); the model call happens in render.

Source code in src/calibrated_agent_evaluation/execute/bedrock_renderer.py
def __init__(self, model: str, *, timeout_seconds: float = 60.0) -> None:
    self._model = InspectJudgeModel(model, timeout_seconds=timeout_seconds)

render

render(cell)

Render one taxonomy cell into a non-empty NL input.

Raises SynthError if the model is unreachable/errors (wrapping the vendor exception) or returns empty/whitespace text; never fabricates or returns an empty input.

Source code in src/calibrated_agent_evaluation/execute/bedrock_renderer.py
def render(self, cell: dict[str, str]) -> str:
    """Render one taxonomy cell into a non-empty NL input.

    Raises :class:`SynthError` if the model is unreachable/errors (wrapping the vendor
    exception) or returns empty/whitespace text; never fabricates or returns an empty input.
    """
    prompt = self._prompt_for(cell)
    try:
        text = self._model.generate_text(system=_SYSTEM, user=prompt)
    except Exception as exc:
        # Wrap the vendor/execution exception into the synth error family (no raw escape).
        raise SynthError(
            f"LLM renderer failed for cell {cell}: {type(exc).__name__}: {exc}"
        ) from exc
    if not text or not text.strip():
        raise SynthError(
            f"LLM renderer returned empty text for cell {cell}; refusing to fabricate an input"
        )
    return text.strip()

InspectExecutor

InspectExecutor(
    model="cassette/orderbot",
    *,
    log_dir=None,
    cassette_path=None,
    cassette_mode=None,
    cassette_inner_model=None,
    frame="production-representative",
)

An ExecutorBackend over Inspect AI's eval().

Context manager: __enter__/__exit__ manage the temp log directory (if no explicit log_dir was provided) and ensure resources are cleaned up.

Parameters:

Name Type Description Default
model str

The Inspect model the agent runs on.

'cassette/orderbot'
log_dir Path | None

Where Inspect writes its logs; a temporary directory when omitted.

None
cassette_path str | Path | None

A cassette to record or replay model calls through.

None
cassette_mode Mode | None

Whether the cassette replays, records, or both.

None
cassette_inner_model str | None

The real model a recording cassette calls on a miss.

None
frame Frame

The sampling frame the produced traces are stamped with.

'production-representative'
Source code in src/calibrated_agent_evaluation/execute/inspect_executor.py
def __init__(
    self,
    model: str = "cassette/orderbot",
    *,
    log_dir: Path | None = None,
    cassette_path: str | Path | None = None,
    cassette_mode: Mode | None = None,
    cassette_inner_model: str | None = None,
    frame: Frame = "production-representative",
) -> None:
    self._model = model
    self._explicit_log_dir = log_dir
    self._cassette_path = None if cassette_path is None else Path(cassette_path)
    self._cassette_mode = cassette_mode or mode_from_env(default=Mode.AUTO)
    # The dataset frame the produced traces carry. Defaults to "production-representative"
    # (the InspectEvalLog default), but a caller synthesizing a discovery-frame dataset
    # passes frame="discovery" so the frame tag survives execution; the normalizer reads it
    # from the source, not the .eval log (frame propagation for the synthesis route-through).
    self._frame: Frame = frame
    # The real model a `cassette/...` run records against on a RECORD-mode miss. Explicit
    # (not derived from the cassette model string): `cassette/orderbot` is a logical replay
    # handle, not a Bedrock id, so `.replace("cassette/", "bedrock/")` would produce the
    # invalid `bedrock/orderbot`. Defaults to the provider's DEFAULT_INNER_MODEL when unset.
    self._cassette_inner_model = cassette_inner_model
    self._tmpdir: tempfile.TemporaryDirectory[str] | None = None

log_dir property

log_dir

The directory where .eval logs are persisted.

run

run(task, *, log_dir=None)

Run an Inspect task and return normalized, validated Traces.

Raises ExecutionError on any failure; eval error, cassette miss, unreachable model, or invalid trace from normalization.

Source code in src/calibrated_agent_evaluation/execute/inspect_executor.py
def run(self, task: Any, *, log_dir: Path | None = None) -> list[Trace]:
    """Run an Inspect task and return normalized, validated Traces.

    Raises ``ExecutionError`` on any failure; eval error, cassette miss, unreachable model,
    or invalid trace from normalization.
    """
    eval_fn, _read_eval_log = self._load_inspect()
    effective_log_dir = log_dir or self.log_dir
    effective_log_dir.mkdir(parents=True, exist_ok=True)

    try:
        # Inside the try: building the Cassette loads (and validates) the file, so a corrupt
        # cassette raises CassetteError HERE; it must be wrapped as ExecutionError, never leak
        # to the caller (cassette error types are implementation details).
        self._configure_cassette_provider()
        logs = eval_fn(
            task,
            model=self._model,
            log_dir=str(effective_log_dir),
            score=False,
        )
    except (CassetteMiss, CassetteError) as exc:
        raise ExecutionError(f"cassette error during eval: {exc}") from exc
    except ExecutionError:
        raise
    except Exception as exc:
        raise ExecutionError(f"eval failed: {type(exc).__name__}: {exc}") from exc

    traces: list[Trace] = []
    for eval_log in logs:
        if eval_log.status != "success":
            error_detail = getattr(eval_log, "error", None)
            raise ExecutionError(
                f"eval did not succeed: status={eval_log.status}, error={error_detail}"
            )
        log_path = self._resolve_log_path(eval_log, effective_log_dir)
        traces.extend(self._normalize_log(log_path))

    return traces

CaptureProviderError

Bases: ExecutionError

The process's global tracer provider cannot carry spans to a capture.

Raised when something other than the OpenTelemetry SDK's TracerProvider is already the global provider (a vendor distribution, or the API's no-op provider), so capture has no way to attach a span processor to it.

SpanCapture

SpanCapture(service_name)

Owns a TracerProvider + in-memory exporter; yields a tracer and the captured spans.

Use as a context manager so the provider is flushed and shut down deterministically (I/O resources are context managers). After the with block, finished_spans holds every span the agent emitted, in completion order.

Parameters:

Name Type Description Default
service_name str

The OTel service name the captured spans are recorded under.

required
Source code in src/calibrated_agent_evaluation/execute/strands_capture.py
def __init__(self, service_name: str) -> None:
    self._exporter = InMemorySpanExporter()
    self._provider = TracerProvider(resource=Resource.create({"service.name": service_name}))
    # SimpleSpanProcessor exports on span end; synchronous, deterministic, no batching delay.
    self._provider.add_span_processor(SimpleSpanProcessor(self._exporter))
    self._attached = False
    self._active_token: contextvars.Token[SpanCapture | None] | None = None

provider property

provider

The real provider: hand this to StrandsTelemetry(provider) or set it as global.

finished_spans property

finished_spans

The spans collected so far, in the order they finished.

register_global

register_global()

Receive the spans the agent run inside this capture emits through global tracers.

Framework-instrumented code (Strands, or a module-level trace.get_tracer(...)) emits through the OTel global provider, and OTel lets that be set only once per process. A ProxyTracer handed out before then also binds to the first real provider for good. So capture never replaces the global: it installs one long-lived provider the first time (or joins an SDK provider the application already installed) and hangs a router off it.

The router gives each capture only its own spans. A span belongs to the capture whose context it started in, or to the capture that owns its parent, so the agent's spans on a worker thread still arrive and two captures running at once never see each other's spans. Host telemetry that started outside any capture (a web request, a database call) is not captured. Strands' tracer is pointed at a private provider while any capture is open, so its spans reach the capture without passing through an exporter the global provider carries (a Langfuse client's, for example). Self-built emitters (OrderBot) can use tracer.

Raises CaptureProviderError if the global provider is not an SDK provider.

Source code in src/calibrated_agent_evaluation/execute/strands_capture.py
def register_global(self) -> None:
    """Receive the spans the agent run inside this capture emits through global tracers.

    Framework-instrumented code (Strands, or a module-level ``trace.get_tracer(...)``) emits
    through the OTel global provider, and OTel lets that be set only once per process. A
    ``ProxyTracer`` handed out before then also binds to the first real provider for good. So
    capture never replaces the global: it installs one long-lived provider the first time (or
    joins an SDK provider the application already installed) and hangs a router off it.

    The router gives each capture only its own spans. A span belongs to the capture whose
    context it started in, or to the capture that owns its parent, so the agent's spans on a
    worker thread still arrive and two captures running at once never see each other's spans.
    Host telemetry that started outside any capture (a web request, a database call) is not
    captured. Strands' tracer is pointed at a private provider while any capture is open, so
    its spans reach the capture without passing through an exporter the global provider
    carries (a Langfuse client's, for example). Self-built emitters (OrderBot) can use
    :meth:`tracer`.

    Raises :class:`CaptureProviderError` if the global provider is not an SDK provider.
    """
    _router().attach(self)
    self._attached = True
    self._active_token = _ACTIVE_CAPTURE.set(self)

tracer

tracer(name)

A tracer whose spans this capture collects.

Source code in src/calibrated_agent_evaluation/execute/strands_capture.py
def tracer(self, name: str) -> Tracer:
    """A tracer whose spans this capture collects."""
    return self._provider.get_tracer(name)

StrandsExecutor

StrandsExecutor(
    *,
    log_dir=None,
    cassette_path=None,
    cassette_mode=None,
    model_id=_DEFAULT_MODEL_ID,
    region=_DEFAULT_REGION,
    timeout_seconds=_DEFAULT_TIMEOUT_SECONDS,
    frame="production-representative",
    temperature=_DEFAULT_TEMPERATURE,
)

An ExecutorBackend over Strands. Context manager for the temp log dir.

temperature defaults to 0.0 (what this executor has always sent) and accepts None to send no decoding parameter. That escape hatch exists because it is not a preference: the newest Anthropic generations reject an explicit temperature as deprecated and fail the request outright, so a hard-coded value silently restricted this executor to older models, and a caller whose own agent sends no decoding parameters could not reproduce its own subject here. Determinism for those models comes from the prompt and a structured output schema instead.

Parameters:

Name Type Description Default
log_dir Path | None

Where captured spans are written; a temporary directory when omitted.

None
cassette_path str | Path | None

A cassette to record or replay model calls through.

None
cassette_mode Mode | None

Whether the cassette replays, records, or both.

None
model_id str

The Bedrock model the agent runs on.

_DEFAULT_MODEL_ID
region str

The Bedrock region.

_DEFAULT_REGION
timeout_seconds float

The per-request timeout.

_DEFAULT_TIMEOUT_SECONDS
frame Frame

The sampling frame the produced traces are stamped with.

'production-representative'
temperature float | None

The decoding temperature, or None to send none.

_DEFAULT_TEMPERATURE
Source code in src/calibrated_agent_evaluation/execute/strands_executor.py
def __init__(
    self,
    *,
    log_dir: Path | None = None,
    cassette_path: str | Path | None = None,
    cassette_mode: Mode | None = None,
    model_id: str = _DEFAULT_MODEL_ID,
    region: str = _DEFAULT_REGION,
    timeout_seconds: float = _DEFAULT_TIMEOUT_SECONDS,
    frame: Frame = "production-representative",
    temperature: float | None = _DEFAULT_TEMPERATURE,
) -> None:
    self._explicit_log_dir = log_dir
    self._cassette_path = None if cassette_path is None else Path(cassette_path)
    self._cassette_mode = cassette_mode or mode_from_env(default=Mode.AUTO)
    self._model_id = model_id
    self._region = region
    self._temperature = temperature
    # Explicit bound on the live Bedrock connect/read so an unreachable peer raises within a
    # known window rather than hanging on a shifting SDK default (every network call is bounded
    # by an explicit timeout).
    self._timeout_seconds = timeout_seconds
    # The dataset frame the produced traces carry; the caller (synthesis of a discovery
    # frame) passes frame="discovery" so the tag survives execution (mirrors InspectExecutor).
    self._frame: Frame = frame
    self._tmpdir: tempfile.TemporaryDirectory[str] | None = None

log_dir property

log_dir

Where the captured OTLP-JSON export is persisted.

run

run(task, *, log_dir=None)

Run a StrandsTask and return one normalized, validated Trace.

Raises ExecutionError on any failure; missing extra, cassette error, unreachable model, empty capture, or invalid trace. Never returns a partial Trace.

Source code in src/calibrated_agent_evaluation/execute/strands_executor.py
def run(self, task: Any, *, log_dir: Path | None = None) -> list[Trace]:
    """Run a ``StrandsTask`` and return one normalized, validated ``Trace``.

    Raises ``ExecutionError`` on any failure; missing extra, cassette error, unreachable model,
    empty capture, or invalid trace. Never returns a partial Trace.
    """
    if not isinstance(task, StrandsTask):
        raise ExecutionError(
            f"StrandsExecutor.run expects a StrandsTask, got {type(task).__name__}"
        )
    span_capture_cls, bedrock_model_cls, cassette_model_cls = self._load_strands()
    effective_log_dir = log_dir or self.log_dir
    effective_log_dir.mkdir(parents=True, exist_ok=True)

    # Build+validate the cassette FIRST (stdlib); a corrupt file raises CassetteError here,
    # before any Strands work, and is wrapped as ExecutionError.
    model = self._build_model(bedrock_model_cls, cassette_model_cls)

    capture = span_capture_cls("calibrated_agent_evaluation.strands")
    try:
        with capture:
            capture.register_global()
            runnable = task.build(model)
            runnable(task.input)
        spans = capture.finished_spans
    except (CassetteMiss, CassetteError) as exc:
        raise ExecutionError(f"cassette error during strands run: {exc}") from exc
    except ExecutionError:
        raise
    except Exception as exc:
        raise ExecutionError(f"strands run failed: {type(exc).__name__}: {exc}") from exc

    if not spans:
        raise ExecutionError(
            "no spans captured for this run; the Strands run emitted nothing "
            "(was the capture provider installed / did the agent actually run?)"
        )

    log_path = self._write_export(spans, effective_log_dir, task)
    return [self._normalize(log_path, task)]

StrandsTask dataclass

StrandsTask(
    build,
    input,
    mode="latest",
    trace_id=None,
    taxonomy_cell=dict(),
)

A runnable Strands target for run.

build receives the executor's cassette-backed model and returns a runnable Strands object (an Agent/Graph/Swarm; all callable on the input). The executor injects the model so it owns replay/capture wiring; the caller only assembles topology. mode/ trace_id/taxonomy_cell are passed through to the StrandsSession normalizer so the produced Trace carries the right GenAI-convention branch and headers.

build instance-attribute

build

input instance-attribute

input

mode class-attribute instance-attribute

mode = 'latest'

trace_id class-attribute instance-attribute

trace_id = None

taxonomy_cell class-attribute instance-attribute

taxonomy_cell = field(default_factory=dict)

InspectTaskBuilder

InspectTaskBuilder(solver)

Assemble an Inspect Task from synthesized inputs.

solver is the agent under test (an Inspect solver/agent). Each SynthInput becomes a Sample whose metadata carries the taxonomy_cell + frame so they survive execution + normalization. Inspect is imported lazily so this module loads without the [inspect] extra.

Source code in src/calibrated_agent_evaluation/execute/task_builders.py
def __init__(self, solver: Any) -> None:
    self._solver = solver

build_all

build_all(inputs)

A single batched task: an Inspect Task already holds one Sample per input.

Batching is the point here (N separate evals would be slower and would scatter N logs) so this returns one task, which is what makes build_all the builder's decision rather than generate_and_run's.

Source code in src/calibrated_agent_evaluation/execute/task_builders.py
def build_all(self, inputs: list[SynthInput]) -> list[Any]:
    """A single batched task: an Inspect ``Task`` already holds one ``Sample`` per input.

    Batching is the point here (N separate evals would be slower and would scatter N logs) so
    this returns one task, which is what makes ``build_all`` the builder's decision rather than
    ``generate_and_run``'s.
    """
    return [self.build(inputs)]

build

build(inputs)

Build an Inspect Task with one Sample per input.

Source code in src/calibrated_agent_evaluation/execute/task_builders.py
def build(self, inputs: list[SynthInput]) -> Any:
    """Build an Inspect ``Task`` with one ``Sample`` per input."""
    if not inputs:
        raise SynthError("InspectTaskBuilder.build received no inputs; refusing an empty task")
    from inspect_ai import Task
    from inspect_ai.dataset import Sample

    samples = [
        Sample(
            input=item.text,
            metadata={"taxonomy_cell": dict(item.cell), "frame": item.frame},
        )
        for item in inputs
    ]
    return Task(dataset=samples, solver=self._solver)

StrandsTaskBuilder

StrandsTaskBuilder(build_topology, *, mode='latest')

Assemble a StrandsTask from synthesized inputs.

build_topology is the Strands topology builder (Callable[[model], Agent|Graph|Swarm]) the StrandsExecutor injects the model into. The produced StrandsTask copies the first input's taxonomy_cell so the normalized Trace is taxonomy-attributable; the load-bearing contract between synthesis and the executor: synth._verify_frame_propagation rejects a Trace whose cell doesn't match a synthesized input.

Source code in src/calibrated_agent_evaluation/execute/task_builders.py
def __init__(
    self,
    build_topology: Callable[[Any], Any],
    *,
    mode: str = "latest",
) -> None:
    self._build_topology = build_topology
    self._mode = mode

build_all

build_all(inputs)

One StrandsTask per input: a StrandsTask carries exactly one input string.

This is the fix for the collapse. build takes a list for protocol compatibility but can only represent the first element, so a caller that built once and ran once evaluated 1 of N inputs; with no error, because the single returned trace carried a valid frame and cell.

Source code in src/calibrated_agent_evaluation/execute/task_builders.py
def build_all(self, inputs: list[SynthInput]) -> list[StrandsTask]:
    """One ``StrandsTask`` per input: a StrandsTask carries exactly one ``input`` string.

    This is the fix for the collapse. ``build`` takes a list for protocol compatibility but can
    only represent the first element, so a caller that built once and ran once evaluated 1 of N
    inputs; with no error, because the single returned trace carried a valid frame and cell.
    """
    if not inputs:
        raise SynthError(
            "StrandsTaskBuilder.build_all received no inputs; refusing an empty run"
        )
    return [self.build([item]) for item in inputs]

build_conversation_turn

build_conversation_turn(item, history)

Build the task for one agent turn of a multi-turn conversation.

history is the conversation so far as Strands messages ({"role", "content"}), which is what Agent.__call__ takes as a prior-message list (verified against strands-agents==1.45.0), so the agent sees every earlier turn the way it would in production. The cell and frame ride along as they do single-shot, so the final turn's trace stays taxonomy-attributable.

Source code in src/calibrated_agent_evaluation/execute/task_builders.py
def build_conversation_turn(
    self, item: SynthInput, history: list[dict[str, Any]]
) -> StrandsTask:
    """Build the task for one agent turn of a multi-turn conversation.

    ``history`` is the conversation so far as Strands messages (``{"role", "content"}``), which
    is what ``Agent.__call__`` takes as a prior-message list (verified against
    ``strands-agents==1.45.0``), so the agent sees every earlier turn the way it would in
    production. The cell and frame ride along as they do single-shot, so the final turn's trace
    stays taxonomy-attributable.
    """
    if not history:
        raise SynthError("a conversation turn needs the history up to it, which was empty")
    return StrandsTask(
        build=self._build_topology,
        input=list(history),
        mode=self._mode,  # type: ignore[arg-type]
        taxonomy_cell=dict(item.cell),
    )

build

build(inputs)

Build a StrandsTask for inputs[0]. Prefer build_all for a list.

Kept single-task because that is what a StrandsTask is. It no longer silently ignores the tail: more than one input here means the caller bypassed build_all, and one input would be evaluated while the rest vanished, so it refuses instead.

Source code in src/calibrated_agent_evaluation/execute/task_builders.py
def build(self, inputs: list[SynthInput]) -> StrandsTask:
    """Build a ``StrandsTask`` for ``inputs[0]``. Prefer :meth:`build_all` for a list.

    Kept single-task because that is what a ``StrandsTask`` is. It no longer silently ignores
    the tail: more than one input here means the caller bypassed ``build_all``, and one input
    would be evaluated while the rest vanished, so it refuses instead.
    """
    if len(inputs) > 1:
        raise SynthError(
            f"StrandsTaskBuilder.build received {len(inputs)} inputs but a StrandsTask "
            "carries exactly one; call build_all(inputs) to run them all"
        )
    if not inputs:
        raise SynthError("StrandsTaskBuilder.build received no inputs; refusing an empty task")
    first = inputs[0]
    return StrandsTask(
        build=self._build_topology,
        input=first.text,
        mode=self._mode,  # type: ignore[arg-type]
        taxonomy_cell=dict(first.cell),
    )

model_stamp

model_stamp(model_id, config=None)

The model identity an entry is recorded under: its id and the decoding config it ran with.

Returns None when the model id is unknown, which records and checks nothing.

Source code in src/calibrated_agent_evaluation/execute/cassette.py
def model_stamp(
    model_id: str | None, config: dict[str, Any] | None = None
) -> dict[str, Any] | None:
    """The model identity an entry is recorded under: its id and the decoding config it ran with.

    Returns ``None`` when the model id is unknown, which records and checks nothing.
    """
    if model_id is None:
        return None
    return _canonical({"id": model_id, "config": dict(config or {})})

capture_spans

capture_spans(service_name)

Collect the spans emitted inside the block under service_name.

Use it as with capture_spans("orderbot") as cap:, then read cap.finished_spans.

Source code in src/calibrated_agent_evaluation/execute/strands_capture.py
@contextmanager
def capture_spans(service_name: str) -> Generator[SpanCapture]:
    """Collect the spans emitted inside the block under ``service_name``.

    Use it as ``with capture_spans("orderbot") as cap:``, then read ``cap.finished_spans``.
    """
    cap = SpanCapture(service_name)
    with cap:
        yield cap

spans_to_otlp_json

spans_to_otlp_json(spans)

Encode finished SDK spans into a canonical OTLP-JSON dict ({"resourceSpans": [...]}).

Source code in src/calibrated_agent_evaluation/execute/strands_capture.py
def spans_to_otlp_json(spans: tuple[ReadableSpan, ...]) -> dict[str, Any]:
    """Encode finished SDK spans into a canonical OTLP-JSON dict (``{"resourceSpans": [...]}``)."""
    traces_data = encode_spans(spans)
    # preserving_proto_field_name=False -> camelCase wire keys (resourceSpans/scopeSpans/...),
    # which is the JSON form the OTLP/HTTP exporter and collectors emit and the normalizer reads.
    return MessageToDict(traces_data, preserving_proto_field_name=False)

write_otlp_json

write_otlp_json(spans, path)

Freeze spans to path as pretty OTLP-JSON; returns the written dict.

Source code in src/calibrated_agent_evaluation/execute/strands_capture.py
def write_otlp_json(spans: tuple[ReadableSpan, ...], path: Path) -> dict[str, Any]:
    """Freeze ``spans`` to ``path`` as pretty OTLP-JSON; returns the written dict."""
    payload = spans_to_otlp_json(spans)
    path.parent.mkdir(parents=True, exist_ok=True)
    path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n", encoding="utf-8")
    return payload