calibrated_agent_evaluation.execute¶
calibrated_agent_evaluation.execute ¶
Execution plane: run agents programmatically and get normalized Traces.
The public surface: ExecutorBackend (the protocol), InspectExecutor (Inspect-backed) and
StrandsExecutor/StrandsTask (Strands-backed), ExecutionError (the typed error
family), and the Cassette record/replay layer. The vendor SDKs are imported lazily so
import calibrated_agent_evaluation.execute succeeds without the [inspect]/[strands]
extras installed.
ExecutorBackend ¶
Bases: Protocol
Run agents and return normalized, validated Traces.
The methodology plane programs against this protocol; it never sees Inspect or boto3 types.
run ¶
Execute an agent task and return normalized Traces.
Raises ExecutionError on any failure (eval error, cassette miss, unreachable model,
invalid trace from normalization). Never returns an unvalidated Trace.
Source code in src/calibrated_agent_evaluation/execute/base.py
Cassette ¶
A JSON-file-backed record/replay store for one logical capture run.
Thread-safe: the in-memory index and the append are guarded by a lock. Safe across processes
too: each append takes an advisory lock on a sidecar .<name>.lock file, merges in what
other recorders wrote since, and rewrites the file atomically through a uniquely named
temporary, so a crashed capture leaves a valid (if partial) cassette and two recorders on one
file keep both their entries.
Source code in src/calibrated_agent_evaluation/execute/cassette.py
lookup ¶
Return a copy of the recorded response for fingerprint fp, or None if absent.
model is this run's model_stamp. An entry recorded under a different stamp
raises CassetteModelMismatch. An entry recorded with no stamp is returned, unless
require_model is set: then it raises CassetteModelMismatch too, since nothing
shows it came from model.
Source code in src/calibrated_agent_evaluation/execute/cassette.py
record ¶
Record response for fp under model and write the cassette file.
Raises:
| Type | Description |
|---|---|
CassetteError
|
The file could not be written. The cassette is left as it was, in memory and on file: the response is not recorded, and an entry it was replacing stays. |
Source code in src/calibrated_agent_evaluation/execute/cassette.py
play ¶
Return the response for request, honoring the cassette mode.
live_call is a zero-arg callable that performs the real provider request and returns its
response dict; it is invoked only in RECORD mode or on an AUTO miss. In REPLAY mode a miss
raises CassetteMiss. model is the run's model_stamp, checked against
a hit and recorded with a new entry; require_model refuses a hit with no stamp, as
lookup does.
Source code in src/calibrated_agent_evaluation/execute/cassette.py
aplay
async
¶
Async twin of play; live_call is an async (awaitable) callable.
Source code in src/calibrated_agent_evaluation/execute/cassette.py
CassetteError ¶
Bases: RuntimeError
A cassette could not be read or written (wraps the underlying OS/JSON error).
CassetteMiss ¶
Bases: RuntimeError
Raised in REPLAY mode when a request fingerprint has no recorded response.
CassetteModelMismatch ¶
Bases: CassetteError
A cassette entry was recorded with a different model or decoding config than this run's.
Replaying it would attribute one model's answers to another. Entries record the model that produced them; an entry with no record of its model (a cassette recorded before entries carried one) is replayed as before, unless the caller requires the record, which refuses it too.
Mode ¶
ExecutionError ¶
Bases: Exception
A typed error from the execution plane.
Wraps vendor/SDK exceptions via chaining (raise ExecutionError(...) from exc). The message
carries diagnostic detail (file paths, eval status, fingerprint); the chained __cause__
preserves the original traceback for debugging.
JudgeConfigurationError ¶
Bases: ExecutionError
The judge model was called in a way that cannot work, whatever the model does.
A subclass so an existing except ExecutionError boundary still catches it, but named so a
caller that distinguishes transient from permanent can. judges/runtime._judge_one retries a
bare ExecutionError as transient, so a permanent configuration fault raised as one burned
the whole retry budget and then reported itself as "model failed after N attempts" -- sending
the reader to look at the model instead of at the call site.
SandboxError ¶
Bases: ExecutionError
Sandboxed execution could not be carried out.
Raised when the process could not start, a remote sandbox was unreachable, or its response was
malformed. A subclass so an existing except ExecutionError boundary still catches it. This
is an infrastructure failure, distinct from a program that ran and failed its tests (that is a
PASS/FAIL verdict, not an error); the execution scorers re-wrap it as ScorerError at their
boundary and never return a Label for it.
CassetteJudgeModel ¶
CassetteJudgeModel(
cassette,
*,
supports_schema=True,
record_model=None,
temperature=None,
model_id=None,
)
Offline judge-model backend backed by a Cassette (test double + deterministic CI).
Mirrors InspectJudgeModel's surface so judges/ code is identical in both worlds.
supports_response_schema is explicit (defaults to True) so a test can exercise either path;
the recorded ModelOutput JSON is replayed by content fingerprint. A REPLAY miss raises
CassetteMiss.
A repeat of an identical request (sample above 0) and a set temperature are part of
the fingerprint, so a repeat replays its own recorded draw. Neither is in the fingerprint at
its default, so cassettes recorded before either existed still replay.
model_id names the model this run judges with (it defaults to record_model). Entries
are recorded under it, and replaying an entry recorded with a different model or temperature
raises CassetteModelMismatch.
Source code in src/calibrated_agent_evaluation/execute/judge_model.py
supports_response_schema ¶
generate_structured ¶
Return the recorded structured reply for this request.
Source code in src/calibrated_agent_evaluation/execute/judge_model.py
InspectJudgeModel ¶
Real judge-model backend on Inspect model-runners.
Constructed with an Inspect model string (e.g. a "bedrock/…claude-sonnet-4-6" id).
A model whose provider/name does not natively support response_schema is served through the
forced-tool path; never free text.
temperature is sent only when set. A set temperature the provider did not send (Inspect
drops it for Claude 4.7 and later on Bedrock) is refused as a
JudgeConfigurationError, so a judge never runs at a temperature other than the one
recorded for it.
Source code in src/calibrated_agent_evaluation/execute/judge_model.py
supports_response_schema ¶
True only for Bedrock Claude models (verified _providers/bedrock.py:642-657).
Bedrock honors response_schema for Claude via the Converse output_config.format, and
warn-drops it for non-Claude Bedrock; SageMaker has no support. Rather than trust a sent
schema was honored, we gate on the known-supporting family and route everything else through
the forced-tool path. Conservative by design: a model we can't prove supports it
uses the tool path, which is always structured.
Source code in src/calibrated_agent_evaluation/execute/judge_model.py
generate_text ¶
Return the model's plain-text completion for a (system, user) prompt.
The free-text sibling of generate_structured; used by the BedrockRenderer
to render a taxonomy cell into a natural-language input. Wraps the verified Inspect model
seam (get_model(model).generate(messages).completion, the exact call this class already
makes internally). Raises ExecutionError on any model failure (wrapped from the
vendor exception); never returns a fabricated or partial string.
Source code in src/calibrated_agent_evaluation/execute/judge_model.py
generate_structured ¶
Drive the async Inspect model call under a sync boundary; extract a structured reply.
sample needs nothing here: Inspect does not cache by default, so every call is a draw.
Refuses up front when an event loop is already running. asyncio.run raises RuntimeError
there, which the broad except Exception below wrapped as ExecutionError -- and
judges/runtime._judge_one treats any ExecutionError as transient and retries it. So
calling a judge from inside an event loop burned max_retries attempts and then failed with
"model failed after 4 attempts", naming a permanent config error as flakiness.
JudgeConfigurationError is outside the retry family, so it surfaces on the first attempt.
Source code in src/calibrated_agent_evaluation/execute/judge_model.py
JudgeModelBackend ¶
Bases: Protocol
Call a judge model and return a forced-structured reply: the judge model seam.
judges/ programs against this; it never imports inspect_ai. Implementations raise
ExecutionError on any model failure (the judges/ boundary re-wraps it as
JudgeRuntimeError).
supports_response_schema ¶
Whether the selected model honors native response_schema.
Drives the path choice: True → native structured output; False → forced tool call.
generate_structured ¶
Return a forced-structured reply for the (system, user) prompt against schema.
schema is a plain JSON-schema dict describing the verdict object (critique-before
-verdict is guaranteed by the caller). sample numbers repeats of an identical request
(0 is the first): a backend that answers by request content keeps repeats apart, so a
repeat is a fresh draw and not the first reply played back. Raises
ExecutionError on model failure or when no structured payload can be extracted
(enforced with the typed error at the judges/ boundary).
Source code in src/calibrated_agent_evaluation/execute/judge_model.py
StructuredReply
dataclass
¶
A model's forced-structured verdict, normalized off any vendor type.
fields is the structured payload (e.g. {"critique": ..., "verdict": "pass"|"fail"});
via records which path produced it; "response_schema" (native structured output) or
"tool_call" (the forced-tool fallback); for audit. judges/ validates fields against
the verdict schema; this seam does not judge correctness, only that a structured payload exists.
LocalSubprocessSandbox ¶
The default, dependency-free sandbox: a timed, isolated-mode subprocess with a minimal env.
See the module docstring for the threat model: this is best-effort resource and accident
isolation, not a security boundary. max_memory_bytes caps the child's address space where
POSIX allows it.
Source code in src/calibrated_agent_evaluation/execute/sandbox.py
run_code ¶
Run code as a Python program in a temp cwd seeded with files.
Raises SandboxError for an unsupported language, a files name that resolves
outside the sandbox directory (absolute, or escaping through ..), or if the child
cannot be started.
Source code in src/calibrated_agent_evaluation/execute/sandbox.py
SandboxBackend ¶
Bases: Protocol
Run a snippet of code and return a normalized SandboxResult.
The methodology plane programs against this; it never imports a vendor SDK type.
Implementations raise SandboxError when the execution could not be carried out (as
opposed to a program that ran and exited non-zero, which is a normal SandboxResult).
run_code ¶
Execute code (seeded with files in its working directory) and return the result.
Raises SandboxError if the execution could not be carried out.
Source code in src/calibrated_agent_evaluation/execute/sandbox.py
SandboxResult
dataclass
¶
The normalized outcome of one sandboxed execution, free of any vendor type.
exit_code is 0 on clean completion; timed_out is True when the wall-clock limit fired
(exit_code is then non-zero and undefined). limits_applied is False when the platform
offered no POSIX resource limits, so a run on a limit-less host is auditable rather than
silently unbounded.
BedrockEmbedder ¶
BedrockEmbedder(
*,
model_id=_DEFAULT_MODEL_ID,
region=_DEFAULT_REGION,
timeout_seconds=_DEFAULT_TIMEOUT_SECONDS,
)
Concrete Embedder on Bedrock Titan v2 via boto3 invoke_model.
Construction is cheap (the boto3 client is built lazily on first embed, so importing
this module needs no AWS session and importing calibrated_agent_evaluation.execute needs no
boto3).
Source code in src/calibrated_agent_evaluation/execute/bedrock_embedder.py
embed ¶
Return one embedding vector per input text, in order.
Raises ClusteringError if the backend is unreachable, a call fails, or the number
of returned vectors does not equal len(texts); never a truncated/misaligned list.
Source code in src/calibrated_agent_evaluation/execute/bedrock_embedder.py
BedrockRenderer ¶
Concrete Renderer on Bedrock via the Inspect model seam.
model is an Inspect model string (e.g. "bedrock/global.anthropic.claude-sonnet-4-6").
Construction is cheap (no network); the model call happens in render.
Source code in src/calibrated_agent_evaluation/execute/bedrock_renderer.py
render ¶
Render one taxonomy cell into a non-empty NL input.
Raises SynthError if the model is unreachable/errors (wrapping the vendor
exception) or returns empty/whitespace text; never fabricates or returns an empty input.
Source code in src/calibrated_agent_evaluation/execute/bedrock_renderer.py
InspectExecutor ¶
InspectExecutor(
model="cassette/orderbot",
*,
log_dir=None,
cassette_path=None,
cassette_mode=None,
cassette_inner_model=None,
frame="production-representative",
)
An ExecutorBackend over Inspect AI's eval().
Context manager: __enter__/__exit__ manage the temp log directory (if no explicit
log_dir was provided) and ensure resources are cleaned up.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
str
|
The Inspect model the agent runs on. |
'cassette/orderbot'
|
log_dir
|
Path | None
|
Where Inspect writes its logs; a temporary directory when omitted. |
None
|
cassette_path
|
str | Path | None
|
A cassette to record or replay model calls through. |
None
|
cassette_mode
|
Mode | None
|
Whether the cassette replays, records, or both. |
None
|
cassette_inner_model
|
str | None
|
The real model a recording cassette calls on a miss. |
None
|
frame
|
Frame
|
The sampling frame the produced traces are stamped with. |
'production-representative'
|
Source code in src/calibrated_agent_evaluation/execute/inspect_executor.py
run ¶
Run an Inspect task and return normalized, validated Traces.
Raises ExecutionError on any failure; eval error, cassette miss, unreachable model,
or invalid trace from normalization.
Source code in src/calibrated_agent_evaluation/execute/inspect_executor.py
CaptureProviderError ¶
Bases: ExecutionError
The process's global tracer provider cannot carry spans to a capture.
Raised when something other than the OpenTelemetry SDK's TracerProvider is already the
global provider (a vendor distribution, or the API's no-op provider), so capture has no way to
attach a span processor to it.
SpanCapture ¶
Owns a TracerProvider + in-memory exporter; yields a tracer and the captured spans.
Use as a context manager so the provider is flushed and shut down deterministically
(I/O resources are context managers). After the with block,
finished_spans holds every span the agent emitted, in completion order.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
service_name
|
str
|
The OTel service name the captured spans are recorded under. |
required |
Source code in src/calibrated_agent_evaluation/execute/strands_capture.py
provider
property
¶
The real provider: hand this to StrandsTelemetry(provider) or set it as global.
register_global ¶
Receive the spans the agent run inside this capture emits through global tracers.
Framework-instrumented code (Strands, or a module-level trace.get_tracer(...)) emits
through the OTel global provider, and OTel lets that be set only once per process. A
ProxyTracer handed out before then also binds to the first real provider for good. So
capture never replaces the global: it installs one long-lived provider the first time (or
joins an SDK provider the application already installed) and hangs a router off it.
The router gives each capture only its own spans. A span belongs to the capture whose
context it started in, or to the capture that owns its parent, so the agent's spans on a
worker thread still arrive and two captures running at once never see each other's spans.
Host telemetry that started outside any capture (a web request, a database call) is not
captured. Strands' tracer is pointed at a private provider while any capture is open, so
its spans reach the capture without passing through an exporter the global provider
carries (a Langfuse client's, for example). Self-built emitters (OrderBot) can use
tracer.
Raises CaptureProviderError if the global provider is not an SDK provider.
Source code in src/calibrated_agent_evaluation/execute/strands_capture.py
StrandsExecutor ¶
StrandsExecutor(
*,
log_dir=None,
cassette_path=None,
cassette_mode=None,
model_id=_DEFAULT_MODEL_ID,
region=_DEFAULT_REGION,
timeout_seconds=_DEFAULT_TIMEOUT_SECONDS,
frame="production-representative",
temperature=_DEFAULT_TEMPERATURE,
)
An ExecutorBackend over Strands. Context manager for the temp log dir.
temperature defaults to 0.0 (what this executor has always sent) and accepts None
to send no decoding parameter. That escape hatch exists because it is not a preference: the
newest Anthropic generations reject an explicit temperature as deprecated and fail the
request outright, so a hard-coded value silently restricted this executor to older models, and a
caller whose own agent sends no decoding parameters could not reproduce its own subject here.
Determinism for those models comes from the prompt and a structured output schema instead.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
log_dir
|
Path | None
|
Where captured spans are written; a temporary directory when omitted. |
None
|
cassette_path
|
str | Path | None
|
A cassette to record or replay model calls through. |
None
|
cassette_mode
|
Mode | None
|
Whether the cassette replays, records, or both. |
None
|
model_id
|
str
|
The Bedrock model the agent runs on. |
_DEFAULT_MODEL_ID
|
region
|
str
|
The Bedrock region. |
_DEFAULT_REGION
|
timeout_seconds
|
float
|
The per-request timeout. |
_DEFAULT_TIMEOUT_SECONDS
|
frame
|
Frame
|
The sampling frame the produced traces are stamped with. |
'production-representative'
|
temperature
|
float | None
|
The decoding temperature, or |
_DEFAULT_TEMPERATURE
|
Source code in src/calibrated_agent_evaluation/execute/strands_executor.py
run ¶
Run a StrandsTask and return one normalized, validated Trace.
Raises ExecutionError on any failure; missing extra, cassette error, unreachable model,
empty capture, or invalid trace. Never returns a partial Trace.
Source code in src/calibrated_agent_evaluation/execute/strands_executor.py
StrandsTask
dataclass
¶
A runnable Strands target for run.
build receives the executor's cassette-backed model and returns a runnable Strands object
(an Agent/Graph/Swarm; all callable on the input). The executor injects the model
so it owns replay/capture wiring; the caller only assembles topology. mode/
trace_id/taxonomy_cell are passed through to the StrandsSession normalizer so the
produced Trace carries the right GenAI-convention branch and headers.
InspectTaskBuilder ¶
Assemble an Inspect Task from synthesized inputs.
solver is the agent under test (an Inspect solver/agent). Each SynthInput becomes a
Sample whose metadata carries the taxonomy_cell + frame so they survive
execution + normalization. Inspect is imported lazily so this module loads without the
[inspect] extra.
Source code in src/calibrated_agent_evaluation/execute/task_builders.py
build_all ¶
A single batched task: an Inspect Task already holds one Sample per input.
Batching is the point here (N separate evals would be slower and would scatter N logs) so
this returns one task, which is what makes build_all the builder's decision rather than
generate_and_run's.
Source code in src/calibrated_agent_evaluation/execute/task_builders.py
build ¶
Build an Inspect Task with one Sample per input.
Source code in src/calibrated_agent_evaluation/execute/task_builders.py
StrandsTaskBuilder ¶
Assemble a StrandsTask from synthesized inputs.
build_topology is the Strands topology builder (Callable[[model], Agent|Graph|Swarm])
the StrandsExecutor injects the model into. The produced StrandsTask copies the first
input's taxonomy_cell so the normalized Trace is taxonomy-attributable; the load-bearing
contract between synthesis and the executor: synth._verify_frame_propagation rejects a
Trace whose cell doesn't match a synthesized input.
Source code in src/calibrated_agent_evaluation/execute/task_builders.py
build_all ¶
One StrandsTask per input: a StrandsTask carries exactly one input string.
This is the fix for the collapse. build takes a list for protocol compatibility but can
only represent the first element, so a caller that built once and ran once evaluated 1 of N
inputs; with no error, because the single returned trace carried a valid frame and cell.
Source code in src/calibrated_agent_evaluation/execute/task_builders.py
build_conversation_turn ¶
Build the task for one agent turn of a multi-turn conversation.
history is the conversation so far as Strands messages ({"role", "content"}), which
is what Agent.__call__ takes as a prior-message list (verified against
strands-agents==1.45.0), so the agent sees every earlier turn the way it would in
production. The cell and frame ride along as they do single-shot, so the final turn's trace
stays taxonomy-attributable.
Source code in src/calibrated_agent_evaluation/execute/task_builders.py
build ¶
Build a StrandsTask for inputs[0]. Prefer build_all for a list.
Kept single-task because that is what a StrandsTask is. It no longer silently ignores
the tail: more than one input here means the caller bypassed build_all, and one input
would be evaluated while the rest vanished, so it refuses instead.
Source code in src/calibrated_agent_evaluation/execute/task_builders.py
model_stamp ¶
The model identity an entry is recorded under: its id and the decoding config it ran with.
Returns None when the model id is unknown, which records and checks nothing.
Source code in src/calibrated_agent_evaluation/execute/cassette.py
capture_spans ¶
Collect the spans emitted inside the block under service_name.
Use it as with capture_spans("orderbot") as cap:, then read cap.finished_spans.
Source code in src/calibrated_agent_evaluation/execute/strands_capture.py
spans_to_otlp_json ¶
Encode finished SDK spans into a canonical OTLP-JSON dict ({"resourceSpans": [...]}).
Source code in src/calibrated_agent_evaluation/execute/strands_capture.py
write_otlp_json ¶
Freeze spans to path as pretty OTLP-JSON; returns the written dict.