Skip to content

calibrated_agent_evaluation.agentcore

calibrated_agent_evaluation.agentcore

Amazon Bedrock AgentCore plug-in: opt-in adapters behind the library's existing seams.

Nothing here is a default, and importing this package loads no AWS SDK. Adapter names load on first access; without the [agentcore] extra that access raises AgentCoreExtraMissing naming the command that installs it.

The plug-in has six parts:

  • the read-only account check behind calibrated-agent-evaluation doctor --agentcore (run_checks);
  • the window ingest behind calibrated-agent-evaluation ingest --source agentcore, which reads a runtime's traces for a time window from AgentCore Observability (AgentCoreTraceSource, ingest_agentcore);
  • the Runtime executor behind calibrated-agent-evaluation generate --run --executor agentcore, which sends each generated input to an agent deployed to AgentCore Runtime and stores one trace per input (AgentCoreRuntimeExecutor), reading each run's span tree with the per-run span reader (AgentCoreSpanReader);
  • dataset publish and pull behind calibrated-agent-evaluation dataset publish and pull, which publish the test scenarios built from the failures reviewers caught as a version of an AgentCore evaluation dataset, and read a version back (publish, pull);
  • the Code Interpreter sandbox behind calibrated-agent-evaluation score --scorer code --sandbox agentcore, which runs an agent's generated code in a managed AgentCore Code Interpreter session (CodeInterpreterSandbox);
  • the strands-agents-evals failure detector: failure_vocabulary reads the detector's failure categories as typed entries, FailureDetector runs it on a normalized trace and returns a Detection or a typed refusal, and DetectorDraftNamer names draft failure-mode clusters from that vocabulary for calibrated_agent_evaluation.cluster.build_draft. Both call only the strands Model the caller passes, refuse a live call past max_live_calls (0 by default), and can record and replay their calls from a cassette. The result types and the detector errors are vendor-free, so they import without the extra.

ARM_ARN_ATTRIBUTE module-attribute

ARM_ARN_ATTRIBUTE = (
    "aws.agentcore.gateway.routing_experiment_arn"
)

ARM_VARIANT_ATTRIBUTE module-attribute

ARM_VARIANT_ATTRIBUTE = (
    "aws.agentcore.gateway.routing_experiment_variant_name"
)

DEFAULT_CHECKS module-attribute

DEFAULT_CHECKS = (
    RegionCheck(),
    TransactionSearchCheck(),
    IndexingCheck(),
    SpanLogGroupCheck(),
    RuntimeLogGroupCheck(),
    OmniDomainCheck(),
    ContentExtractionCheck(),
)

AUDIT_READ_OPERATIONS module-attribute

AUDIT_READ_OPERATIONS = frozenset(
    {
        (_DATA, "GetABTest"),
        (_DATA, "GetRecommendation"),
        (_CONTROL, "GetConfigurationBundleVersion"),
        (_CONTROL, "GetOnlineEvaluationConfig"),
    }
)

INGEST_READ_OPERATIONS module-attribute

INGEST_READ_OPERATIONS = frozenset(
    {
        ("logs", "StartQuery"),
        ("logs", "GetQueryResults"),
        ("logs", "FilterLogEvents"),
        ("cloudwatchomni", "StartTelemetryQuerySession"),
        ("cloudwatchomni", "StartTelemetryQuery"),
        ("cloudwatchomni", "GetTelemetryQueryResults"),
        ("cloudwatchomni", "StopTelemetryQuerySession"),
    }
)

AGENTCORE_POLICY_VOCABULARY module-attribute

AGENTCORE_POLICY_VOCABULARY = PolicyVocabulary(
    selector_key="rpc.method",
    per_call_value="AuthorizeAction",
    tool_list_value="PartiallyAuthorizeActions",
    gateway_call_key="url.path",
    gateway_call_value="tools/call",
    tool_key="tool.name",
    decision_key="aws.agentcore.policy.authorization_decision",
    decisions={"ALLOW": "allow", "DENY": "deny"},
    asynchronous_decision="LOG_ONLY_ASYNC",
    mode_key="aws.agentcore.gateway.policy.mode",
    modes={"ENFORCE": "enforce", "LOG_ONLY": "log_only"},
    determining_key="aws.agentcore.policy.determining_policies",
    flipping_key="aws.agentcore.policy.log_only_decision_flipping_policies",
    incomplete_key="aws.agentcore.policy.log_only_eval_incomplete",
    engine_key="aws.agentcore.gateway.policy.arn",
    reason_key="aws.agentcore.policy.authorization_reason",
)

AgentCoreAppliedBundleMismatch

Bases: AgentCoreError

A trace lacks the bundle attribute or value the operator required, or carries another.

AgentCoreArmHalfStamped

Bases: AgentCoreError

A span carries one of the two arm attributes without the other.

A request whose baggage named one key stamps its spans by halves, so such a trace's arm cannot be read safely; the trace is refused.

AgentCoreArmSessionAmbiguous

Bases: AgentCoreError

A trace's /invocations server spans name two runtime sessions.

AgentCoreArmSessionSplit

Bases: AgentCoreError

One runtime session's traces sit in two variants, so the session's arm is ambiguous.

AgentCoreArmsMissing

Bases: AgentCoreError

An A/B arms read found no trace it can assign to the audited test.

No span in the window carries both arm attributes, or no stamped trace names the audited experiment. Refused rather than stored as a read with every trace unassigned.

AgentCoreArmSourceIdRepeated

Bases: AgentCoreError

Two or more traces arrived under one source id, so none of them can be assigned.

AgentCoreArmsReadChanged

Bases: AgentCoreError

A re-read under a stored read id delivered other traces or assigned one differently.

A stored read never changes; the re-read is refused and nothing from it is written.

AgentCoreContentMissing

Bases: AgentCoreError

A span's message content was not found in the joined otel-rt-logs records.

Raised rather than yielding a span with empty input/output/tool content: an under-joined trace reads as a complete one, which is the silent-loss failure this ingest exists to prevent.

AgentCoreError

Bases: CalibratedAgentEvaluationDataError

An AgentCore integration call or configuration failed.

AgentCoreExtraMissing

AgentCoreExtraMissing(purpose)

Bases: MissingExtraError, AgentCoreError

An AgentCore module was asked for but the [agentcore] extra is not installed.

Source code in src/calibrated_agent_evaluation/agentcore/errors.py
def __init__(self, purpose: str) -> None:
    super().__init__(("agentcore",), purpose)

AgentCoreIndexingTooLow

Bases: AgentCoreError

X-Ray indexes below 100% while a read goes through X-Ray or the trace list.

A read that depends on the index would see only the indexed fraction, so the result is an unknown subset. Refused, naming the observed percentage.

AgentCoreMixedArm

Bases: AgentCoreError

One trace's spans name two experiments or two variants.

AgentCoreMixedSession

Bases: AgentCoreError

A single trace carries more than one session.id.

The AgentCore Evaluate API rejects such input, and fusing the sessions would mislabel the trace. Refused by default; an explicit re-key opts into collapsing them to the runtime session.

AgentCorePartialResult

Bases: AgentCoreError

A CloudWatch Omni query returned partialResultsDetected.

The rows are an unknown subset of the match, so returning them would under-count silently. Refused rather than returned.

AgentCoreScoreIncomplete

Bases: AgentCoreError

An evaluator result lacks a field a Label requires (trace/session id, or a value).

Also raised when an Evaluate call returns no result at all for the trace it targeted. Refused rather than stored as a label with an empty or inferred field, or a run returning fewer verdicts than traces, which would enter an unfounded or incomplete set into the store.

AgentCoreSettlingLag

Bases: AgentCoreError

The requested window ends inside the read-after-write settling lag.

A trace written moments ago is not yet fully readable, so the window would return a partial trace. Refused rather than returned mid-write.

AgentCoreTruncatedResult

Bases: AgentCoreError

A CloudWatch Logs Insights query matched more records than it returned.

recordsMatched exceeded the rows retrieved, so the trace set is incomplete. Refused rather than returned truncated.

AwsCredentialsMissing

Bases: AgentCoreError

No AWS credentials could be resolved for the requested profile.

BilledCallBudgetExceeded

BilledCallBudgetExceeded(message, *, operation=None)

Bases: AgentCoreError

A billed AWS operation would exceed the billed-call budget; refused before sending.

operation is the refused call, (service, operation), or None when not named.

Source code in src/calibrated_agent_evaluation/agentcore/errors.py
def __init__(self, message: str, *, operation: tuple[str, str] | None = None) -> None:
    super().__init__(message)
    self.operation = operation

operation instance-attribute

operation = operation

DatasetDecodeError

Bases: AgentCoreError

A dataset example has a field the codec does not map; refused rather than dropped.

DatasetInputError

Bases: AgentCoreError

A dataset name or version given to publish or pull is one AgentCore can't accept.

DatasetNameConflict

Bases: AgentCoreError

A dataset with the requested name exists but is not the one this slice was published to.

DatasetPublishError

Bases: AgentCoreError

Publishing a slice could not complete; the message says what state the dataset is left in.

DatasetSchemaUnsupported

Bases: AgentCoreError

A dataset uses a schema type the codec does not support.

DatasetVersionNotFound

Bases: AgentCoreError

The requested dataset or dataset version does not exist (or was deleted).

DetectorCallError

Bases: DetectorError

A model call made for the detector or the namer failed; the cause is chained.

DetectorCallLimitExceeded

DetectorCallLimitExceeded(limit)

Bases: DetectorError

A live model call would exceed max_live_calls; refused before it was sent.

limit is the max_live_calls value that refused it.

Source code in src/calibrated_agent_evaluation/agentcore/errors.py
def __init__(self, limit: int) -> None:
    self.limit = limit
    super().__init__(
        "a live model call was refused before it was sent, because it would exceed "
        f"max_live_calls={limit}; raise max_live_calls to allow more live calls, or replay "
        "the calls from a cassette"
    )

limit instance-attribute

limit = limit

DetectorConfigError

Bases: DetectorError

The detector or the namer was built with a configuration it cannot run under.

DetectorError

Bases: AgentCoreError

The failure detector or the draft namer refused an input, a configuration or a response.

DetectorOutputError

Bases: DetectorError

The detector's model returned a response that cannot be read faithfully.

DetectorSessionTooLarge

Bases: DetectorError

A trace is too large for the detector's model to read in one call.

DraftNamingError

Bases: DetectorError

The namer's response holds no usable vocabulary name for every cluster.

PolicySpanMalformed

Bases: AgentCoreError

A Gateway span carrying a Policy decision could not be read whole.

Raised for a per-call decision span whose tool, decision or mode is missing or unknown, whose list attribute is in an unknown encoding, and for a span carrying the decision attribute that is neither a per-call nor a tools/list authorization. Guessing would add or drop a decision.

ReadOnlyViolation

ReadOnlyViolation(message, *, operation=None)

Bases: AgentCoreError

A read-only command tried an AWS operation outside its allowlist; refused before sending.

operation is the refused call, (service, operation), or None when not named.

Source code in src/calibrated_agent_evaluation/agentcore/errors.py
def __init__(self, message: str, *, operation: tuple[str, str] | None = None) -> None:
    super().__init__(message)
    self.operation = operation

operation instance-attribute

operation = operation

SessionConversionError

Bases: DetectorError

A trace cannot be converted into the session the failure detector reads.

VocabularyParseError

VocabularyParseError(row, check)

Bases: DetectorError

The detector's category table has a row the library cannot read as a vocabulary entry.

row is the number of the table row that failed and check names the check it failed.

Source code in src/calibrated_agent_evaluation/agentcore/errors.py
def __init__(self, row: int, check: str) -> None:
    self.row = row
    self.check = check
    super().__init__(
        f"the failure detector's category table cannot be read: row {row} fails the check "
        f"'{check}'; the installed strands-agents-evals prompt template may have changed"
    )

row instance-attribute

row = row

check instance-attribute

check = check

WrongAwsIdentity

Bases: AgentCoreError

The credentials belong to a different principal than the one required.

ArmRead dataclass

ArmRead(read_id, assignments, counts=dict(), arns_seen=())

One arms read: its id, one assignment per source id, its counts and the ARNs it saw.

read_id instance-attribute

read_id

assignments instance-attribute

assignments

counts class-attribute instance-attribute

counts = field(default_factory=dict)

arns_seen class-attribute instance-attribute

arns_seen = ()

KeptTrace dataclass

KeptTrace(
    trace_id,
    session_ids,
    rekey_target,
    spans_total,
    spans_stamped,
    spans_half_stamped,
    log_groups=(),
    content_missing=False,
    experiment_arns=(),
)

What RawTraceLog keeps of one raw trace the ingest's reader yielded.

trace_id instance-attribute

trace_id

session_ids instance-attribute

session_ids

rekey_target instance-attribute

rekey_target

spans_total instance-attribute

spans_total

spans_stamped instance-attribute

spans_stamped

spans_half_stamped instance-attribute

spans_half_stamped

log_groups class-attribute instance-attribute

log_groups = ()

content_missing class-attribute instance-attribute

content_missing = False

experiment_arns class-attribute instance-attribute

experiment_arns = ()

RawTraceLog

RawTraceLog(reader)

Wraps one of the trace ingest's readers: each raw trace passes through, its counts kept.

The ingest's source drops a trace it refuses for a re-key collision, so the arm read takes what it needs from here: each trace's id, session ids, re-key target and span counts, and the log groups its spans name.

Source code in src/calibrated_agent_evaluation/agentcore/arms.py
def __init__(self, reader: AgentCoreReader) -> None:
    self._reader = reader
    self.kept: list[KeptTrace] = []
    self._stripped: dict[str, dict[str, Any]] = {}

kept instance-attribute

kept = []

read_traces

read_traces()

Yield the wrapped reader's traces unchanged, keeping what the arm read needs.

Source code in src/calibrated_agent_evaluation/agentcore/arms.py
def read_traces(self) -> Iterator[RawTrace]:
    """Yield the wrapped reader's traces unchanged, keeping what the arm read needs."""
    from calibrated_agent_evaluation.agentcore.adot import adot_to_otlp
    from calibrated_agent_evaluation.agentcore.errors import AgentCoreContentMissing

    for raw in self._reader.read_traces():
        kept = self._keep(raw)
        try:
            adot_to_otlp(raw.span_records, raw.log_records)
        except AgentCoreContentMissing:
            # The ingest's source refuses this trace for its content; its spans still carry
            # the arm stamps, so they are kept, without content, for the arm read.
            kept = replace(kept, content_missing=True)
            self._stripped[raw.trace_id] = {
                "resourceSpans": [
                    {
                        "resource": {"attributes": {}},
                        "scopeSpans": [{"spans": [dict(s) for s in raw.span_records]}],
                    }
                ]
            }
        self.kept.append(kept)
        yield raw

read_scores

read_scores()

The wrapped reader's evaluator results, unchanged.

Source code in src/calibrated_agent_evaluation/agentcore/arms.py
def read_scores(self) -> Iterator[dict[str, Any]]:
    """The wrapped reader's evaluator results, unchanged."""
    return self._reader.read_scores()

collisions

collisions(rekeyed)

Per runtime session id two or more traces re-keyed to, every trace that did.

A mixed-session trace absent from rekeyed was refused by the ingest because an earlier trace already took its runtime session id; the trace the ingest delivered under that id is the one rekeyed maps to it. No error text is read.

Source code in src/calibrated_agent_evaluation/agentcore/arms.py
def collisions(self, rekeyed: Sequence[tuple[str, str]]) -> dict[str, tuple[KeptTrace, ...]]:
    """Per runtime session id two or more traces re-keyed to, every trace that did.

    A mixed-session trace absent from ``rekeyed`` was refused by the ingest because an earlier
    trace already took its runtime session id; the trace the ingest delivered under that id is
    the one ``rekeyed`` maps to it. No error text is read.
    """
    rekeyed_ids = {old for old, _ in rekeyed}
    delivered_under = {new: old for old, new in rekeyed}
    by_id = {k.trace_id: k for k in self.kept}
    out: dict[str, list[KeptTrace]] = {}
    for kept in self.kept:
        if len(kept.session_ids) <= 1 or kept.trace_id in rekeyed_ids:
            continue
        target = kept.rekey_target
        winner = delivered_under.get(target)
        entry = out.setdefault(target, [])
        if winner is not None and winner in by_id and by_id[winner] not in entry:
            entry.append(by_id[winner])
        entry.append(kept)
    return {t: tuple(sorted(v, key=lambda k: k.trace_id)) for t, v in out.items()}

contentless

contentless(rekeyed=())

Per trace id, the spans of each trace whose message content did not come back.

A trace whose spans name two session ids, and which the source did not re-key, is left out: the ingest refuses it for its sessions whatever its content, so its arm stays unread.

Source code in src/calibrated_agent_evaluation/agentcore/arms.py
def contentless(self, rekeyed: Sequence[tuple[str, str]] = ()) -> dict[str, dict[str, Any]]:
    """Per trace id, the spans of each trace whose message content did not come back.

    A trace whose spans name two session ids, and which the source did not re-key, is left
    out: the ingest refuses it for its sessions whatever its content, so its arm stays unread.
    """
    rekeyed_ids = {old for old, _ in rekeyed}
    sessions = {k.trace_id: k.session_ids for k in self.kept}
    return {
        trace_id: doc
        for trace_id, doc in self._stripped.items()
        if len(sessions.get(trace_id, ())) <= 1 or trace_id in rekeyed_ids
    }

groups_read

groups_read()

aws/spans and the log groups the spans read so far name, sorted.

Source code in src/calibrated_agent_evaluation/agentcore/arms.py
def groups_read(self) -> tuple[str, ...]:
    """``aws/spans`` and the log groups the spans read so far name, sorted."""
    return tuple(sorted({_SPANS_LOG_GROUP, *(g for k in self.kept for g in k.log_groups)}))

AbTestClaimRecord pydantic-model

Bases: _AuditModel

What AgentCore said about an A/B test when it was read: a vendor claim, not evidence.

Fields:

Validators:

record_id pydantic-field

record_id

claim_kind pydantic-field

claim_kind = 'vendor-claim'

ab_test_arn pydantic-field

ab_test_arn

ab_test_id pydantic-field

ab_test_id

name pydantic-field

name

status pydantic-field

status

execution_status pydantic-field

execution_status

gateway_arn pydantic-field

gateway_arn

variants pydantic-field

variants

evaluation_config_arns pydantic-field

evaluation_config_arns

current_run_id pydantic-field

current_run_id

error_details pydantic-field

error_details

started_at pydantic-field

started_at

stopped_at pydantic-field

stopped_at

max_duration_expires_at pydantic-field

max_duration_expires_at

analysis_timestamp pydantic-field

analysis_timestamp

updated_at pydantic-field

updated_at

evaluator_claims pydantic-field

evaluator_claims

identical_arms pydantic-field

identical_arms

provisional pydantic-field

provisional

read_at pydantic-field

read_at

schema_version pydantic-field

schema_version = AUDIT_SCHEMA_VERSION

provenance pydantic-field

provenance

ArmAssignment pydantic-model

Bases: _AuditModel

Which arm of the audited A/B test one delivered trace ran in, or why it has none.

Fields:

Validators:

trace_id pydantic-field

trace_id

otlp_trace_ids pydantic-field

otlp_trace_ids

read_id pydantic-field

read_id

runtime_session_id pydantic-field

runtime_session_id

span_session_ids pydantic-field

span_session_ids

ab_test_arn pydantic-field

ab_test_arn

runtime pydantic-field

runtime

window_start pydantic-field

window_start

window_end pydantic-field

window_end

omni pydantic-field

omni

rekey_sessions pydantic-field

rekey_sessions

shared_id pydantic-field

shared_id = None

status pydantic-field

status

experiment_arn pydantic-field

experiment_arn

variant pydantic-field

variant

spans_total pydantic-field

spans_total

spans_stamped pydantic-field

spans_stamped

spans_half_stamped pydantic-field

spans_half_stamped = 0

applied_bundle_attribute pydantic-field

applied_bundle_attribute = None

applied_bundle_require pydantic-field

applied_bundle_require

applied_bundle_values pydantic-field

applied_bundle_values

reason pydantic-field

reason = None

content_missing pydantic-field

content_missing = False

schema_version pydantic-field

schema_version = AUDIT_SCHEMA_VERSION

BundleVersionRef pydantic-model

Bases: _AuditModel

One configuration bundle version, as read, with its components digested.

Fields:

Validators:

bundle_arn pydantic-field

bundle_arn

version_id pydantic-field

version_id

version_created_at pydantic-field

version_created_at

parent_version_ids pydantic-field

parent_version_ids

branch_name pydantic-field

branch_name

created_by pydantic-field

created_by

components pydantic-field

components

components_sha256 pydantic-field

components_sha256

CodeInterpreterSandbox

CodeInterpreterSandbox(
    *,
    region,
    clients=None,
    cassette=None,
    budget=None,
    role_arn=None,
    identifier=_DEFAULT_IDENTIFIER,
    execute=None,
)

A SandboxBackend backed by Bedrock AgentCore Code Interpreter.

Policy refusals (billed-call budget, wrong identity) surface as their own AgentCoreError subclasses; an execution/response failure surfaces as SandboxError (the seam's contract, which the execution scorer re-wraps into ScorerError).

Source code in src/calibrated_agent_evaluation/agentcore/code_interpreter.py
def __init__(
    self,
    *,
    region: str,
    clients: AwsClients | None = None,
    cassette: Cassette | None = None,
    budget: BilledCallBudget | None = None,
    role_arn: str | None = None,
    identifier: str = _DEFAULT_IDENTIFIER,
    execute: ExecuteFn | None = None,
) -> None:
    self._region = region
    self._clients = clients
    self._cassette = cassette
    self._budget = budget
    self._role_arn = role_arn
    self._identifier = identifier
    self._execute = execute
    self._model = model_stamp(f"agentcore-code-interpreter:{identifier}", {})

run_code

run_code(code, *, language='python', timeout_s, files=None)

Run code in a managed Code Interpreter session and return a normalized result.

Raises:

Type Description
WrongAwsIdentity

a role is required and the caller is not it (pre-flight).

BilledCallBudgetExceeded

the billed-call budget is used up (before the call is sent).

SandboxError

the language is unsupported, seeded files were asked for, the call failed, the response was unparseable, or (REPLAY) nothing was recorded for it.

Source code in src/calibrated_agent_evaluation/agentcore/code_interpreter.py
def run_code(
    self,
    code: str,
    *,
    language: str = "python",
    # The caller (the execution scorer's harness) enforces the real timeout in-process, so it
    # behaves the same here as on the local sandbox; the managed session's own timeout is only a
    # backstop.
    timeout_s: float,  # noqa: ARG002
    files: dict[str, str] | None = None,
) -> SandboxResult:
    """Run ``code`` in a managed Code Interpreter session and return a normalized result.

    Raises:
        WrongAwsIdentity: a role is required and the caller is not it (pre-flight).
        BilledCallBudgetExceeded: the billed-call budget is used up (before the call is sent).
        SandboxError: the language is unsupported, seeded files were asked for, the call failed,
            the response was unparseable, or (REPLAY) nothing was recorded for it.
    """
    if language != "python":
        # Validate up front, before a session is started or the budget is charged, so an
        # unsupported language is a typed refusal here rather than a bare ValueError escaping
        # from deep in the SDK after we have already paid for the call.
        raise SandboxError(
            f"the Code Interpreter sandbox runs Python only in v1, got language={language!r}"
        )
    if files:
        # The seam declares SandboxError for run-failures; a capability gap is one of those.
        raise SandboxError(
            "the Code Interpreter sandbox does not support seeded files in v1; inline the "
            "inputs into the code"
        )
    if self._role_arn is not None:
        if self._clients is None:
            raise AgentCoreError(
                "a role_arn was set but no AwsClients was provided, so the identity cannot be "
                "checked; pass clients= or drop role_arn"
            )
        require_role(self._clients, self._role_arn)
    request = {
        "code": code,
        "language": language,
        "clearContext": True,
        "identifier": self._identifier,
    }
    normalized = self._dispatch(request, code, language)
    return _to_result(normalized)

DatasetPlan dataclass

DatasetPlan(
    name,
    region,
    schema_type,
    n_scenarios,
    n_with_assertions,
    n_with_trajectory,
    digest,
    existing_id,
    next_version,
    unchanged,
)

What a publish would do, worked out from reads only.

name instance-attribute

name

region instance-attribute

region

schema_type instance-attribute

schema_type

n_scenarios instance-attribute

n_scenarios

n_with_assertions instance-attribute

n_with_assertions

n_with_trajectory instance-attribute

n_with_trajectory

digest instance-attribute

digest

existing_id instance-attribute

existing_id

next_version instance-attribute

next_version

unchanged instance-attribute

unchanged

creates_dataset property

creates_dataset

Whether the publish creates a new dataset rather than versioning an existing one.

FailureDetector

FailureDetector(
    model,
    *,
    model_id=None,
    confidence_threshold="low",
    cassette_path=None,
    cassette_mode=None,
    max_live_calls=0,
    now=None,
)

Detects failures in normalized traces with the strands-agents-evals failure detector.

The model stack is built and checked once, when the detector is built, so one live-call cap and one cassette serve every detection it makes, and one cassette can hold the calls for many traces. Each detection makes one model call, with the library's own prompt; the cassette records and replays it under that prompt. Detections on one detector run one at a time.

Parameters:

Name Type Description Default
model Model

The strands Model to call, for example detector_bedrock_model("global.anthropic.claude-sonnet-4-6"). None and a model id string are refused, because the library would build a Bedrock model of its own from them.

required
model_id str | None

The model id, when the model's configuration has none; if both are given they must be equal. It is recorded on every detection and cassette entry.

None
confidence_threshold Confidence

The lowest confidence a reported failure keeps: "low", "medium" or "high". Failures below it are counted, not returned.

'low'
cassette_path str | Path | None

A cassette file that records and replays the model calls, or None to make every call live.

None
cassette_mode Mode | None

The cassette mode; None reads CALIBRATED_AGENT_EVALUATION_CASSETTE_MODE (default auto).

None
max_live_calls int

How many live model calls this detector may make over its lifetime. A cassette replay is not a live call.

0
now datetime | None

The time every detection's provenance stamp records, for reproducible results; it must be timezone-aware. None reads the clock at each detection.

None

Raises:

Type Description
DetectorConfigError

The model, the model id, the cassette, the cassette mode, the threshold, the live-call limit or now cannot be used. Nothing is called.

Source code in src/calibrated_agent_evaluation/agentcore/detectors/detect.py
def __init__(
    self,
    model: Model,
    *,
    model_id: str | None = None,
    confidence_threshold: Confidence = "low",
    cassette_path: str | Path | None = None,
    cassette_mode: Mode | None = None,
    max_live_calls: int = 0,
    now: datetime | None = None,
) -> None:
    if confidence_threshold is None:
        # The shared check reads a missing threshold as "no threshold", which a detector needs.
        raise DetectorConfigError(
            "confidence_threshold=None is not a confidence level: use 'low', 'medium' or "
            "'high', in lower case"
        )
    self._prepared = prepare_model(
        model,
        model_id=model_id,
        cassette_path=cassette_path,
        cassette_mode=cassette_mode,
        max_live_calls=max_live_calls,
        confidence_threshold=confidence_threshold,
    )
    if now is not None and (not isinstance(now, datetime) or now.utcoffset() is None):
        # Checked here, because the stamp is taken after the model call has been paid for.
        raise DetectorConfigError(
            f"now={now!r} is not a timezone-aware datetime, so the detections' stamps could "
            "not record when they ran: pass one with a timezone, such as "
            "datetime.now(tz=UTC), or leave now out"
        )
    self._threshold: Confidence = confidence_threshold
    self._now = now
    # The capturing wrapper holds one detection's prompt and answer at a time.
    self._lock = threading.Lock()

detect

detect(trace)

Detect the failures in one trace with one model call.

Parameters:

Name Type Description Default
trace Trace

A normalized, owned trace with spans.

required

Returns:

Type Description
Detection

One failure for each category the model named at each span or tool call, at or

Detection

above the confidence threshold, in the converted session's span order and then the

Detection

answer's order; a category named twice at one location is kept twice. The failures

Detection

the threshold dropped are counted in below_threshold.

Raises:

Type Description
SessionConversionError

The trace cannot be converted for the detector.

DetectorSessionTooLarge

The trace is too large for one model call, by the library's own estimate before any call or by the model's own refusal.

DetectorCallLimitExceeded

The call would exceed max_live_calls; it was not sent.

DetectorCallError

The model, the cassette or the library failed, before or during the call; or the call could not be prepared, because the installed template is unusable or the trace could not be serialized or sized, and was not made. The original error is chained.

DetectorConfigError

The model's config now names another model id than the one the detector was built for, or cannot be read; nothing was sent.

DetectorOutputError

The library returned without its model call, or the model's answer cannot be read faithfully: not one JSON object with an errors key, a malformed entry, an unknown category or location, or a reading the library's own parse disagrees with.

Source code in src/calibrated_agent_evaluation/agentcore/detectors/detect.py
def detect(self, trace: Trace) -> Detection:
    """Detect the failures in one trace with one model call.

    Args:
        trace: A normalized, owned trace with spans.

    Returns:
        One failure for each category the model named at each span or tool call, at or
        above the confidence threshold, in the converted session's span order and then the
        answer's order; a category named twice at one location is kept twice. The failures
        the threshold dropped are counted in ``below_threshold``.

    Raises:
        SessionConversionError: The trace cannot be converted for the detector.
        DetectorSessionTooLarge: The trace is too large for one model call, by the library's
            own estimate before any call or by the model's own refusal.
        DetectorCallLimitExceeded: The call would exceed ``max_live_calls``; it was not sent.
        DetectorCallError: The model, the cassette or the library failed, before or during
            the call; or the call could not be prepared, because the installed template is
            unusable or the trace could not be serialized or sized, and was not made. The
            original error is chained.
        DetectorConfigError: The model's config now names another model id than the one the
            detector was built for, or cannot be read; nothing was sent.
        DetectorOutputError: The library returned without its model call, or the model's
            answer cannot be read faithfully: not one JSON object with an ``errors`` key, a
            malformed entry, an unknown category or location, or a reading the library's own
            parse disagrees with.
    """
    with self._lock:
        return self._detect(trace)

DetectorDraftNamer

DetectorDraftNamer(
    model,
    vocabulary,
    *,
    model_id=None,
    max_critiques_per_cluster=_DEFAULT_MAX_CRITIQUES,
    cassette_path=None,
    cassette_mode=None,
    max_live_calls=0,
    now=None,
)

Names draft clusters from the detector vocabulary, one model call per naming.

A DraftNamer for build_draft, which it satisfies structurally. The configuration is checked when the namer is built, before any model call. The model is always the one passed: None and a model id string are refused, because the detector library would build a Bedrock model of its own from them.

Parameters:

Name Type Description Default
model Model

The strands Model that names the clusters, for example detector_bedrock_model(model_id).

required
vocabulary FailureVocabulary

The names the clusters are named from, usually failure_vocabulary().

required
model_id str | None

The model id, when the model's config has none; if both are given they must be equal. Every naming result and cassette entry records it.

None
max_critiques_per_cluster int

How many of a cluster's critiques the prompt shows, its first ones in member order; the prompt states how many were shown out of how many.

_DEFAULT_MAX_CRITIQUES
cassette_path str | Path | None

The cassette file that records and replays the naming calls, or None to make every call live.

None
cassette_mode Mode | None

The cassette mode; None reads CALIBRATED_AGENT_EVALUATION_CASSETTE_MODE (default auto).

None
max_live_calls int

How many live model calls the namer may make over its lifetime. The default 0 allows none, so only cassette replays run.

0
now datetime | None

The time every naming result's provenance stamp records, for tests; None reads the clock at each call. It must be timezone-aware.

None

Attributes:

Name Type Description
results list[NamingResult]

One NamingResult per naming call that returned names, in call order.

Raises:

Type Description
DetectorConfigError

The model is missing, a string or not a strands Model; no model id can be resolved, or the passed one differs from the config's; the cassette cannot be loaded, or is missing in replay mode; CALIBRATED_AGENT_EVALUATION_CASSETTE_MODE names no mode; max_live_calls is negative; max_critiques_per_cluster is below 1; the vocabulary is empty or not a FailureVocabulary; or now is naive.

AgentCoreError

The installed detector package reports no version.

Source code in src/calibrated_agent_evaluation/agentcore/detectors/namer.py
def __init__(
    self,
    model: Model,
    vocabulary: FailureVocabulary,
    *,
    model_id: str | None = None,
    max_critiques_per_cluster: int = _DEFAULT_MAX_CRITIQUES,
    cassette_path: str | Path | None = None,
    cassette_mode: Mode | None = None,
    max_live_calls: int = 0,
    now: datetime | None = None,
) -> None:
    _check_cap(max_critiques_per_cluster)
    _check_vocabulary(vocabulary)
    _check_now(now)
    self._prepared = prepare_model(
        model,
        model_id=model_id,
        cassette_path=cassette_path,
        cassette_mode=cassette_mode,
        max_live_calls=max_live_calls,
    )
    # Read once here, so a broken install is refused before any billed call.
    self._template = template_provenance()
    self._vocabulary = vocabulary
    self._known = frozenset(vocabulary.names())
    self._max_critiques = max_critiques_per_cluster
    self._now = now
    # The model stack allows one call per arm, so one naming runs at a time.
    self._lock = threading.Lock()
    self.results: list[NamingResult] = []

results instance-attribute

results = []

name_clusters

name_clusters(clusters)

Name each cluster from the vocabulary, with one model call.

Parameters:

Name Type Description Default
clusters list[list[str]]

The member critiques of each cluster, in cluster order.

required

Returns:

Type Description
list[str]

One name per cluster, in cluster order: a vocabulary name, with #2, #3

list[str]

appended to the second and later clusters given the same name. The call's

list[str]

NamingResult is appended

list[str]

to results.

Raises:

Type Description
DraftNamingError

clusters is empty or holds an empty cluster (refused before any call), or the response holds no JSON array of exactly one vocabulary name per cluster.

DetectorCallLimitExceeded

The call would be a live call past max_live_calls; it was not sent.

DetectorCallError

The model or the cassette failed; the original error is chained.

DetectorConfigError

The model's config now names another model id than the one the namer was built for, or cannot be read; nothing was sent.

Source code in src/calibrated_agent_evaluation/agentcore/detectors/namer.py
def name_clusters(self, clusters: list[list[str]]) -> list[str]:
    """Name each cluster from the vocabulary, with one model call.

    Args:
        clusters: The member critiques of each cluster, in cluster order.

    Returns:
        One name per cluster, in cluster order: a vocabulary name, with `` #2``, `` #3``
        appended to the second and later clusters given the same name. The call's
        :class:`~calibrated_agent_evaluation.agentcore.detectors.types.NamingResult` is appended
        to :attr:`results`.

    Raises:
        DraftNamingError: ``clusters`` is empty or holds an empty cluster (refused before
            any call), or the response holds no JSON array of exactly one vocabulary name
            per cluster.
        DetectorCallLimitExceeded: The call would be a live call past ``max_live_calls``;
            it was not sent.
        DetectorCallError: The model or the cassette failed; the original error is chained.
        DetectorConfigError: The model's config now names another model id than the one the
            namer was built for, or cannot be read; nothing was sent.
    """
    _check_clusters(clusters)
    shown = tuple(
        (min(len(cluster), self._max_critiques), len(cluster)) for cluster in clusters
    )
    prompt = _render_prompt(self._vocabulary, clusters, self._max_critiques)
    messages: Messages = [{"role": "user", "content": [{"text": prompt}]}]
    with self._lock:
        # Read before the call, so the result records the configuration it was made with.
        config_sent = self._prepared.config_sent()
        stack = self._prepared.model
        stack.arm(prompt)
        try:
            text = run_stream(stack, messages)
        except Exception as exc:
            error = call_error(exc)
            if error is exc:
                raise
            raise error from error.__cause__
        names = _suffix_repeats(_parse_names(text, known=self._known, count=len(clusters)))
        self.results.append(
            NamingResult(
                names=names,
                shown=shown,
                namer_prompt_version=NAMER_PROMPT_VERSION,
                namer_prompt_sha256=NAMER_PROMPT_SHA256,
                max_critiques_per_cluster=self._max_critiques,
                provenance=self._provenance(config_sent),
            )
        )
    return list(names)

DetectedFailure pydantic-model

Bases: CalibratedAgentEvaluationModel

One category the detector named at one location, with its evidence.

span_id is the Calibrated Agent Evaluation span the failure is on. tool_call_index and tool_name are set when the detector pointed at one of that span's tool calls. A category the detector named twice at one location appears twice, each with its own evidence.

Config:

  • frozen: True
  • extra: forbid

Fields:

span_id pydantic-field

span_id

tool_call_index pydantic-field

tool_call_index

tool_name pydantic-field

tool_name

category pydantic-field

category

parent pydantic-field

parent

confidence pydantic-field

confidence

evidence pydantic-field

evidence

Detection pydantic-model

Bases: CalibratedAgentEvaluationModel

The detector's findings on one trace.

trace_sha256 is the SHA-256 of the canonical JSON of the trace that was detected. times says whether the spans' recorded times were sent or placeholders stood in for them. failures holds the failures at or above the confidence threshold, in span order and then response order; below_threshold counts the ones the threshold dropped. No failures with below_threshold 0 means the detector found none.

Config:

  • frozen: True
  • extra: forbid

Fields:

trace_id pydantic-field

trace_id

trace_sha256 pydantic-field

trace_sha256

times pydantic-field

times

failures pydantic-field

failures

below_threshold pydantic-field

below_threshold

provenance pydantic-field

provenance

oracle_view

oracle_view()

The detection as JSON, without the values that describe the run's environment.

Returns:

Type Description
dict[str, Any]

model_dump(mode="json") without the stamp's run_at, git_sha,

dict[str, Any]

calibrated_agent_evaluation_version and python_version, and without the

dict[str, Any]

numeric-library versions capture_provenance records (STATS_DISTRIBUTIONS). Every

dict[str, Any]

other field is kept.

Source code in src/calibrated_agent_evaluation/agentcore/detectors/types.py
def oracle_view(self) -> dict[str, Any]:
    """The detection as JSON, without the values that describe the run's environment.

    Returns:
        ``model_dump(mode="json")`` without the stamp's ``run_at``, ``git_sha``,
        ``calibrated_agent_evaluation_version`` and ``python_version``, and without the
        numeric-library versions ``capture_provenance`` records (``STATS_DISTRIBUTIONS``). Every
        other field is kept.
    """
    return _without_run_environment(self)

DetectorProvenance pydantic-model

Bases: CalibratedAgentEvaluationModel

What produced a detection or a set of cluster names.

model_config_sent is the model's config without its model_id, as sent. confidence_threshold is the threshold the detection was filtered at, None for cluster names. stamp.dep_versions records the strands-agents-evals version beside the numeric libraries'.

Config:

  • frozen: True
  • extra: forbid
  • protected_namespaces: ()

Fields:

detector_package pydantic-field

detector_package

detector_version pydantic-field

detector_version

prompt_template pydantic-field

prompt_template

prompt_sha256 pydantic-field

prompt_sha256

model_id pydantic-field

model_id

model_config_sent pydantic-field

model_config_sent

confidence_threshold pydantic-field

confidence_threshold

stamp pydantic-field

stamp

FailureVocabulary pydantic-model

Bases: CalibratedAgentEvaluationModel

The detector's categories in table order, then other, and the template they came from.

detector_version is the installed strands-agents-evals version, prompt_template the template version and prompt_sha256 the SHA-256 of the prompt the detector sends for an empty session.

Config:

  • frozen: True
  • extra: forbid

Fields:

entries pydantic-field

entries

detector_version pydantic-field

detector_version

prompt_template pydantic-field

prompt_template

prompt_sha256 pydantic-field

prompt_sha256

names

names()

The category names in table order, other last.

Returns:

Type Description
tuple[str, ...]

One name per entry.

Source code in src/calibrated_agent_evaluation/agentcore/detectors/types.py
def names(self) -> tuple[str, ...]:
    """The category names in table order, ``other`` last.

    Returns:
        One name per entry.
    """
    return tuple(entry.name for entry in self.entries)

descriptions

descriptions()

Each category's definition by name, as cluster.ratify(descriptions=...) takes them.

Returns:

Type Description
dict[str, str]

A mapping from category name to definition.

Source code in src/calibrated_agent_evaluation/agentcore/detectors/types.py
def descriptions(self) -> dict[str, str]:
    """Each category's definition by name, as ``cluster.ratify(descriptions=...)`` takes them.

    Returns:
        A mapping from category name to definition.
    """
    return {entry.name: entry.definition for entry in self.entries}

NamingResult pydantic-model

Bases: CalibratedAgentEvaluationModel

The names one draft-naming call gave a set of clusters.

names holds one name per cluster, a repeated name suffixed #2, #3. shown holds, per cluster, how many critiques the prompt showed and how many the cluster has, at most max_critiques_per_cluster shown. namer_prompt_sha256 hashes the part of the namer prompt that holds no cluster data.

Config:

  • frozen: True
  • extra: forbid

Fields:

names pydantic-field

names

shown pydantic-field

shown

namer_prompt_version pydantic-field

namer_prompt_version

namer_prompt_sha256 pydantic-field

namer_prompt_sha256

max_critiques_per_cluster pydantic-field

max_critiques_per_cluster

provenance pydantic-field

provenance

oracle_view

oracle_view()

The naming result as JSON, without the values that describe the run's environment.

Returns:

Type Description
dict[str, Any]

model_dump(mode="json") without the stamp's run_at, git_sha,

dict[str, Any]

calibrated_agent_evaluation_version and python_version, and without the

dict[str, Any]

numeric-library versions capture_provenance records (STATS_DISTRIBUTIONS). Every

dict[str, Any]

other field is kept.

Source code in src/calibrated_agent_evaluation/agentcore/detectors/types.py
def oracle_view(self) -> dict[str, Any]:
    """The naming result as JSON, without the values that describe the run's environment.

    Returns:
        ``model_dump(mode="json")`` without the stamp's ``run_at``, ``git_sha``,
        ``calibrated_agent_evaluation_version`` and ``python_version``, and without the
        numeric-library versions ``capture_provenance`` records (``STATS_DISTRIBUTIONS``). Every
        other field is kept.
    """
    return _without_run_environment(self)

VocabularyEntry pydantic-model

Bases: CalibratedAgentEvaluationModel

One failure category the detector may name.

parent is the category's group with its bold markup removed, None only for other. definition and examples are the table's cells with their markup (such as <br>) kept as written; examples is empty when the table gives none, and for other, whose definition is the template's guideline for using it.

Config:

  • frozen: True
  • extra: forbid

Fields:

name pydantic-field

name

parent pydantic-field

parent

definition pydantic-field

definition

examples pydantic-field

examples

DoctorTarget pydantic-model

Bases: _DoctorModel

The account slice to check, from the CLI flags.

Fields:

region pydantic-field

region

profile pydantic-field

profile = None

runtime_name pydantic-field

runtime_name = None

omni_domain pydantic-field

omni_domain = None

BilledCallBudget

BilledCallBudget(limit=None)

Counts billed operations and refuses the first one past limit; no limit when None.

Source code in src/calibrated_agent_evaluation/agentcore/guards.py
def __init__(self, limit: int | None = None) -> None:
    if limit is not None and limit < 0:
        raise ValueError(f"a billed-call budget can't be negative (got {limit})")
    self.limit = limit
    self.used = 0
    # Charged from every thread a run sends calls on, so the check and the count are one step.
    self._lock = threading.Lock()

limit instance-attribute

limit = limit

used instance-attribute

used = 0

remaining property

remaining

Billed calls left before the limit, or None with no limit; charges nothing.

from_env classmethod

from_env(environ)

The budget the environment sets; no limit when unset.

The variable is CALIBRATED_AGENT_EVALUATION_AGENTCORE_MAX_BILLED_CALLS.

Source code in src/calibrated_agent_evaluation/agentcore/guards.py
@classmethod
def from_env(cls, environ: Mapping[str, str]) -> BilledCallBudget:
    """The budget the environment sets; no limit when unset.

    The variable is ``CALIBRATED_AGENT_EVALUATION_AGENTCORE_MAX_BILLED_CALLS``.
    """
    raw = environ.get(BUDGET_ENV_VAR, "").strip()
    if not raw:
        return cls()
    try:
        return cls(int(raw))
    except ValueError as exc:
        raise AgentCoreError(
            f"{BUDGET_ENV_VAR}={raw!r} is not a non-negative whole number"
        ) from exc

charge

charge(operation)

Count one operation against the budget, or refuse it past the limit.

The explicit entry point for callers that do not go through a botocore before-call handler (an adapter that builds its own client, e.g. the Code Interpreter sandbox). A non-billed operation is ignored; a billed one past limit raises before it is sent.

Source code in src/calibrated_agent_evaluation/agentcore/guards.py
def charge(self, operation: Operation) -> None:
    """Count one ``operation`` against the budget, or refuse it past the limit.

    The explicit entry point for callers that do not go through a botocore ``before-call``
    handler (an adapter that builds its own client, e.g. the Code Interpreter sandbox). A
    non-billed operation is ignored; a billed one past ``limit`` raises before it is sent.
    """
    if operation not in BILLED_OPERATIONS:
        return
    with self._lock:
        if self.limit is not None and self.used >= self.limit:
            service, name = operation
            raise BilledCallBudgetExceeded(
                f"refused {service}:{name}: the billed-call budget of {self.limit} is used up "
                "(raise CALIBRATED_AGENT_EVALUATION_AGENTCORE_MAX_BILLED_CALLS or unset it to "
                "remove the limit)",
                operation=operation,
            )
        self.used += 1

ReadOnlyGuard

ReadOnlyGuard(allowed)

Refuses every operation not in allowed.

Source code in src/calibrated_agent_evaluation/agentcore/guards.py
def __init__(self, allowed: Iterable[Operation]) -> None:
    self.allowed = frozenset(allowed)

allowed instance-attribute

allowed = frozenset(allowed)

AgentCorePolicyDecisionSource

AgentCorePolicyDecisionSource(
    clients,
    *,
    run_records,
    settle_deadline_seconds=None,
    poll_interval_seconds=POLL_INTERVAL_SECONDS,
    clock=None,
    sleep=None,
    monotonic=None,
)

Reads each trial's Policy decisions from aws/spans by the trial's W3C trace ID.

run_records is read when the decisions are read, never when the source is built: it maps each trial's probe trace ID to the run that sent it. Each poll reads every trial not yet settled through the Logs Insights index and straight from the log store. A trial settles when the two hold the same spans on two reads in a row, and is not read again. A trial with no span in either may simply not have reached the store yet, so it settles only at the bound, and then counts as a trial with no decision. The wait is bounded by settle_deadline_seconds, by default settle_deadline_for the number of trials; a trial whose last read before the bound matched still gets its confirming read. Every read is read-only, and none is billed.

Source code in src/calibrated_agent_evaluation/agentcore/policy_spans.py
def __init__(
    self,
    clients: AwsClients,
    *,
    run_records: Callable[[], Sequence[RunRecord]],
    settle_deadline_seconds: float | None = None,
    poll_interval_seconds: float = POLL_INTERVAL_SECONDS,
    clock: Callable[[], datetime] | None = None,
    sleep: Callable[[float], None] | None = None,
    monotonic: Callable[[], float] | None = None,
) -> None:
    if poll_interval_seconds <= 0 or (
        settle_deadline_seconds is not None and settle_deadline_seconds < poll_interval_seconds
    ):
        raise AgentCoreError(
            f"poll_interval_seconds must be > 0 and settle_deadline_seconds at least that long "
            f"(got {poll_interval_seconds:g}s and {settle_deadline_seconds}s)"
        )
    self._clients = clients
    self._run_records = run_records
    self._deadline = settle_deadline_seconds
    self._interval = poll_interval_seconds
    self._clock = clock or (lambda: datetime.now(tz=UTC))
    # Looked up when called, so a test that replaces ``time.sleep`` still reaches it.
    self._sleep = sleep or (lambda seconds: time.sleep(seconds))
    self._monotonic = monotonic or (lambda: time.monotonic())

check_readable

check_readable()

One read-only Logs Insights query on aws/spans over the last second.

A truncated result counts as readable, since the query ran; any other failure, a missing logs:StartQuery permission among them, is raised before any trial is sent.

Source code in src/calibrated_agent_evaluation/agentcore/policy_spans.py
def check_readable(self) -> None:
    """One read-only Logs Insights query on ``aws/spans`` over the last second.

    A truncated result counts as readable, since the query ran; any other failure, a missing
    ``logs:StartQuery`` permission among them, is raised before any trial is sent.
    """
    end = self._clock()
    window = TimeWindow(start=(end - timedelta(seconds=1)).isoformat(), end=end.isoformat())
    config = AgentCoreSourceConfig(runtime="runtime", window=window)
    try:
        run_logs_insights(self._clients.client(_LOGS_SERVICE), _SPANS_LOG_GROUP, config)
    except AgentCoreTruncatedResult:
        return

read_decisions

read_decisions(trials)

One answer per trial, keyed by probe trace ID, once every trial's spans have settled.

Raises:

Type Description
ProbeError

A trial has no run record, or its record's W3C trace ID or times are unreadable.

AgentCoreSettlingLag

Some trial's spans were still arriving at the deadline.

PolicySpanMalformed

A decision span can't be read whole.

Source code in src/calibrated_agent_evaluation/agentcore/policy_spans.py
def read_decisions(self, trials: Sequence[TrialRef]) -> Mapping[str, TrialDecisions]:
    """One answer per trial, keyed by probe trace ID, once every trial's spans have settled.

    Raises:
        ProbeError: A trial has no run record, or its record's W3C trace ID or times are
            unreadable.
        AgentCoreSettlingLag: Some trial's spans were still arriving at the deadline.
        PolicySpanMalformed: A decision span can't be read whole.
    """
    records = {r.trace_id: r for r in self._run_records()}
    runs: list[tuple[TrialRef, RunRecord, AgentCoreSourceConfig]] = []
    # Each trial's W3C trace ID as validated and normalized once, used for every read and for
    # its answer, so the reads and the extracted decisions name it the same way.
    w3c_ids: dict[str, str] = {}
    for trial in trials:
        record = records.get(trial.probe_trace_id)
        if record is None:
            raise ProbeError(
                f"trial {trial.probe_trace_id} has no run record, so its W3C trace ID and "
                "times are unknown and its spans can't be read"
            )
        try:
            w3c_ids[trial.probe_trace_id] = require_otel_trace_id(record.w3c_trace_id)
        except AgentCoreError as exc:
            raise ProbeError(
                f"trial {trial.probe_trace_id}'s run record gives its W3C trace ID as "
                f"{record.w3c_trace_id!r}, which is not 32 hex characters, so its spans "
                "can't be read; run the battery again for fresh run records"
            ) from exc
        pad = timedelta(seconds=READ_WINDOW_PADDING_SECONDS)
        try:
            started = datetime.fromisoformat(record.started_at)
            ended = datetime.fromisoformat(record.ended_at)
        except ValueError as exc:
            raise ProbeError(
                f"trial {trial.probe_trace_id}'s run record gives its start and end as "
                f"{record.started_at!r} and {record.ended_at!r}, which are not ISO-8601 times, "
                "so its spans can't be read; run the battery again for fresh run records"
            ) from exc
        window = TimeWindow(start=(started - pad).isoformat(), end=(ended + pad).isoformat())
        config = AgentCoreSourceConfig(runtime=_safe_runtime(record.runtime_arn), window=window)
        runs.append((trial, record, config))
    client = self._clients.client(_LOGS_SERVICE)
    bound = self._deadline if self._deadline is not None else settle_deadline_for(len(runs))
    deadline = self._monotonic() + bound
    settled: dict[str, list[dict[str, Any]]] = {}
    matched: dict[str, frozenset[str]] = {}
    pending = list(runs)
    confirming = False
    while pending:
        still: list[tuple[TrialRef, RunRecord, AgentCoreSourceConfig]] = []
        for run in pending:
            trial, record, config = run
            key = trial.probe_trace_id
            w3c = w3c_ids[key]
            index = _records(
                run_logs_insights(client, _SPANS_LOG_GROUP, config, trace_id=w3c), w3c
            )
            store = _records(read_trace_events(client, _SPANS_LOG_GROUP, config, w3c), w3c)
            ids = _span_ids(index)
            if ids == _span_ids(store):
                if not ids and self._monotonic() < deadline:
                    # Nothing in either yet: the spans may still be on their way to the store,
                    # so an empty trial is not taken as settled before the bound.
                    matched[key] = ids
                    still.append(run)
                    continue
                if matched.get(key) == ids:
                    settled[key] = index
                    continue
                matched[key] = ids
            else:
                matched.pop(key, None)
            still.append(run)
        pending = still
        if not pending:
            break
        if confirming or self._monotonic() >= deadline:
            # Past the bound, a trial whose last read matched gets one confirming read; a trial
            # still behind, or one whose confirming read changed, is refused.
            unconfirmed = [
                r[0].probe_trace_id for r in pending if r[0].probe_trace_id not in matched
            ]
            if confirming or unconfirmed:
                lagging = unconfirmed or [r[0].probe_trace_id for r in pending]
                raise AgentCoreSettlingLag(
                    f"the spans of {len(lagging)} trial(s) were still arriving at the "
                    f"{bound:g}s bound ({', '.join(lagging[:5])}); scoring them now could miss "
                    "decisions. Refused; run the battery again, since the index's lag "
                    "varies from run to run"
                )
            confirming = True
        self._sleep(self._interval)
    return {
        trial.probe_trace_id: TrialDecisions(
            w3c_trace_id=w3c_ids[trial.probe_trace_id],
            decisions=extract_decisions(settled[trial.probe_trace_id]),
        )
        for trial, record, _ in runs
    }

OmniReader

OmniReader(clients, config)

Reads a runtime's traces and evaluator scores through one CloudWatch Omni SQL client.

Source code in src/calibrated_agent_evaluation/agentcore/readers.py
def __init__(self, clients: AwsClients, config: AgentCoreSourceConfig) -> None:
    self._clients = clients
    self._config = config

read_traces

read_traces()

Yield each trace in the window, with its spans and its content log records.

Source code in src/calibrated_agent_evaluation/agentcore/readers.py
def read_traces(self) -> Iterator[RawTrace]:
    """Yield each trace in the window, with its spans and its content log records."""
    client = self._client()
    span_rows, _ = run_omni_query(client, build_omni_sql(self._config, session_name="spans"))
    content_rows, _ = run_omni_query(client, _content_sql(self._config))
    logs_by_trace = _group_content(content_rows)
    yield from _group_traces(span_rows, logs_by_trace)

read_scores

read_scores()

Yield each successful evaluator-result record in the window (empty when none).

Reads the gen_ai.evaluation.result records (whichever log groups the Omni dataset forwards them from) and maps each to an Evaluate-result dict; error-event records (a failed evaluation, no score) are skipped, not yielded.

Source code in src/calibrated_agent_evaluation/agentcore/readers.py
def read_scores(self) -> Iterator[dict[str, Any]]:
    """Yield each successful evaluator-result record in the window (empty when none).

    Reads the ``gen_ai.evaluation.result`` records (whichever log groups the Omni dataset
    forwards them from) and maps each to an Evaluate-result dict; error-event records (a failed
    evaluation, no score) are skipped, not yielded.
    """
    if not self._config.with_scores:
        return
    rows, _ = run_omni_query(self._client(), _scores_sql(self._config))
    for row in rows:
        result = eval_record_to_result(decode_content_row(row))
        if result is not None:
            yield result

RunRecord pydantic-model

Bases: CalibratedAgentEvaluationModel

One record of the run manifest: where one run went, what it carried and how it ended.

Attributes:

Name Type Description
schema_version str

The run record format, AGENTCORE_RUN_SCHEMA_VERSION.

trace_id str

The ID of the run's Trace; the key the record is found by.

w3c_trace_id str

The W3C trace ID the run's invocations carried.

session_id str

The runtime session the run used.

runtime_arn str

The runtime invoked.

qualifier str

The endpoint invoked.

runtime_version str | None

The runtime version that served the run, or None when the run ended before the version was known.

runtime_version_source VersionSource | None

endpoint or latest, or None with no version.

config_bundle_arn str | None

The configuration bundle the run named, or None.

config_bundle_version str | None

That bundle's version, or None.

payload_format str

The request payload format.

started_at str

When the run started, ISO-8601 with its UTC offset.

ended_at str

When the run ended, in the same form.

duration_ms int

How long the run took, in milliseconds.

attempts int

How many invocation attempts the run sent, retries included.

status_code int | None

The last response's HTTP status, or None with no response.

content_type str | None

The last response's content type, or None.

response_traceparent str | None

The traceparent the runtime sent back, or None.

replayed bool

Whether the run was replayed from a cassette instead of sent.

session_stop SessionStop

How the run's session ended.

error_code str | None

Why the run failed: the AWS error code for an AWS service error, else the type name of the error that ended it. None for a run that succeeded.

Config:

  • frozen: True
  • extra: forbid
  • defer_build: True

Fields:

schema_version pydantic-field

schema_version = AGENTCORE_RUN_SCHEMA_VERSION

trace_id pydantic-field

trace_id

w3c_trace_id pydantic-field

w3c_trace_id

session_id pydantic-field

session_id

runtime_arn pydantic-field

runtime_arn

qualifier pydantic-field

qualifier

runtime_version pydantic-field

runtime_version

runtime_version_source pydantic-field

runtime_version_source

config_bundle_arn pydantic-field

config_bundle_arn

config_bundle_version pydantic-field

config_bundle_version

payload_format pydantic-field

payload_format

started_at pydantic-field

started_at

ended_at pydantic-field

ended_at

duration_ms pydantic-field

duration_ms

attempts pydantic-field

attempts

status_code pydantic-field

status_code

content_type pydantic-field

content_type

response_traceparent pydantic-field

response_traceparent

replayed pydantic-field

replayed

session_stop pydantic-field

session_stop

error_code pydantic-field

error_code = None

RunSpanQuery pydantic-model

Bases: CalibratedAgentEvaluationModel

The run whose spans to read, named by the IDs the executor generated for it.

Attributes:

Name Type Description
trace_id str

The Calibrated Agent Evaluation trace ID of the run's Trace; every span returned must carry it.

w3c_trace_id str

The W3C trace ID every invocation of the run carried in traceparent, which the executor always marks sampled (flag 01): the runtime records no spans for an unsampled trace, so such a run could never be read.

session_id str

The runtime session ID every invocation of the run was sent with.

runtime_arn str

The runtime invoked.

qualifier str

The endpoint invoked.

started_at str

When the invocation started: ISO-8601 in UTC, with its offset.

ended_at str

When the invocation ended, in the same form.

Config:

  • frozen: True
  • extra: forbid
  • defer_build: True

Fields:

trace_id pydantic-field

trace_id

w3c_trace_id pydantic-field

w3c_trace_id

session_id pydantic-field

session_id

runtime_arn pydantic-field

runtime_arn

qualifier pydantic-field

qualifier

started_at pydantic-field

started_at

ended_at pydantic-field

ended_at

RunSpanReader

Bases: Protocol

Reads the span tree of one AgentCore Runtime run, under the contract in this module.

read_spans

read_spans(query)

The run's whole span tree, every span keyed to query.trace_id.

Parameters:

Name Type Description Default
query RunSpanQuery

The run to read.

required

Returns:

Type Description
list[Span]

The run's spans: never empty, never partial.

Raises:

Type Description
AgentCoreError

A typed subclass when the tree can't be read whole, or a guard's refusal, as it is or chained.

Source code in src/calibrated_agent_evaluation/agentcore/run_spans.py
def read_spans(self, query: RunSpanQuery) -> list[Span]:
    """The run's whole span tree, every span keyed to ``query.trace_id``.

    Args:
        query: The run to read.

    Returns:
        The run's spans: never empty, never partial.

    Raises:
        AgentCoreError: A typed subclass when the tree can't be read whole, or a guard's
            refusal, as it is or chained.
    """
    ...

RuntimeTarget pydantic-model

Bases: _RuntimeModel

The AgentCore Runtime agent to invoke, and how to ask it.

Attributes:

Name Type Description
runtime_arn str

The runtime's ARN, arn:aws:bedrock-agentcore:<region>:<account>:runtime/<name>-<10-character suffix>. Its region is where every call goes.

qualifier str

The endpoint every invocation goes to: a letter, then up to 47 letters, digits and underscores.

payload_format str

How a prompt becomes a request body: one of generic_message, openai, agentcore_sse and nested_agentcore, or custom for a body the caller's own builder makes.

config_bundle_arn str | None

The configuration bundle the agent runs with. Name it together with its version, or name neither; an empty string counts as not named.

config_bundle_version str | None

That bundle's version, a lowercase UUID.

Fields:

Validators:

runtime_arn pydantic-field

runtime_arn

qualifier pydantic-field

qualifier = DEFAULT_QUALIFIER

payload_format pydantic-field

payload_format

config_bundle_arn pydantic-field

config_bundle_arn = None

config_bundle_version pydantic-field

config_bundle_version = None

region property

region

The AWS region in the runtime ARN.

runtime_id property

runtime_id

The runtime's ID: the ARN's resource after runtime/.

AgentReportedError

AgentReportedError(message, *, error_code, trace_id=None)

Bases: RuntimeInvocationError

The agent's reply reported an error, so the run has no answer.

A caller that treats such a run as a result rather than a failure (a policy battery, where a denied tool call can come back as an agent error) tells it apart from every other refusal by type. error_code is the report's error_type when that is an identifier of at most 100 characters, else "agent_error"; the error text is never kept. trace_id is the run's trace ID when the raiser knows it, else None.

Source code in src/calibrated_agent_evaluation/agentcore/runtime_errors.py
def __init__(self, message: str, *, error_code: str, trace_id: str | None = None) -> None:
    super().__init__(message)
    self.error_code = error_code
    self.trace_id = trace_id

error_code instance-attribute

error_code = error_code

trace_id instance-attribute

trace_id = trace_id

RuntimeInvocationError

Bases: ExecutionError, AgentCoreError

A run against an AgentCore Runtime agent was refused or failed.

An ExecutionError and an AgentCoreError, so a caller of any executor and a caller of the AgentCore plug-in both catch it. The underlying error is chained as __cause__. The message names identifiers and metadata only (error codes, operations, session and trace IDs, paths), never a payload, a response body or the agent's error text, with one exception: when a span reader fails with an AgentCoreError other than a billed-call or read-only guard's refusal, the message quotes that error's text after The reader's message:. The span reader contract keeps a reader's typed refusals to IDs, log groups, counts and times. An AgentCoreError from a caller's own guard on the reader's clients looks the same, so its text is quoted too, whatever it holds.

AgentCoreRuntimeExecutor

AgentCoreRuntimeExecutor(
    target,
    *,
    span_reader=None,
    clients=None,
    profile=None,
    payload_builder=None,
    builder_name=None,
    cassette_path=None,
    cassette_mode=None,
    log_dir=None,
    frame="production-representative",
    read_timeout_seconds=DEFAULT_READ_TIMEOUT_SECONDS,
    attempt_limit=DEFAULT_ATTEMPT_LIMIT,
    sleep=time.sleep,
    clock=utc_now,
)

An ExecutorBackend that runs each task against an AgentCore Runtime agent.

Use it as a context manager. Leaving the context stops every session a run opened whose stop did not succeed, logging each result, closes the AWS clients the executor built (and the invocation clients it derived from yours), and removes its temporary log directory. A failure on the way out is raised when the block succeeded; when the block raised, the block's error is raised, with the failure logged as a warning and noted on it.

Tasks run one after another; an executor is not safe to share across threads.

Parameters:

Name Type Description Default
target RuntimeTarget

The runtime, endpoint qualifier, payload format and configuration bundle to use.

required
span_reader RunSpanReader | None

Reads each run's span tree: any object with the read_spans method of RunSpanReader. Needed whenever the executor can call AWS: with no cassette, or a cassette in record or auto mode.

None
clients AwsClients | None

AWS clients to use instead of building them, for the runtime ARN's region. The executor installs no billed-call budget on them. Give these or profile, not both.

None
profile str | None

The AWS profile to build clients from; the default credential chain when None. Read on the first live call, never before.

None
payload_builder PayloadBuilder | None

Builds the request body for the custom payload format; give none with a named format.

None
builder_name str | None

Names payload_builder, so two builders are told apart. custom needs one; a named format takes none. An empty name counts as none.

None
cassette_path str | Path | None

A cassette to record runs into or replay them from.

None
cassette_mode Mode | str | None

Whether the cassette records, replays or does both; pass it only with cassette_path. When None, CALIBRATED_AGENT_EVALUATION_CASSETTE_MODE decides (auto when unset); without a cassette, its record or auto mode runs every task live.

None
log_dir str | Path | None

The directory whose run manifest gets each run's record; a temporary directory, removed on exit, when None. A log_dir passed to run wins.

None
frame Frame

The sampling frame every returned Trace is stamped with.

'production-representative'
read_timeout_seconds float

How long an invocation waits for each read of its response: a positive, finite number of seconds of at most threading.TIMEOUT_MAX, the longest a socket can wait.

DEFAULT_READ_TIMEOUT_SECONDS
attempt_limit int

The most attempts one invocation may send, the first one included.

DEFAULT_ATTEMPT_LIMIT
sleep Callable[[float], object]

Waits between retried attempts.

sleep
clock Callable[[], datetime]

The current time, timezone-aware; it times every run.

utc_now

Raises:

Type Description
RuntimeInvocationError

clients and profile were both given, or clients are for another region; CALIBRATED_AGENT_EVALUATION_CASSETTE_MODE or cassette_mode names no mode; frame is not a sampling frame; the payload format and the builder or builder name don't pair up; the read timeout is out of the range above (a bool is not a number here), or the attempt limit is below 1; replay is asked for without a cassette, or cassette_mode without cassette_path; or the executor can call AWS but has no span reader, or span_reader is not one.

Source code in src/calibrated_agent_evaluation/agentcore/runtime_executor.py
def __init__(
    self,
    target: RuntimeTarget,
    *,
    span_reader: RunSpanReader | None = None,
    clients: AwsClients | None = None,
    profile: str | None = None,
    payload_builder: PayloadBuilder | None = None,
    builder_name: str | None = None,
    cassette_path: str | Path | None = None,
    cassette_mode: Mode | str | None = None,
    log_dir: str | Path | None = None,
    frame: Frame = "production-representative",
    read_timeout_seconds: float = DEFAULT_READ_TIMEOUT_SECONDS,
    attempt_limit: int = DEFAULT_ATTEMPT_LIMIT,
    sleep: Callable[[float], object] = time.sleep,
    clock: Callable[[], datetime] = utc_now,
) -> None:
    if clients is not None and profile is not None:
        # Refused rather than one of them silently ignored.
        raise RuntimeInvocationError(
            f"both clients= and profile={profile!r} were given, but clients passed in are "
            "used as they are and the profile would be ignored; pass clients=, or profile= "
            "and let the executor build the clients, not both"
        )
    self._target = target
    self._cassette_path = None if cassette_path is None else Path(cassette_path)
    if cassette_mode is not None:
        cassette_mode = _cassette_mode(cassette_mode)
    self._cassette_mode = cassette_mode or _mode_from_environment()
    _check_frame(frame)
    if clients is not None and clients.region != target.region:
        raise RuntimeInvocationError(
            f"the AWS clients passed in are for region {clients.region!r}, but runtime "
            f"{target.runtime_arn} is in {target.region!r}; pass clients for "
            f"{target.region!r}, or none and let the executor build them"
        )
    _check_builder(target, payload_builder, builder_name)
    check_limits(read_timeout_seconds, attempt_limit)
    replay = self._cassette_mode is Mode.REPLAY
    if replay and self._cassette_path is None:
        raise RuntimeInvocationError(
            "replay needs a cassette file: the cassette mode is replay, but no cassette was "
            "given. Pass the cassette to replay, or use record or auto mode to call AWS"
        )
    if cassette_mode is not None and self._cassette_path is None:
        # Refused, not ignored: every run would go live with nothing recorded.
        raise RuntimeInvocationError(
            f"cassette_mode={cassette_mode.value!r} was given without a cassette_path, so "
            "there is no cassette to record into and the mode would be ignored, with every "
            "run going live and billed. Pass cassette_path= too, or drop cassette_mode= to "
            "run every task live without a cassette"
        )
    if not replay and span_reader is None:
        raise RuntimeInvocationError(
            "this executor can call AWS, and live runs need the AgentCore span reader to "
            "read each run's spans, but none was given; offline runs need a cassette in "
            "replay mode. Pass span_reader=, or replay from a cassette"
        )
    if not replay and not isinstance(span_reader, RunSpanReader):
        # Refused here rather than after the first billed invocation, when its spans are read.
        raise RuntimeInvocationError(
            f"span_reader= is a {type(span_reader).__name__}, which has no read_spans method, "
            "so it is not a span reader, and live runs need the AgentCore span reader to read "
            "each run's spans; offline runs need a cassette in replay mode. Pass a span "
            "reader, or replay from a cassette"
        )
    self._span_reader = span_reader
    self._payload_builder = payload_builder
    self._builder_name = builder_name
    self._explicit_log_dir = None if log_dir is None else Path(log_dir)
    self._frame: Frame = frame
    # The clients, the preflight, the invocations, the session stops and the run records.
    self._boundary = RuntimeCallBoundary(
        target,
        clients=clients,
        profile=profile,
        read_timeout_seconds=read_timeout_seconds,
        attempt_limit=attempt_limit,
        sleep=sleep,
        clock=clock,
        owner=_OWNER,
    )
    self._tmpdir: tempfile.TemporaryDirectory[str] | None = None
    self._cassette: Cassette | None = None
    # Each fingerprint this executor recorded, tried to record or replayed. A later task with
    # one is refused, so no two traces carry one run's IDs and no failed recording is rebilled.
    self._used_fingerprints: dict[str, _UsedFingerprint] = {}
    # The session each conversation in progress runs on, by conversation ID.
    self._conversations: dict[str, str] = {}

run_records property

run_records

The record of every task run this executor finished, whatever its outcome, in order.

A record is listed even when the run manifest could not take it; manifest_errors says which writes failed.

manifest_errors property

manifest_errors

Why each run record that could not be added to the run manifest was not, in order.

One error per failed write, each naming the manifest's path and what to do next. Empty when every record in run_records is in the manifest. A successful run whose record can't be written raises its error; a failed run keeps its own error, with this one noted.

close

close()

Stop the sessions left open, close the clients built here, remove the temp log dir.

Each session whose stop did not succeed, a failed or interrupted one included, gets one more StopRuntimeSession, its result logged with its session ID. Every step runs even when an earlier one fails.

Raises:

Type Description
RuntimeInvocationError

The client to stop the open sessions could not be built, an AWS client could not be closed, or the temporary log directory could not be removed. The first failure is raised; each later one is logged as a warning and noted on it.

Source code in src/calibrated_agent_evaluation/agentcore/runtime_executor.py
def close(self) -> None:
    """Stop the sessions left open, close the clients built here, remove the temp log dir.

    Each session whose stop did not succeed, a failed or interrupted one included, gets one
    more ``StopRuntimeSession``, its result logged with its session ID. Every step runs even
    when an earlier one fails.

    Raises:
        RuntimeInvocationError: The client to stop the open sessions could not be built, an
            AWS client could not be closed, or the temporary log directory could not be
            removed. The first failure is raised; each later one is logged as a warning and
            noted on it.
    """
    # The boundary's two cleanup steps and the executor's own, all run even when one fails.
    run_all(
        (self._stop_open_sessions, self._boundary.close_clients, self._remove_tmpdir),
        owner=_OWNER,
    )

run

run(task, *, log_dir=None)

Run one task against the runtime and return its validated, owned Trace.

The run's record goes to the run manifest (agentcore_run_record.json and .csv) in log_dir, else in the executor's log directory, whatever the outcome. With a cassette, a recorded run is replayed, and a live run is recorded once its Trace is valid.

Parameters:

Name Type Description Default
task Any required
log_dir str | Path | None

The directory whose run manifest gets this run's record, instead of the executor's log directory.

None

Returns:

Type Description
list[Trace]

One Trace.

Raises:

Type Description
RuntimeInvocationError

Before any AWS call and with no run record: the task is of another type, its prompt is not a non-empty string, its trace_id is not a string or None, or its taxonomy_cell is not a mapping of strings to strings. Otherwise the run was refused or failed, named by the failing step and the session ID. A run record that can't be written is refused naming the manifest's path, or that path is noted on the run's own error when the run already failed. With a cassette, refusals name its path or the fingerprint: the cassette is missing in replay mode, or can't be read, or can't be written after a live run; a replay miss; an entry recorded under another runtime, qualifier or configuration bundle, or with no record of which one; or a fingerprint this executor already recorded, tried to record or replayed, named with that earlier run's session.

Source code in src/calibrated_agent_evaluation/agentcore/runtime_executor.py
def run(self, task: Any, *, log_dir: str | Path | None = None) -> list[Trace]:
    """Run one task against the runtime and return its validated, owned ``Trace``.

    The run's record goes to the run manifest (``agentcore_run_record.json`` and ``.csv``)
    in ``log_dir``, else in the executor's log directory, whatever the outcome. With a
    cassette, a recorded run is replayed, and a live run is recorded once its ``Trace`` is
    valid.

    Args:
        task: An :class:`AgentCoreRuntimeTask`.
        log_dir: The directory whose run manifest gets this run's record, instead of the
            executor's log directory.

    Returns:
        One ``Trace``.

    Raises:
        RuntimeInvocationError: Before any AWS call and with no run record: the task is of
            another type, its prompt is not a non-empty string, its ``trace_id`` is not a
            string or None, or its ``taxonomy_cell`` is not a mapping of strings to strings.
            Otherwise the run was refused or failed, named by the failing step and the
            session ID. A run record that can't be written is refused naming the manifest's
            path, or that path is noted on the run's own error when the run already failed.
            With a cassette, refusals name its path or the fingerprint: the cassette is
            missing in replay mode, or can't be read, or can't be written after a live run; a
            replay miss; an entry recorded under another runtime, qualifier or configuration
            bundle, or with no record of which one; or a fingerprint this executor already
            recorded, tried to record or replayed, named with that earlier run's session.
    """
    if not isinstance(task, AgentCoreRuntimeTask):
        raise RuntimeInvocationError(
            f"AgentCoreRuntimeExecutor.run expects an AgentCoreRuntimeTask, got "
            f"{type(task).__name__}; build tasks with AgentCoreTaskBuilder"
        )
    # Checked first, as the probe adapter checks: the prompt becomes the billed request's body.
    check_prompt(
        task.prompt,
        named="the task's prompt",
        remedy="Give each AgentCoreRuntimeTask the text to send to the agent",
    )
    _check_task_fields(task)
    log_dir = None if log_dir is None else Path(log_dir)
    ids = self._ids_for(task.conversation)
    run = self._boundary.start_run(ids, task.trace_id or f"agentcore-{ids.w3c_trace_id}")
    try:
        trace = self._execute(task, run)
    except BaseException as exc:
        self._finish(run, log_dir, failure=exc)
        raise
    self._finish(run, log_dir, failure=None)
    return [trace]

end_conversation

end_conversation(conversation_id)

Stop the runtime session a conversation ran on, once its last turn has run.

Call it when the conversation ends, fails or is interrupted. A conversation with no open session (unknown, already ended, or replayed from a cassette) is let go and nothing is stopped. A session whose stop did not succeed stays held, and close tries it once more.

Each turn's run record says held_open; the stop's own result is returned, and a stop that did not succeed is logged as a warning.

Parameters:

Name Type Description Default
conversation_id str

The conversation's ID, as its tasks' conversation named it.

required

Returns:

Type Description
SessionStop | None

How the stop ended, or None when no session was open.

Raises:

Type Description
RuntimeInvocationError

The client to stop the session could not be built.

Source code in src/calibrated_agent_evaluation/agentcore/runtime_executor.py
def end_conversation(self, conversation_id: str) -> SessionStop | None:
    """Stop the runtime session a conversation ran on, once its last turn has run.

    Call it when the conversation ends, fails or is interrupted. A conversation with no open
    session (unknown, already ended, or replayed from a cassette) is let go and nothing is
    stopped. A session whose stop did not succeed stays held, and :meth:`close` tries it once
    more.

    Each turn's run record says ``held_open``; the stop's own result is returned, and a stop
    that did not succeed is logged as a warning.

    Args:
        conversation_id: The conversation's ID, as its tasks' ``conversation`` named it.

    Returns:
        How the stop ended, or ``None`` when no session was open.

    Raises:
        RuntimeInvocationError: The client to stop the session could not be built.
    """
    session = self._conversations.pop(conversation_id, None)
    if session is None:
        return None
    return self._boundary.stop_session(session)

AgentCoreRuntimeTask dataclass

AgentCoreRuntimeTask(
    prompt,
    trace_id=None,
    taxonomy_cell=dict(),
    conversation=None,
)

One input to send to the AgentCore Runtime agent.

Attributes:

Name Type Description
prompt str

The test input, a non-empty string; run refuses any other before any AWS call. The returned Trace's input is {"prompt": prompt}, whatever request body the payload format builds from it.

trace_id str | None

The ID the returned Trace carries. None (or an empty string) names it after the run's W3C trace ID, agentcore-<trace ID>.

taxonomy_cell dict[str, str]

The taxonomy cell the input was written for, carried onto the Trace.

conversation ConversationTurn | None

For one turn of a simulated conversation, which conversation and turn. Every turn of a conversation is sent on one runtime session, left open between turns and stopped by end_conversation; None is a single-turn run with its own session, stopped after it.

prompt instance-attribute

prompt

trace_id class-attribute instance-attribute

trace_id = None

taxonomy_cell class-attribute instance-attribute

taxonomy_cell = field(default_factory=dict)

conversation class-attribute instance-attribute

conversation = None

AgentCoreTaskBuilder

Turns synthesized inputs into AgentCoreRuntimeTasks, one per input.

Each task's prompt is the input's text and its taxonomy_cell the input's cell, so every returned trace can be traced to the cell it was written for.

build_all

build_all(inputs)

One task per input, in order.

Parameters:

Name Type Description Default
inputs list[SynthInput]

The synthesized inputs.

required

Returns:

Type Description
list[AgentCoreRuntimeTask]

The tasks.

Raises:

Type Description
SynthError

inputs is empty.

Source code in src/calibrated_agent_evaluation/agentcore/runtime_executor.py
def build_all(self, inputs: list[SynthInput]) -> list[AgentCoreRuntimeTask]:
    """One task per input, in order.

    Args:
        inputs: The synthesized inputs.

    Returns:
        The tasks.

    Raises:
        SynthError: ``inputs`` is empty.
    """
    # Imported here: calibrated_agent_evaluation.synth loads the sampling stack, which
    # [agentcore] does not install.
    from calibrated_agent_evaluation.synth.errors import SynthError

    if not inputs:
        raise SynthError(
            "AgentCoreTaskBuilder.build_all received no inputs; refusing an empty run"
        )
    return [self.build([item]) for item in inputs]

build

build(inputs)

The task for a single input. Use build_all for a list.

Parameters:

Name Type Description Default
inputs list[SynthInput]

Exactly one synthesized input.

required

Returns:

Type Description
AgentCoreRuntimeTask

The task.

Raises:

Type Description
SynthError

inputs is empty, or holds more than one input, which only build_all can run without dropping the rest.

Source code in src/calibrated_agent_evaluation/agentcore/runtime_executor.py
def build(self, inputs: list[SynthInput]) -> AgentCoreRuntimeTask:
    """The task for a single input. Use :meth:`build_all` for a list.

    Args:
        inputs: Exactly one synthesized input.

    Returns:
        The task.

    Raises:
        SynthError: ``inputs`` is empty, or holds more than one input, which only
            :meth:`build_all` can run without dropping the rest.
    """
    from calibrated_agent_evaluation.synth.errors import SynthError

    if len(inputs) > 1:
        raise SynthError(
            f"AgentCoreTaskBuilder.build received {len(inputs)} inputs but a task carries "
            "exactly one; call build_all(inputs) to run them all"
        )
    if not inputs:
        raise SynthError(
            "AgentCoreTaskBuilder.build received no inputs; refusing an empty task"
        )
    item = inputs[0]
    return AgentCoreRuntimeTask(prompt=item.text, taxonomy_cell=dict(item.cell))

build_conversation_turn

build_conversation_turn(item, history, turn)

The task for one agent turn of a simulated conversation.

A runtime keeps the conversation's memory in its session, so the turn sends only the latest customer message; turn keeps every turn of the conversation on one session.

Parameters:

Name Type Description Default
item SynthInput

The synthesized input the conversation started from.

required
history list[dict[str, Any]]

The conversation so far, ending with the customer's message for this turn.

required
turn ConversationTurn

Which conversation and turn this is.

required

Returns:

Type Description
AgentCoreRuntimeTask

The turn's task.

Raises:

Type Description
SynthError

history is empty or does not end with a customer message.

Source code in src/calibrated_agent_evaluation/agentcore/runtime_executor.py
def build_conversation_turn(
    self, item: SynthInput, history: list[dict[str, Any]], turn: ConversationTurn
) -> AgentCoreRuntimeTask:
    """The task for one agent turn of a simulated conversation.

    A runtime keeps the conversation's memory in its session, so the turn sends only the
    latest customer message; ``turn`` keeps every turn of the conversation on one session.

    Args:
        item: The synthesized input the conversation started from.
        history: The conversation so far, ending with the customer's message for this turn.
        turn: Which conversation and turn this is.

    Returns:
        The turn's task.

    Raises:
        SynthError: ``history`` is empty or does not end with a customer message.
    """
    from calibrated_agent_evaluation.synth.conversation import message_text
    from calibrated_agent_evaluation.synth.errors import SynthError

    if not history or history[-1].get("role") != "user":
        raise SynthError(
            "a conversation turn needs the history up to the customer's latest message"
        )
    return AgentCoreRuntimeTask(
        prompt=message_text(history[-1]),
        taxonomy_cell=dict(item.cell),
        conversation=turn,
    )

RuntimeProbeAdapter

RuntimeProbeAdapter(
    target,
    *,
    clients,
    log_dir,
    read_timeout_seconds=DEFAULT_READ_TIMEOUT_SECONDS,
    attempt_limit=DEFAULT_ATTEMPT_LIMIT,
    sleep=time.sleep,
    clock=utc_now,
)

A probe adapter that sends each prompt to an AgentCore Runtime agent as one run.

Pass it to probe_with, with target_label as the probe's target. Use it as a context manager: leaving the context stops every runtime session a prompt opened whose stop did not succeed, logging each result, and closes the invocation view it derived from your clients. Your clients stay open; they are yours to close. A failure on the way out is raised when the block succeeded. When the block raised, its own error is the one raised, and the failure is logged as a warning and noted on it.

Prompts run one after another; an adapter is not safe to share across threads.

Parameters:

Name Type Description Default
target RuntimeTarget

The runtime, endpoint qualifier, payload format and configuration bundle to probe. The format is one of the probe's request shapes: generic_message, openai, agentcore_sse or nested_agentcore.

required
clients AwsClients

The AWS clients every call goes through, built for the region in the runtime ARN, with any billed-call budget already installed. The adapter adds no guard.

required
log_dir Path

The directory whose run manifest (agentcore_run_record.json and .csv) gets each prompt's run record.

required
read_timeout_seconds float

How long an invocation waits for each read of its reply: a positive, finite number of seconds of at most threading.TIMEOUT_MAX, the longest a socket can wait.

DEFAULT_READ_TIMEOUT_SECONDS
attempt_limit int

The most attempts one prompt's invocation may send, the first included.

DEFAULT_ATTEMPT_LIMIT
sleep Callable[[float], object]

Waits between retried attempts.

sleep
clock Callable[[], datetime]

The current time, timezone-aware; it times every prompt's run.

utc_now

Raises:

Type Description
RuntimeInvocationError

Raised before any client is built: the payload format is custom or not one of the probe's four, which the message lists; clients are for another region than the runtime ARN's, both named; the read timeout is not a positive, finite number of seconds of at most threading.TIMEOUT_MAX (a bool is not one); or the attempt limit is below 1. Each message names the value.

Source code in src/calibrated_agent_evaluation/agentcore/runtime_probe.py
def __init__(
    self,
    target: RuntimeTarget,
    *,
    clients: AwsClients,
    log_dir: Path,
    read_timeout_seconds: float = DEFAULT_READ_TIMEOUT_SECONDS,
    attempt_limit: int = DEFAULT_ATTEMPT_LIMIT,
    sleep: Callable[[float], object] = time.sleep,
    clock: Callable[[], datetime] = utc_now,
) -> None:
    _check_format(target)
    if clients.region != target.region:
        raise RuntimeInvocationError(
            f"the AWS clients passed in are for region {clients.region!r}, but runtime "
            f"{target.runtime_arn} is in {target.region!r}; pass clients built for "
            f"{target.region!r}"
        )
    check_limits(read_timeout_seconds, attempt_limit)
    self._target = target
    self._log_dir = Path(log_dir)
    self._label = endpoint_label(target.runtime_arn, target.qualifier)
    # The endpoint fingerprint every probed trace is stamped with, so a run record's trace_id
    # names exactly one stored trace.
    self._fingerprint = endpoint_fingerprint(self._label)
    # The clients, the invocation view, the preflight, the stops and the run records.
    self._boundary = RuntimeCallBoundary(
        target,
        clients=clients,
        read_timeout_seconds=read_timeout_seconds,
        attempt_limit=attempt_limit,
        sleep=sleep,
        clock=clock,
        setup_hint=_SETUP_HINT,
        closed_on="close",
        owner=_OWNER,
    )

target_label property

target_label

What this adapter probes, <runtime ARN>@<qualifier>, to pass as target.

Every trace ID this adapter names carries the endpoint fingerprint of this label. Passed to probe_with as target, it names the target in the authorization refusal, and every trace and the finding carry that same fingerprint.

run_records property

run_records

The record of every prompt this adapter sent, whatever its outcome, in order.

A record is listed even when the run manifest could not take it; manifest_errors says which writes failed.

manifest_errors property

manifest_errors

Why each run record that could not be added to the run manifest was not, in order.

One error per failed write, each naming the manifest's path and what to do next. Empty when every record in run_records is in the manifest. A prompt that succeeded but whose record can't be written raises its error; a failed prompt keeps its own error, with this one noted.

close

close()

Stop the sessions left open, then close the invocation view derived from your clients.

Each session whose stop did not succeed, a failed or interrupted one included, gets one more StopRuntimeSession, and its result is logged at INFO with its session ID. The view is closed even when stopping fails. Your clients are left open.

Raises:

Type Description
RuntimeInvocationError

The client to stop the open sessions could not be built, or the view could not be closed, naming the error's type and never its text. When both fail, the first is raised, and the second is logged as a warning and noted on it.

Source code in src/calibrated_agent_evaluation/agentcore/runtime_probe.py
def close(self) -> None:
    """Stop the sessions left open, then close the invocation view derived from your clients.

    Each session whose stop did not succeed, a failed or interrupted one included, gets one
    more ``StopRuntimeSession``, and its result is logged at INFO with its session ID. The
    view is closed even when stopping fails. Your clients are left open.

    Raises:
        RuntimeInvocationError: The client to stop the open sessions could not be built, or
            the view could not be closed, naming the error's type and never its text. When
            both fail, the first is raised, and the second is logged as a warning and noted
            on it.
    """
    self._boundary.close()

send

send(prompt)

Send one probe prompt as a run of its own, and return the agent's reply.

The prompt gets a new runtime session and W3C trace context. Its body is built in the target's payload format before any AWS call. Before the first invocation the runtime is checked with read-only calls, and only a check that succeeded is kept for later prompts. A busy session or throttling is retried with the same IDs and body. Once a request may have reached the service, the session is stopped before this returns or raises. The run record goes to the run manifest of the log directory, whatever the outcome.

Parameters:

Name Type Description Default
prompt str

The probe prompt, a non-empty string.

required

Returns:

Type Description
ProbeResponse

The reply: status 200, its final text, the tools its stream events name (each

ProbeResponse

string tool_name, in first-seen order, once; a single JSON document names none),

ProbeResponse

its decoded objects as the events, and

ProbeResponse

its trace ID, probe-<fingerprint>-<W3C trace ID>.

Raises:

Type Description
RuntimeInvocationError

The prompt is not a string or is empty, refused before any AWS call and with no run record; or the prompt was refused or failed, named by the step and the session ID: the body could not be built; the clients could not be built; the preflight failed, or found a runtime that only accepts OAuth or does not serve HTTP; the invocation was refused by a guard, never sent, refused by the service, still busy or throttled on its last attempt, or failed after it was sent; or the reply could not be decoded, reported an agent error, or held no answer. A prompt whose run record could not be written is refused naming the manifest's path; when the prompt had already failed, its own error is raised with that path noted.

Source code in src/calibrated_agent_evaluation/agentcore/runtime_probe.py
def send(self, prompt: str) -> ProbeResponse:
    """Send one probe prompt as a run of its own, and return the agent's reply.

    The prompt gets a new runtime session and W3C trace context. Its body is built in the
    target's payload format before any AWS call. Before the first invocation the runtime is
    checked with read-only calls, and only a check that succeeded is kept for later prompts.
    A busy session or throttling is retried with the same IDs and body. Once a request may
    have reached the service, the session is stopped before this returns or raises. The run
    record goes to the run manifest of the log directory, whatever the outcome.

    Args:
        prompt: The probe prompt, a non-empty string.

    Returns:
        The reply: status 200, its final text, the tools its stream events name (each
        string ``tool_name``, in first-seen order, once; a single JSON document names none),
        its decoded objects as the events, and
        its trace ID, ``probe-<fingerprint>-<W3C trace ID>``.

    Raises:
        RuntimeInvocationError: The prompt is not a string or is empty, refused before any
            AWS call and with no run record; or the prompt was refused or failed, named by
            the step and the session ID: the body could not be built; the clients could not
            be built; the
            preflight failed, or found a runtime that only accepts OAuth or does not serve
            HTTP; the invocation was refused by a guard, never sent, refused by the service,
            still busy or throttled on its last attempt, or failed after it was sent; or the
            reply could not be decoded, reported an agent error, or held no answer. A prompt
            whose run record could not be written is refused naming the manifest's path;
            when the prompt had already failed, its own error is raised with that path noted.
    """
    # Before the run starts, as the executor checks a task's prompt: a refused prompt costs no
    # call and leaves no run record.
    check_prompt(
        prompt,
        named="the probe prompt",
        remedy="Give each probe prompt the text to send to the agent",
    )
    ids = new_run_ids()
    run = self._boundary.start_run(ids, f"probe-{self._fingerprint}-{ids.w3c_trace_id}")
    try:
        response = self._send(prompt, run)
    except BaseException as exc:
        self._boundary.finish(run, self._log_dir, failure=exc)
        raise
    self._boundary.finish(run, self._log_dir, failure=None)
    return response

AwsClients

AwsClients(
    session, *, handlers=(), config=DEFAULT_CLIENT_CONFIG
)

Builds and caches one client per AWS service from an injected boto3 session.

Every handler in handlers runs before each API call on every client this builds, ahead of any other before-call handler, including botocore's Stubber. Use it as a context manager so the clients' connection pools are closed. Not safe to share across threads; the clients it hands out are.

Source code in src/calibrated_agent_evaluation/agentcore/session.py
def __init__(
    self,
    session: boto3.Session,
    *,
    handlers: Sequence[BeforeCallHandler] = (),
    config: Config = DEFAULT_CLIENT_CONFIG,
) -> None:
    self._session = session
    self._handlers = tuple(handlers)
    self._config = config
    self._clients: dict[tuple[str, str | None], BaseClient] = {}

region property

region

The region every client is built for.

boto_session property

boto_session

The underlying boto3 session, for an adapter that must build its own vendor client.

Named boto_session (not session) to stay clear of the unrelated review session and of subclasses that keep their own session attribute. The Code Interpreter SDK (code_session) constructs its own bedrock-agentcore client from a session rather than taking one of ours, so it needs the session itself; identity and billed-call limits are enforced by the adapter around it (require_role + the budget).

billed_call_budget property

billed_call_budget

The first billed-call budget among the handlers, or None (no limit).

Reading it charges nothing, so a caller can check remaining before a run.

close

close()

Close every client built so far; later client() calls build fresh ones.

Source code in src/calibrated_agent_evaluation/agentcore/session.py
def close(self) -> None:
    """Close every client built so far; later ``client()`` calls build fresh ones."""
    clients, self._clients = self._clients, {}
    for built in clients.values():
        built.close()

from_profile classmethod

from_profile(
    *,
    region=None,
    profile=None,
    handlers=(),
    config=DEFAULT_CLIENT_CONFIG,
)

Clients for a named profile (or the default credential chain) in region.

Without region, the profile's configured region is used.

Source code in src/calibrated_agent_evaluation/agentcore/session.py
@classmethod
def from_profile(
    cls,
    *,
    region: str | None = None,
    profile: str | None = None,
    handlers: Sequence[BeforeCallHandler] = (),
    config: Config = DEFAULT_CLIENT_CONFIG,
) -> AwsClients:
    """Clients for a named profile (or the default credential chain) in ``region``.

    Without ``region``, the profile's configured region is used.
    """
    try:
        session = boto3.Session(profile_name=profile, region_name=region)
    except ProfileNotFound as exc:
        raise AwsCredentialsMissing(
            f"AWS profile {profile!r} doesn't exist; set AWS_PROFILE or pass --profile"
        ) from exc
    except BotoCoreError as exc:
        raise AgentCoreError(f"could not open AWS profile {profile!r}: {exc}") from exc
    return cls(session, handlers=handlers, config=config)

with_config

with_config(config)

A second set of clients with config merged over this one's settings.

The new set shares this one's class, boto3 session and handlers, so every guard installed here runs on its calls too, and a billed-call budget counts both sets' calls together. Only the settings config names change; the rest (timeouts it leaves out, proxies, the user agent) carry over. This set's clients are left as they are. The new set builds and caches its own clients, and closing one set leaves the other's open, so close each.

Parameters:

Name Type Description Default
config Config

The settings to change, such as a single-attempt retry policy.

required

Returns:

Type Description
Self

The new set of clients, with no client built yet.

Source code in src/calibrated_agent_evaluation/agentcore/session.py
def with_config(self, config: Config) -> Self:
    """A second set of clients with ``config`` merged over this one's settings.

    The new set shares this one's class, boto3 session and handlers, so every guard installed
    here runs on its calls too, and a billed-call budget counts both sets' calls together.
    Only the settings ``config`` names change; the rest (timeouts it leaves out, proxies, the
    user agent) carry over. This set's clients are left as they are. The new set builds and
    caches its own clients, and closing one set leaves the other's open, so close each.

    Args:
        config: The settings to change, such as a single-attempt retry policy.

    Returns:
        The new set of clients, with no client built yet.
    """
    # A shallow copy, so a subclass keeps its own state too; only the settings and the client
    # cache differ from this set.
    view = copy.copy(self)
    view._config = self._config.merge(config)
    view._clients = {}
    return view

client

client(service_name, *, region=None)

The cached client for service_name, built on first use.

region builds the client in another region (for an API that answers only from a few endpoint regions, such as the Price List API), with the same handlers and settings; it is cached per service and region. Without it, the client is built in this set's region.

Source code in src/calibrated_agent_evaluation/agentcore/session.py
def client(self, service_name: str, *, region: str | None = None) -> BaseClient:
    """The cached client for ``service_name``, built on first use.

    ``region`` builds the client in another region (for an API that answers only from a few
    endpoint regions, such as the Price List API), with the same handlers and settings; it is
    cached per service and region. Without it, the client is built in this set's region.
    """
    key = (service_name, region)
    cached = self._clients.get(key)
    if cached is not None:
        return cached
    try:
        built = self._session.client(  # pyright: ignore[reportUnknownMemberType]
            service_name, region_name=region, config=self._config
        )
    except BotoCoreError as exc:
        raise AgentCoreError(
            f"could not build an AWS client for {service_name!r} in region "
            f"{region or self._session.region_name!r}: {exc}"
        ) from exc
    for handler in self._handlers:
        built.meta.events.register_first("before-call.*.*", handler)
    self._clients[key] = built
    return built

caller_identity

caller_identity()

The caller's ARN, which also proves credentials resolve before any other call.

Source code in src/calibrated_agent_evaluation/agentcore/session.py
def caller_identity(self) -> str:
    """The caller's ARN, which also proves credentials resolve before any other call."""
    try:
        identity = self.client("sts").get_caller_identity()  # pyright: ignore[reportAttributeAccessIssue]
    except NoCredentialsError as exc:
        raise AwsCredentialsMissing(
            "no AWS credentials found; set AWS_PROFILE or pass --profile"
        ) from exc
    except (SSOError, TokenRetrievalError) as exc:
        raise AwsCredentialsMissing(_REFRESH.format(exc=exc)) from exc
    except BotoCoreError as exc:
        raise AgentCoreError(f"could not confirm the AWS identity: {exc}") from exc
    except ClientError as exc:
        # Not a BotoCoreError: an expired or revoked token comes back from STS as a service
        # error, and used to escape untyped from
        # `calibrated-agent-evaluation doctor --agentcore`.
        code = str(exc.response.get("Error", {}).get("Code", ""))
        if code in _EXPIRED_OR_INVALID:
            raise AwsCredentialsMissing(_REFRESH.format(exc=exc)) from exc
        raise AgentCoreError(f"could not confirm the AWS identity: {exc}") from exc
    return str(identity["Arn"])

SettlePolicy

SettlePolicy(
    lag_seconds=DEFAULT_LAG_SECONDS,
    *,
    max_wait_seconds=None,
    now=None,
    sleep=None,
)

Decides whether a window has settled, waiting a bounded interval when it has not.

Source code in src/calibrated_agent_evaluation/agentcore/settle.py
def __init__(
    self,
    lag_seconds: float = DEFAULT_LAG_SECONDS,
    *,
    max_wait_seconds: float | None = None,
    now: Callable[[], datetime] | None = None,
    sleep: Callable[[float], None] | None = None,
) -> None:
    self.lag_seconds = lag_seconds
    # Without an explicit bound, wait at most one full lag — long enough for a window that just
    # entered the lag to clear it, bounded so a far-future window cannot block forever.
    self.max_wait_seconds = lag_seconds if max_wait_seconds is None else max_wait_seconds
    self._now = now or (lambda: datetime.now(tz=UTC))
    self._sleep = sleep or time.sleep

lag_seconds instance-attribute

lag_seconds = lag_seconds

max_wait_seconds instance-attribute

max_wait_seconds = (
    lag_seconds
    if max_wait_seconds is None
    else max_wait_seconds
)

from_env classmethod

from_env(environ=None)

A policy whose lag is the one the environment sets, when set.

The variable is CALIBRATED_AGENT_EVALUATION_AGENTCORE_SETTLE_SECONDS.

Source code in src/calibrated_agent_evaluation/agentcore/settle.py
@classmethod
def from_env(cls, environ: dict[str, str] | None = None) -> SettlePolicy:
    """A policy whose lag is the one the environment sets, when set.

    The variable is ``CALIBRATED_AGENT_EVALUATION_AGENTCORE_SETTLE_SECONDS``.
    """
    env = environ if environ is not None else dict(os.environ)
    raw = env.get(SETTLE_ENV_VAR, "").strip()
    if not raw:
        return cls()
    try:
        return cls(float(raw))
    except ValueError as exc:
        raise AgentCoreSettlingLag(
            f"{SETTLE_ENV_VAR}={raw!r} is not a number of seconds"
        ) from exc

is_settled

is_settled(window, *, now=None)

Whether the window ended at least lag_seconds before now.

Source code in src/calibrated_agent_evaluation/agentcore/settle.py
def is_settled(self, window: TimeWindow, *, now: datetime | None = None) -> bool:
    """Whether the window ended at least ``lag_seconds`` before ``now``."""
    current = now or self._now()
    return (current - _parse_end(window)).total_seconds() >= self.lag_seconds

refuse_if_unsettled

refuse_if_unsettled(window, *, now=None)

Raise AgentCoreSettlingLag when the window ends inside the lag.

Source code in src/calibrated_agent_evaluation/agentcore/settle.py
def refuse_if_unsettled(self, window: TimeWindow, *, now: datetime | None = None) -> None:
    """Raise :class:`AgentCoreSettlingLag` when the window ends inside the lag."""
    if not self.is_settled(window, now=now):
        raise AgentCoreSettlingLag(
            f"the window ends at {window.end}, inside the {self.lag_seconds:g}s settling lag; "
            f"reading now would return a partial trace. Retry after the lag, or raise "
            f"{SETTLE_ENV_VAR} if the account settles faster"
        )

wait_for_window

wait_for_window(window)

Wait up to max_wait_seconds for the window to settle, else refuse.

Returns as soon as the window has settled. Raises AgentCoreSettlingLag if it cannot settle within the bounded wait, so a window too recent to ever settle in time is a loud refusal, not a silent partial read.

Source code in src/calibrated_agent_evaluation/agentcore/settle.py
def wait_for_window(self, window: TimeWindow) -> None:
    """Wait up to ``max_wait_seconds`` for the window to settle, else refuse.

    Returns as soon as the window has settled. Raises :class:`AgentCoreSettlingLag` if it
    cannot settle within the bounded wait, so a window too recent to ever settle in time is a
    loud refusal, not a silent partial read.
    """
    deadline = self._now() + timedelta(seconds=self.max_wait_seconds)
    while True:
        if self.is_settled(window):
            return
        if self._now() >= deadline:
            self.refuse_if_unsettled(window)
            return
        self._sleep(_POLL_SECONDS)

AgentCoreSourceConfig pydantic-model

Bases: CalibratedAgentEvaluationModel

What to read, resolved from the CLI flags.

omni selects the CloudWatch Omni SQL path; otherwise spans are read from Logs Insights. (Omni queries need no domain — verified live — so this is a plain flag, not a domain name: a domain value would only pick the path while never reaching a call, so a typo would read the default silently.) rekey_sessions opts into collapsing a mixed-session trace to the runtime session id. with_scores additionally reads AgentCore evaluator results as advisory labels.

Config:

  • frozen: True
  • extra: forbid

Fields:

Validators:

runtime pydantic-field

runtime

window pydantic-field

window

omni pydantic-field

omni = False

rekey_sessions pydantic-field

rekey_sessions = False

with_scores pydantic-field

with_scores = False

uses_omni property

uses_omni

Whether this config reads through the Omni SQL path.

AgentCoreTraceSource

AgentCoreTraceSource(reader, config, *, settle=None)

A read-only TraceSource for AgentCore.

Satisfies the TraceSource protocol (pull only); it is not a store and exposes no write. The AWS reads are the injected reader; this class owns the Calibrated Agent Evaluation-side orchestration: wait for the window to settle, resolve session identity, convert to OTLP, and refuse (per trace) a trace whose content did not come back.

Source code in src/calibrated_agent_evaluation/agentcore/sources.py
def __init__(
    self,
    reader: AgentCoreReader,
    config: AgentCoreSourceConfig,
    *,
    settle: SettlePolicy | None = None,
) -> None:
    from calibrated_agent_evaluation.agentcore.settle import SettlePolicy as _SettlePolicy

    self._reader = reader
    self._config = config
    self._settle = settle if settle is not None else _SettlePolicy.from_env()
    # Mixed-session traces re-keyed this pull: (original id, runtime-session id). The ingest
    # report reads this to record the re-key for each such trace.
    self.rekeyed: list[tuple[str, str]] = []

rekeyed instance-attribute

rekeyed = []

pull

pull()

Yield each AgentCore trace as a SourceTrace carrying canonical OTLP-JSON.

Source code in src/calibrated_agent_evaluation/agentcore/sources.py
def pull(self) -> Iterator[SourceTrace]:
    """Yield each AgentCore trace as a :class:`SourceTrace` carrying canonical OTLP-JSON."""
    # Wait for the window's writes to settle (bounded); a window too recent to settle in time
    # raises before any trace is yielded, so a partial trace is never ingested.
    self._settle.wait_for_window(self._config.window)
    # The reader raises before its first yield when the account is unreachable or the read is
    # incomplete (partial/truncated/indexing), so those propagate out of pull untouched.
    for raw in self._reader.read_traces():
        yield self._to_source_trace(raw)

read_scores

read_scores()

The reader's evaluator-result records (mapped to labels by the scores module).

Source code in src/calibrated_agent_evaluation/agentcore/sources.py
def read_scores(self) -> Iterator[dict[str, Any]]:
    """The reader's evaluator-result records (mapped to labels by the scores module)."""
    return self._reader.read_scores()

TimeWindow pydantic-model

Bases: CalibratedAgentEvaluationModel

The read window, as timezone-aware ISO-8601 bounds.

Timezone-aware is required: a naive bound compared against AWS timestamps silently shifts the window by the local offset, so a naive value is refused rather than coerced.

Config:

  • frozen: True
  • extra: forbid

Fields:

Validators:

  • _tz_aware_iso → start, end
  • _start_before_end

start pydantic-field

start

end pydantic-field

end

AgentCoreSpanReader

AgentCoreSpanReader(
    clients,
    *,
    use_omni=False,
    settle_deadline_seconds=_DEFAULT_SETTLE_DEADLINE_SECONDS,
    poll_interval_seconds=_DEFAULT_POLL_INTERVAL_SECONDS,
)

Reads one AgentCore Runtime run's span tree by id, satisfying the executor's reader contract.

Built from the same AwsClients as the executor, so its billed reads count against the shared budget and a guard's refusal (for example BilledCallBudgetExceeded) propagates to the caller. The read path is the Logs Insights group aws/spans by default, or CloudWatch Omni when use_omni=True (Omni needs no domain, so the path is an explicit flag, as in the window ingest). The Logs Insights path needs logs:StartQuery, logs:GetQueryResults and logs:FilterLogEvents on aws/spans and on the runtime's own log group.

Source code in src/calibrated_agent_evaluation/agentcore/span_reader.py
def __init__(
    self,
    clients: AwsClients,
    *,
    use_omni: bool = False,
    settle_deadline_seconds: float = _DEFAULT_SETTLE_DEADLINE_SECONDS,
    poll_interval_seconds: float = _DEFAULT_POLL_INTERVAL_SECONDS,
) -> None:
    # A zero interval would issue billed Omni queries back to back, and a deadline shorter than
    # one interval can never see the two equal polls a whole tree needs.
    if poll_interval_seconds <= 0 or settle_deadline_seconds < poll_interval_seconds:
        raise AgentCoreError(
            f"poll_interval_seconds must be > 0 and settle_deadline_seconds at least that long "
            f"(got {poll_interval_seconds:g}s and {settle_deadline_seconds:g}s)"
        )
    self._clients = clients
    self._use_omni = use_omni
    self._settle_deadline_seconds = settle_deadline_seconds
    self._poll_interval_seconds = poll_interval_seconds

read_spans

read_spans(query)

The run's whole normalized span tree, every span keyed to query.trace_id.

Polls the by-id read until a stable read (the same span count and content-record count on two polls in a row) assembles into a whole tree. On the Logs Insights path a read counts only when it holds the same spans and as many content records as the log store already holds for the run, since the index serves a run's records in batches minutes apart and two equal reads can both be missing one. A stable read that is not yet whole (no agent span tree yet, an agent with no content, or a completed agent missing its input or output) is polled again; at the deadline the reader raises that refusal (AgentCoreContentMissing, or an AgentCoreError naming that no agent tree formed). A run that never appears, or is still changing at the deadline, raises AgentCoreSettlingLag. A run carrying more than one session id raises AgentCoreMixedSession. An errored agent span may lack its answer. A partial or truncated read raises its typed error (from the reused drivers). Never returns a partial or empty list.

Source code in src/calibrated_agent_evaluation/agentcore/span_reader.py
def read_spans(self, query: RunSpanQuery) -> list[Span]:
    """The run's whole normalized span tree, every span keyed to ``query.trace_id``.

    Polls the by-id read until a stable read (the same span count and content-record count on
    two polls in a row) assembles into a whole tree. On the Logs Insights path a read counts
    only when it holds the same spans and as many content records as the log store already
    holds for the run, since the index serves a run's records in batches minutes apart and
    two equal reads can both be missing one. A stable read that is not yet whole (no
    agent span tree yet, an agent with no content, or a completed agent missing its input or
    output) is polled again; at the deadline the reader raises that refusal
    (:class:`~calibrated_agent_evaluation.agentcore.errors.AgentCoreContentMissing`, or an
    ``AgentCoreError`` naming that no agent tree formed). A run that never appears, or is still
    changing at the deadline, raises :class:`AgentCoreSettlingLag`. A run carrying more than one
    session id raises :class:`AgentCoreMixedSession`. An errored agent span may lack its answer.
    A partial or truncated read raises its typed error (from the reused drivers). Never returns
    a partial or empty list.
    """
    w3c = _require_w3c(query.w3c_trace_id)
    # Pad the invocation window: a record stamped at or just after the client-observed end, or
    # clock skew between client and AWS, would otherwise be missed on every poll. The queries
    # filter on the 32-hex trace id, so a wider window cannot pull in another run.
    slack = timedelta(seconds=_WINDOW_SLACK_SECONDS)
    try:
        started = datetime.fromisoformat(query.started_at)
        ended = datetime.fromisoformat(query.ended_at)
    except (TypeError, ValueError) as exc:
        raise AgentCoreError(
            f"run {query.trace_id!r} has an invocation time that is not ISO-8601 "
            f"(started_at={query.started_at!r}, ended_at={query.ended_at!r})"
        ) from exc
    window = TimeWindow(start=(started - slack).isoformat(), end=(ended + slack).isoformat())
    config = AgentCoreSourceConfig(
        runtime=_safe_runtime(query.runtime_arn), window=window, omni=self._use_omni
    )
    deadline = time.monotonic() + self._settle_deadline_seconds
    previous: tuple[int, int] | None = None
    while True:
        try:
            spans, content, sessions = self._read_once(config, w3c)
        except _IndexBehindStore as behind:
            # Two equal index reads can both miss a batch the index has not served yet, so a
            # read the store contradicts never counts toward the two in a row.
            if time.monotonic() >= deadline:
                raise AgentCoreSettlingLag(
                    f"run {w3c!r} was still being indexed at the "
                    f"{self._settle_deadline_seconds:g}s deadline: {behind}. Returning it now "
                    "would hand back a partial tree. Refused; retry with a longer settle "
                    "deadline"
                ) from behind
            previous = None
            time.sleep(self._poll_interval_seconds)
            continue
        # The content records lag the spans in the Logs Insights index (observed live: the spans
        # had settled while an agent's final answer was not yet readable), so the tree has
        # stopped arriving only when both counts hold between two polls.
        signature = (len(spans), len(content))
        not_settled: AgentCoreError | None = None
        if spans and signature == previous:
            if len(sessions) > 1:
                raise AgentCoreMixedSession(
                    f"run {query.trace_id!r} (w3c {w3c!r}) carries {len(sessions)} session ids "
                    f"({', '.join(sorted(sessions))}); a single invocation carries one "
                    "session, so the tree is ambiguous and is refused rather than fused"
                )
            try:
                return self._assemble(spans, content, query.trace_id, w3c)
            except (AgentCoreContentMissing, _NoAgentTree) as exc:
                # Stable but not yet readable as a whole tree: content still lagging in the
                # index, or the agent spans not landed yet. Poll again; refuse at the deadline.
                not_settled = exc
        if time.monotonic() >= deadline:
            if not_settled is not None:
                raise not_settled
            if not spans:
                raise AgentCoreSettlingLag(
                    f"no spans for run {w3c!r} within {self._settle_deadline_seconds:g}s. "
                    "Check that the invocation's trace context was sampled: a traceparent "
                    "without the 01 sampled flag makes the runtime record no agent spans, even "
                    "though its content logs still land. Otherwise the telemetry has not "
                    "settled yet. Refused rather than returned empty"
                )
            raise AgentCoreSettlingLag(
                f"run {w3c!r} was still changing at the {self._settle_deadline_seconds:g}s "
                f"deadline ({len(spans)} spans, {len(content)} content records on the last "
                "poll); returning it now could hand back a partial tree. Refused; retry with a "
                "longer settle deadline"
            )
        previous = signature
        time.sleep(self._poll_interval_seconds)

adot_to_otlp

adot_to_otlp(
    span_records, log_records, *, resource_attributes=None
)

Assemble OTLP span dicts into one canonical OTLP-JSON document, content re-homed.

span_records are already-decoded OTLP span dicts (spanId/traceId/parentSpanId/ attributes/...). log_records are the decoded otel-rt-logs content records, each a plain dict with spanId, eventName and body (and an optional timestamp used only for ordering). Content is re-homed from the chat records onto each invoke_agent and execute_tool span; a content-bearing span that recovers nothing raises AgentCoreContentMissing naming the trace and span.

Source code in src/calibrated_agent_evaluation/agentcore/adot.py
def adot_to_otlp(
    span_records: list[dict[str, Any]],
    log_records: list[dict[str, Any]],
    *,
    resource_attributes: dict[str, Any] | None = None,
) -> dict[str, Any]:
    """Assemble OTLP span dicts into one canonical OTLP-JSON document, content re-homed.

    ``span_records`` are already-decoded OTLP span dicts (``spanId``/``traceId``/``parentSpanId``/
    ``attributes``/...). ``log_records`` are the decoded ``otel-rt-logs`` content records, each a
    plain dict with ``spanId``, ``eventName`` and ``body`` (and an optional timestamp used only for
    ordering). Content is re-homed from the ``chat`` records onto each ``invoke_agent`` and
    ``execute_tool`` span; a content-bearing span that recovers nothing raises
    :class:`AgentCoreContentMissing` naming the trace and span.
    """
    spans = [dict(span) for span in span_records]
    by_id = {s.get("spanId"): s for s in spans if isinstance(s.get("spanId"), str)}
    agent_ids = {sid for sid, s in by_id.items() if _op(s) == "invoke_agent"}

    logs_by_span: dict[str, list[dict[str, Any]]] = defaultdict(list)
    for record in log_records:
        span_id = record.get("spanId")
        if isinstance(span_id, str):
            logs_by_span[span_id].append(record)
    tool_args, tool_results = _index_tool_payloads(log_records)

    for span_id in agent_ids:
        _rehome_agent_content(by_id[span_id], span_id, spans, by_id, agent_ids, logs_by_span)
    for span in spans:
        if _op(span) == "execute_tool":
            _rehome_tool_content(span, tool_args, tool_results)

    # The content refusal gates on agent handoff I/O only. That is where wholesale content
    # stripping shows up (an invoke_agent with no recoverable input or output means its whole
    # conversation was dropped). A leaf execute_tool whose args/result were never emitted as a
    # separate content record is left best-effort (empty args/result) rather than failing the whole
    # trace: the tool call is still structurally recovered, and the normalizer's content gate counts
    # it as contentless, so the gap is reported, not silent. Verified against a live trace whose
    # search_kb tool logged no gen_ai content record of its own.
    for span in spans:
        if _op(span) == "invoke_agent":
            _refuse_if_agent_content_missing(span)

    present = set(by_id)
    for span in spans:
        parent = span.get("parentSpanId")
        if isinstance(parent, str) and parent and parent not in present:
            # A root whose parent is outside the pulled window; clear it so the tree re-roots here
            # rather than being rejected as a dangling edge.
            span["parentSpanId"] = ""

    return {
        "resourceSpans": [
            {
                "resource": {"attributes": dict(resource_attributes or {})},
                "scopeSpans": [{"scope": {"name": "strands.telemetry.tracer"}, "spans": spans}],
            }
        ]
    }

xray_trace_id_to_otel

xray_trace_id_to_otel(raw)

Normalize an X-Ray 1-<8 hex>-<24 hex> trace id to the 32-hex OTel id.

An already-32-hex id is returned lowercased unchanged. A value in neither form raises rather than returning a corrupt id, so a malformed id never silently points at a nonexistent trace.

Source code in src/calibrated_agent_evaluation/agentcore/adot.py
def xray_trace_id_to_otel(raw: str) -> str:
    """Normalize an X-Ray ``1-<8 hex>-<24 hex>`` trace id to the 32-hex OTel id.

    An already-32-hex id is returned lowercased unchanged. A value in neither form raises rather
    than returning a corrupt id, so a malformed id never silently points at a nonexistent trace.
    """
    value = raw.strip()
    if _is_hex(value, 32):
        return value.lower()
    parts = value.split("-")
    if len(parts) == 3 and parts[0] == "1" and _is_hex(parts[1], 8) and _is_hex(parts[2], 24):
        return (parts[1] + parts[2]).lower()
    raise AgentCoreError(
        f"trace id {raw!r} is neither a 32-hex OTel id nor an X-Ray '1-<8hex>-<24hex>' id"
    )

merge_arms_read

merge_arms_read(stored, read)

The assignments of read to store, given what is already stored for its test.

A read under a new read id is stored whole; one that repeats a stored read exactly stores nothing. A stored read never changes.

Raises:

Type Description
AgentCoreArmsReadChanged

The read repeats a stored read id and adds, drops or reassigns a source id; nothing of it may be stored.

Source code in src/calibrated_agent_evaluation/agentcore/arms.py
def merge_arms_read(stored: Sequence[ArmAssignment], read: ArmRead) -> tuple[ArmAssignment, ...]:
    """The assignments of ``read`` to store, given what is already stored for its test.

    A read under a new read id is stored whole; one that repeats a stored read exactly stores
    nothing. A stored read never changes.

    Raises:
        AgentCoreArmsReadChanged: The read repeats a stored read id and adds, drops or reassigns
            a source id; nothing of it may be stored.
    """
    if not read.assignments:
        return ()
    ab_test_arn = read.assignments[0].ab_test_arn
    before = {
        a.trace_id: a for a in stored if a.ab_test_arn == ab_test_arn and a.read_id == read.read_id
    }
    if not before:
        return read.assignments
    after = {a.trace_id: a for a in read.assignments}
    added = sorted(set(after) - set(before))
    missing = sorted(set(before) - set(after))
    changed = sorted(i for i in set(after) & set(before) if after[i] != before[i])
    if added or missing or changed:
        raise AgentCoreArmsReadChanged(
            f"arms read {read.read_id} is already stored and this re-read differs: added "
            f"{added or 'none'}, missing {missing or 'none'}, changed {changed or 'none'}; a "
            "stored read never changes, so nothing was written. Read another window to keep both"
        )
    return ()

read_arms

read_arms(
    source,
    *,
    ab_test_arn,
    runtime,
    window,
    omni=False,
    rekey_sessions=False,
    rekey_collisions=None,
    contentless=None,
    applied_bundle_attribute=None,
    applied_bundle_require=(),
    bundle_versions=None,
)

Read which arm of ab_test_arn each trace the source delivers ran in.

The source is pulled to the end. A source id delivered more than once, and each re-key collision rekey_collisions reports, is one refused assignment. Every other trace is unread (delivered as an error or a normalized trace), refused for a half-stamped span, unassigned (no span carries both attributes), refused (two arms, a variant other than C or T1, two runtime sessions, a failed applied-bundle condition), set aside as another experiment's, or assigned. A runtime session whose traces sit in two arms is then refused.

Parameters:

Name Type Description Default
source TraceSource

Any trace source; for the AgentCore ingest's, the read path and re-key setting used at ingest.

required
ab_test_arn str

The audited test's ARN.

required
runtime str

The runtime read, recorded with the read.

required
window _Window

The read window, with start and end.

required
omni bool

Whether the source read through Omni; recorded, not used.

False
rekey_sessions bool

Whether the source re-keyed mixed-session traces; recorded, not used.

False
rekey_collisions Callable[[], Mapping[str, Sequence[KeptTrace]]] | None

Called once after the pull: per runtime session id, every trace that re-keyed to it, as collisions gives them.

None
contentless Callable[[], Mapping[str, Mapping[str, Any]]] | None

Called once after the pull: per trace id, the spans (as an OTLP-JSON document with no content) of each trace the source delivered as an error because its message content did not come back, as contentless gives them. Its arm is read from those spans and its assignment marked content_missing: an arm stamp is a span attribute, so a trace the ingest refuses for content still shows which arm served it.

None
applied_bundle_attribute str | None

A span attribute that must equal the arm's bundle version.

None
applied_bundle_require Iterable[tuple[str, str]]

(key, value) pairs every assigned trace must carry.

()
bundle_versions Mapping[str, str] | None

Each arm's bundle version id, for applied_bundle_attribute.

None

Returns:

Type Description
ArmRead

The read: its id, one assignment per source id, and its counts.

Raises:

Type Description
AgentCoreArmsMissing

No span is stamped, or no stamped span names the audited test.

Exception

Whatever the source raises before its first trace, unchanged.

Source code in src/calibrated_agent_evaluation/agentcore/arms.py
def read_arms(
    source: TraceSource,
    *,
    ab_test_arn: str,
    runtime: str,
    window: _Window,
    omni: bool = False,
    rekey_sessions: bool = False,
    rekey_collisions: Callable[[], Mapping[str, Sequence[KeptTrace]]] | None = None,
    contentless: Callable[[], Mapping[str, Mapping[str, Any]]] | None = None,
    applied_bundle_attribute: str | None = None,
    applied_bundle_require: Iterable[tuple[str, str]] = (),
    bundle_versions: Mapping[str, str] | None = None,
) -> ArmRead:
    """Read which arm of ``ab_test_arn`` each trace the source delivers ran in.

    The source is pulled to the end. A source id delivered more than once, and each re-key
    collision ``rekey_collisions`` reports, is one refused assignment. Every other trace is
    unread (delivered as an error or a normalized trace), refused for a half-stamped span,
    unassigned (no span carries both attributes), refused (two arms, a variant other than C or
    T1, two runtime sessions, a failed applied-bundle condition), set aside as another
    experiment's, or assigned. A runtime session whose traces sit in two arms is then refused.

    Args:
        source: Any trace source; for the AgentCore ingest's, the read path and re-key setting
            used at ingest.
        ab_test_arn: The audited test's ARN.
        runtime: The runtime read, recorded with the read.
        window: The read window, with ``start`` and ``end``.
        omni: Whether the source read through Omni; recorded, not used.
        rekey_sessions: Whether the source re-keyed mixed-session traces; recorded, not used.
        rekey_collisions: Called once after the pull: per runtime session id, every trace that
            re-keyed to it, as :meth:`RawTraceLog.collisions` gives them.
        contentless: Called once after the pull: per trace id, the spans (as an OTLP-JSON
            document with no content) of each trace the source delivered as an error because its
            message content did not come back, as :meth:`RawTraceLog.contentless` gives them. Its
            arm is read from those spans and its assignment marked ``content_missing``: an arm
            stamp is a span attribute, so a trace the ingest refuses for content still shows
            which arm served it.
        applied_bundle_attribute: A span attribute that must equal the arm's bundle version.
        applied_bundle_require: ``(key, value)`` pairs every assigned trace must carry.
        bundle_versions: Each arm's bundle version id, for ``applied_bundle_attribute``.

    Returns:
        The read: its id, one assignment per source id, and its counts.

    Raises:
        AgentCoreArmsMissing: No span is stamped, or no stamped span names the audited test.
        Exception: Whatever the source raises before its first trace, unchanged.
    """
    require = dict(applied_bundle_require)
    read_id = arms_read_id(
        runtime=runtime,
        window_start=window.start,
        window_end=window.end,
        omni=omni,
        rekey_sessions=rekey_sessions,
        applied_bundle_attribute=applied_bundle_attribute,
        applied_bundle_require=require,
    )
    ctx = _Context(
        ab_test_arn=ab_test_arn,
        runtime=runtime,
        window_start=window.start,
        window_end=window.end,
        omni=omni,
        rekey_sessions=rekey_sessions,
        read_id=read_id,
        applied_bundle_attribute=applied_bundle_attribute,
        applied_bundle_require=require,
    )
    groups: dict[str, list[SourceTrace]] = {}
    for delivered in source.pull():
        groups.setdefault(delivered.source_id, []).append(delivered)
    collisions = dict(rekey_collisions()) if rekey_collisions is not None else {}
    stripped = dict(contentless()) if contentless is not None else {}

    assignments: list[ArmAssignment] = []
    stamped_arns: set[str] = set()
    stamped_total = 0
    half_traces: list[str] = []
    consumed: set[str] = set()

    def note_arns(deliveries: Iterable[SourceTrace]) -> None:
        for d in deliveries:
            for span in _spans_of(d.otlp or {}):
                if span.stamp == "stamped":
                    stamped_arns.add(str(span.attrs[ARM_ARN_ATTRIBUTE]))

    for session, traces in sorted(collisions.items()):
        assignments.append(_shared(session, "rekey-collision", traces, ctx))
        stamped_total += sum(t.spans_stamped for t in traces)
        # Every collided turn counts, not only the one the ingest delivered under the session.
        stamped_arns.update(a for t in traces for a in t.experiment_arns)
        if any(t.spans_half_stamped for t in traces):
            half_traces.append(session)
        note_arns(groups.get(session, []))
        consumed.add(session)
        consumed.update(t.trace_id for t in traces)
    for source_id in sorted(groups):
        if source_id in consumed:
            continue
        deliveries = groups[source_id]
        if len(deliveries) > 1:
            kept = [
                k
                for d in deliveries
                if d.otlp is not None
                for k in _kept_by_trace(source_id, d.otlp)
            ]
            assignments.append(_shared(source_id, "repeated-source-id", kept, ctx))
            stamped_total += sum(k.spans_stamped for k in kept)
            if any(k.spans_half_stamped for k in kept):
                half_traces.append(source_id)
            note_arns(deliveries)
            continue
        delivered = deliveries[0]
        doc = stripped.get(source_id) if delivered.error is not None else None
        if doc is not None:
            from calibrated_agent_evaluation.backends.base import SourceTrace as _Delivered

            delivered = _Delivered(source_id=source_id, otlp=dict(doc))
        assignment, spans = _assign_one(source_id, delivered, ctx, bundle_versions)
        if doc is not None:
            assignment = assignment.model_copy(update={"content_missing": True})
        assignments.append(assignment)
        stamped_total += assignment.spans_stamped
        if assignment.spans_half_stamped:
            half_traces.append(source_id)
        stamped_arns.update(str(s.attrs[ARM_ARN_ATTRIBUTE]) for s in spans if s.stamp == "stamped")
    assignments = _split_sessions(assignments)
    unread = [a for a in assignments if a.status == "unread"]
    where = f"runtime {runtime!r}, window {window.start} to {window.end}"
    if stamped_total == 0:
        notes = []
        if unread:
            notes.append(_unread_note(unread))
        if half_traces:
            notes.append(f"trace(s) with half-stamped spans: {', '.join(sorted(half_traces))}")
        tail = "; ".join(notes) if notes else _GATEWAY_CAUSE
        raise AgentCoreArmsMissing(
            f"no span in {where} carries both {ARM_ARN_ATTRIBUTE} and {ARM_VARIANT_ATTRIBUTE}. "
            f"{tail}"
        )
    if ab_test_arn not in stamped_arns:
        seen = ", ".join(sorted(stamped_arns)) or "none"
        note = f"; {_unread_note(unread)}" if unread else ""
        raise AgentCoreArmsMissing(
            f"no trace in {where} is stamped with the audited test {ab_test_arn}; the "
            f"experiments seen are {seen}{note}"
        )
    return ArmRead(
        read_id=read_id,
        assignments=tuple(sorted(assignments, key=lambda a: a.trace_id)),
        counts=arm_counts(assignments),
        arns_seen=tuple(sorted(stamped_arns)),
    )

runtime_session_id

runtime_session_id(otlp_trace)

The runtime session a trace ran in: the session.id of its /invocations server span.

An agent's own spans may carry another session.id; only the runtime's request span names the session the client sent. None when the trace has no such span or it carries no id.

Raises:

Type Description
AgentCoreArmSessionAmbiguous

Two /invocations server spans name different sessions.

Source code in src/calibrated_agent_evaluation/agentcore/arms.py
def runtime_session_id(otlp_trace: Mapping[str, Any]) -> str | None:
    """The runtime session a trace ran in: the ``session.id`` of its ``/invocations`` server span.

    An agent's own spans may carry another ``session.id``; only the runtime's request span names
    the session the client sent. ``None`` when the trace has no such span or it carries no id.

    Raises:
        AgentCoreArmSessionAmbiguous: Two ``/invocations`` server spans name different sessions.
    """
    found: set[str] = set()
    for span in iter_spans(dict(otlp_trace)):
        attrs = attributes(span)
        if span.get("kind") in _SERVER_KINDS and attrs.get("http.route") == _INVOCATIONS_ROUTE:
            session = _session_id(attrs)
            if session is not None:
                found.add(session)
    if len(found) > 1:
        raise AgentCoreArmSessionAmbiguous(
            f"the trace's /invocations spans name {len(found)} runtime sessions "
            f"({', '.join(sorted(found))}), so its session is ambiguous"
        )
    return next(iter(found), None)

span_stamp

span_stamp(attrs)

stamped with both arm attributes, half with one of them, none with neither.

Source code in src/calibrated_agent_evaluation/agentcore/arms.py
def span_stamp(attrs: Mapping[str, Any]) -> Stamp:
    """``stamped`` with both arm attributes, ``half`` with one of them, ``none`` with neither."""
    present = sum(attrs.get(key) is not None for key in (ARM_ARN_ATTRIBUTE, ARM_VARIANT_ATTRIBUTE))
    return "stamped" if present == 2 else "half" if present == 1 else "none"

dataset_guard

dataset_guard()

The allowlist every publish and pull client carries: the dataset operations, nothing else.

It is a ReadOnlyGuard used as a plain allowlist: the listed operations include the dataset writes publish makes, and every other operation is refused before it is sent.

Source code in src/calibrated_agent_evaluation/agentcore/datasets.py
def dataset_guard() -> ReadOnlyGuard:
    """The allowlist every publish and pull client carries: the dataset operations, nothing else.

    It is a :class:`ReadOnlyGuard` used as a plain allowlist: the listed operations include the
    dataset writes ``publish`` makes, and every other operation is refused before it is sent.
    """
    return ReadOnlyGuard(DATASET_OPERATIONS)

plan_publish

plan_publish(clients, slice_, name)

The publish plan for slice_ under name; checks that need no call come first.

Source code in src/calibrated_agent_evaluation/agentcore/datasets.py
def plan_publish(clients: AwsClients, slice_: DatasetSlice, name: str) -> DatasetPlan:
    """The publish plan for ``slice_`` under ``name``; checks that need no call come first."""
    validate_dataset_name(name)
    n = len(slice_.scenarios)
    if n > MAX_EXAMPLES_PER_REQUEST:
        raise DatasetPublishError(
            f"the slice holds {n} scenarios, more than the {MAX_EXAMPLES_PER_REQUEST} examples "
            "AgentCore accepts in one request; split the slice (S3 uploads are not supported yet)"
        )
    region = _region(clients)
    schema = schema_for_form(slice_.form)
    digest = content_digest(slice_)
    control = clients.client(_SERVICE)
    existing = _find_dataset(control, name)
    if existing is None:
        next_version, existing_id, unchanged = "1", None, False
    else:
        existing_id = str(existing["datasetId"])
        if existing.get("schemaType") != schema:
            raise DatasetNameConflict(
                f"dataset {name!r} ({existing_id}) holds {existing.get('schemaType')} examples, "
                f"but this slice is {schema}"
            )
        versions = [
            int(v["datasetVersion"])
            for page in _pages(control, "list_dataset_versions", datasetId=existing_id)
            for v in page.get("versions", [])
            if VERSION_PATTERN.fullmatch(str(v.get("datasetVersion", "")))
        ]
        pointer = slice_.published
        # A dataset with no version yet is what a first publish that stopped part way leaves
        # behind; publishing again picks it up instead of refusing the name.
        if versions and (pointer is None or pointer.dataset_id != existing_id):
            raise DatasetNameConflict(
                f"a dataset named {name!r} already exists ({existing_id}, "
                f"{existing.get('schemaType')}) and this slice was not published to it; choose "
                "another --name, or pull that dataset to start from it"
            )
        next_version = str(max(versions, default=0) + 1)
        unchanged = pointer is not None and versions != [] and pointer.content_digest == digest
    return DatasetPlan(
        name=name,
        region=region,
        schema_type=schema,
        n_scenarios=n,
        n_with_assertions=sum(1 for s in slice_.scenarios if s.assertions),
        n_with_trajectory=sum(
            1
            for s in slice_.scenarios
            if isinstance(s, PredefinedScenario) and s.expected_trajectory
        ),
        digest=digest,
        existing_id=existing_id,
        next_version=next_version,
        unchanged=unchanged,
    )

publish

publish(
    clients,
    slice_,
    plan,
    *,
    poll_seconds=2.0,
    max_wait_seconds=120.0,
    sleep=time.sleep,
    clock=time.monotonic,
)

Create or version the dataset the plan names and return where the slice now lives.

Source code in src/calibrated_agent_evaluation/agentcore/datasets.py
def publish(
    clients: AwsClients,
    slice_: DatasetSlice,
    plan: DatasetPlan,
    *,
    poll_seconds: float = 2.0,
    max_wait_seconds: float = 120.0,
    sleep: Callable[[float], None] = time.sleep,
    clock: Callable[[], float] = time.monotonic,
) -> PublishedVersion:
    """Create or version the dataset the plan names and return where the slice now lives."""
    if plan.unchanged and slice_.published is not None:
        return slice_.published
    control = clients.client(_SERVICE)
    examples = [encode_example(s) for s in slice_.scenarios]
    source = {"inlineExamples": {"examples": examples}}

    def wait(
        dataset_id: str,
        step: str,
        *,
        draft_touched: bool,
        count: int | None = None,
        unmodified: bool = False,
    ) -> dict[str, Any]:
        return _wait_active(
            control,
            dataset_id,
            step,
            draft_touched,
            count,
            unmodified,
            poll_seconds,
            max_wait_seconds,
            sleep,
            clock,
        )

    if plan.existing_id is None:
        created = _call(
            "CreateDataset",
            control.create_dataset,
            datasetName=plan.name,
            schemaType=plan.schema_type,
            source=source,
            clientToken=_token(),
        )
        dataset_id = str(created["datasetId"])
        try:
            wait(dataset_id, "creating the dataset", draft_touched=False, count=len(examples))
        except _WaitFailed:
            raise
        except AgentCoreError as exc:
            raise DatasetPublishError(
                f"{exc}; dataset {dataset_id} was created but holds no version yet; publish "
                "again to finish (the same --name picks it up)"
            ) from exc
    else:
        dataset_id = plan.existing_id
        draft_ids = [
            str(e["exampleId"])
            for page in _pages(
                control, "list_dataset_examples", datasetId=dataset_id, datasetVersion="DRAFT"
            )
            for e in page.get("examples", [])
            if isinstance(e, dict) and "exampleId" in e
        ]
        try:
            for chunk in _chunks(draft_ids, MAX_EXAMPLES_PER_REQUEST):
                _call(
                    "DeleteDatasetExamples",
                    control.delete_dataset_examples,
                    datasetId=dataset_id,
                    exampleIds=chunk,
                    clientToken=_token(),
                )
            if draft_ids:
                wait(dataset_id, "clearing the draft", draft_touched=True, count=0)
            _call(
                "AddDatasetExamples",
                control.add_dataset_examples,
                datasetId=dataset_id,
                source=source,
                clientToken=_token(),
            )
            wait(dataset_id, "adding the slice's examples", draft_touched=True, count=len(examples))
        except _WaitFailed:
            raise
        except AgentCoreError as exc:
            raise DatasetPublishError(
                f"{exc}; dataset {dataset_id}'s draft may now differ from its last version, which "
                "is unchanged; publish again to finish"
            ) from exc

    try:
        versioned = _call(
            "CreateDatasetVersion",
            control.create_dataset_version,
            datasetId=dataset_id,
            clientToken=_token(),
        )
    except AgentCoreError as exc:
        raise DatasetPublishError(
            f"{exc}; dataset {dataset_id} holds the slice's examples in its draft but no new "
            "version was created; publish again to finish (the same --name picks it up)"
        ) from exc
    # Done only when the draft matches the new version; a poll right after the call can still
    # show the old draft as MODIFIED, so keep polling until the bound instead of failing on it.
    wait(dataset_id, "creating the version", draft_touched=True, unmodified=True)
    return PublishedVersion(
        dataset_arn=str(versioned["datasetArn"]),
        dataset_id=dataset_id,
        dataset_name=plan.name,
        version=str(versioned["datasetVersion"]),
        region=plan.region,
        content_digest=plan.digest,
        # Read back from the version list, as pull does: CreateDatasetVersion's own createdAt is
        # a few seconds earlier than the time AgentCore lists for the finished version.
        published_at=_version_time(
            _version_created(control, dataset_id, str(versioned["datasetVersion"]))
        ),
    )

pull

pull(clients, dataset_id, version)

The slice one published version of a dataset holds, decoded strictly.

Source code in src/calibrated_agent_evaluation/agentcore/datasets.py
def pull(clients: AwsClients, dataset_id: str, version: str) -> DatasetSlice:
    """The slice one published version of a dataset holds, decoded strictly."""
    validate_version(version)
    region = _region(clients)
    control = clients.client(_SERVICE)
    got = _call("GetDataset", control.get_dataset, datasetId=dataset_id, datasetVersion=version)
    schema = str(got.get("schemaType", ""))
    form = form_for_schema(schema)
    documents = [
        doc
        for page in _pages(
            control,
            "list_dataset_examples",
            datasetId=dataset_id,
            datasetVersion=version,
            PaginationConfig={"PageSize": MAX_EXAMPLES_PER_REQUEST},
        )
        for doc in page.get("examples", [])
    ]
    if not documents:
        raise DatasetDecodeError(
            f"dataset {dataset_id} version {version} has no examples, and a slice needs one"
        )
    scenarios = [decode_example(doc, schema) for doc in documents]
    # The version's own creation time, the one publish recorded; the dataset's updatedAt moves
    # every time a later version is published.
    version_time = _version_created(control, dataset_id, version)
    try:
        slice_ = DatasetSlice(form=form, scenarios=scenarios)
        pointer = PublishedVersion(
            dataset_arn=str(got["datasetArn"]),
            dataset_id=dataset_id,
            dataset_name=str(got["datasetName"]),
            version=version,
            region=region,
            content_digest=content_digest(slice_),
            published_at=_version_time(version_time),
        )
        return DatasetSlice(form=form, scenarios=slice_.scenarios, published=pointer)
    except ScenarioError as exc:
        raise DatasetDecodeError(f"dataset {dataset_id} version {version}: {exc}") from exc

content_digest

content_digest(slice_)

A SHA-256 over the slice's encoded examples, so an unchanged slice can be recognized.

Source code in src/calibrated_agent_evaluation/agentcore/datasets_codec.py
def content_digest(slice_: DatasetSlice) -> str:
    """A SHA-256 over the slice's encoded examples, so an unchanged slice can be recognized."""
    canonical = json.dumps(
        {"form": slice_.form, "examples": [encode_example(s) for s in slice_.scenarios]},
        sort_keys=True,
        separators=(",", ":"),
        ensure_ascii=False,
    )
    return hashlib.sha256(canonical.encode("utf-8")).hexdigest()

decode_example

decode_example(doc, schema_type)

The scenario an AgentCore example document holds, refusing any field that does not map.

Source code in src/calibrated_agent_evaluation/agentcore/datasets_codec.py
def decode_example(doc: object, schema_type: str) -> Scenario:
    """The scenario an AgentCore example document holds, refusing any field that does not map."""
    form = form_for_schema(schema_type)
    if not isinstance(doc, dict):
        raise DatasetDecodeError(
            f"a dataset example must be a JSON object, got {type(doc).__name__}"
        )
    scenario_id = doc.get("scenario_id")
    where = f"scenario {scenario_id!r}" if isinstance(scenario_id, str) else "a dataset example"
    allowed = PREDEFINED_FIELDS if form == "predefined" else SIMULATED_FIELDS
    _refuse_unknown(doc, allowed | SERVICE_FIELDS, where, schema_type)
    fields = {k: v for k, v in doc.items() if k not in SERVICE_FIELDS}

    metadata = fields.pop("metadata", None)
    metadata = {} if metadata is None else metadata
    if not isinstance(metadata, dict):
        raise DatasetDecodeError(f"{where}: metadata must be a JSON object")
    metadata = dict(metadata)
    meta_raw = metadata.pop(CALIBRATED_AGENT_EVALUATION_METADATA_KEY, None)
    try:
        meta = CalibratedAgentEvaluationScenarioMeta.model_validate(
            {} if meta_raw is None else meta_raw
        )
        if form == "predefined":
            for turn in fields.get("turns") or []:
                if isinstance(turn, dict):
                    _refuse_unknown(turn, TURN_FIELDS, f"{where} turn", schema_type)
            return PredefinedScenario.model_validate(
                {**fields, "metadata": metadata, "calibrated_agent_evaluation": meta}
            )
        actor = fields.get("actor_profile")
        if isinstance(actor, dict):
            _refuse_unknown(actor, ACTOR_FIELDS, f"{where} actor_profile", schema_type)
        return SimulatedScenario.model_validate(
            {**fields, "metadata": metadata, "calibrated_agent_evaluation": meta}
        )
    except ScenarioError as exc:
        raise DatasetDecodeError(f"{where} does not map to a {form} scenario: {exc}") from exc

encode_example

encode_example(scenario)

The AgentCore example document for scenario; empty optional fields are left out.

Source code in src/calibrated_agent_evaluation/agentcore/datasets_codec.py
def encode_example(scenario: Scenario) -> dict[str, Any]:
    """The AgentCore example document for ``scenario``; empty optional fields are left out."""
    doc: dict[str, Any] = {"scenario_id": scenario.scenario_id}
    if isinstance(scenario, PredefinedScenario):
        doc["turns"] = [
            {"input": t.input}
            | ({} if t.expected_response is None else {"expected_response": t.expected_response})
            for t in scenario.turns
        ]
        if scenario.expected_trajectory:
            doc["expected_trajectory"] = list(scenario.expected_trajectory)
    else:
        actor: dict[str, Any] = {
            "context": scenario.actor_profile.context,
            "goal": scenario.actor_profile.goal,
        }
        if scenario.actor_profile.traits:
            actor["traits"] = dict(scenario.actor_profile.traits)
        doc["actor_profile"] = actor
        doc["input"] = scenario.input
        doc["max_turns"] = scenario.max_turns
        if scenario.scenario_description:
            doc["scenario_description"] = scenario.scenario_description
    if scenario.assertions:
        doc["assertions"] = list(scenario.assertions)
    metadata = dict(scenario.metadata)
    if not scenario.calibrated_agent_evaluation.is_empty():
        metadata[CALIBRATED_AGENT_EVALUATION_METADATA_KEY] = (
            scenario.calibrated_agent_evaluation.model_dump(mode="json", exclude_defaults=True)
        )
    if metadata:
        doc["metadata"] = metadata
    return doc

detector_bedrock_model

detector_bedrock_model(
    model_id,
    *,
    region=None,
    timeout_seconds=_DEFAULT_TIMEOUT_SECONDS,
    temperature=None,
)

A Bedrock model for the detector or the namer, built only when a live call needs it.

The BedrockModel and its client are built on the first live call, with connect and read timeouts of timeout_seconds each, so a cassette replay builds no client, resolves no AWS credentials and opens no connection. The client sends each request once, whatever retries the environment or the AWS profile configure, so one live call is one billed ConverseStream request; a throttled or failed call raises instead of being retried.

Parameters:

Name Type Description Default
model_id str

The Bedrock model id, for example "global.anthropic.claude-sonnet-4-6".

required
region str | None

The AWS region of the Bedrock runtime; None uses the configured default.

None
timeout_seconds float

The connect and read timeout of the Bedrock client, in seconds.

_DEFAULT_TIMEOUT_SECONDS
temperature float | None

The sampling temperature; None leaves it out of the request.

None

Returns:

Type Description
Model

A strands Model whose configuration is {"model_id": model_id}, plus

Model

temperature when one is given.

Raises:

Type Description
DetectorConfigError

timeout_seconds is not a positive, finite number.

Source code in src/calibrated_agent_evaluation/agentcore/detectors/models.py
def detector_bedrock_model(
    model_id: str,
    *,
    region: str | None = None,
    timeout_seconds: float = _DEFAULT_TIMEOUT_SECONDS,
    temperature: float | None = None,
) -> Model:
    """A Bedrock model for the detector or the namer, built only when a live call needs it.

    The ``BedrockModel`` and its client are built on the first live call, with connect and read
    timeouts of ``timeout_seconds`` each, so a cassette replay builds no client, resolves no AWS
    credentials and opens no connection. The client sends each request once, whatever retries the
    environment or the AWS profile configure, so one live call is one billed ``ConverseStream``
    request; a throttled or failed call raises instead of being retried.

    Args:
        model_id: The Bedrock model id, for example ``"global.anthropic.claude-sonnet-4-6"``.
        region: The AWS region of the Bedrock runtime; ``None`` uses the configured default.
        timeout_seconds: The connect and read timeout of the Bedrock client, in seconds.
        temperature: The sampling temperature; ``None`` leaves it out of the request.

    Returns:
        A strands ``Model`` whose configuration is ``{"model_id": model_id}``, plus
        ``temperature`` when one is given.

    Raises:
        DetectorConfigError: ``timeout_seconds`` is not a positive, finite number.
    """
    if (
        isinstance(timeout_seconds, bool)
        or not isinstance(timeout_seconds, int | float)
        or not math.isfinite(timeout_seconds)
        or timeout_seconds <= 0
    ):
        raise DetectorConfigError(
            f"timeout_seconds={timeout_seconds!r} is not a usable timeout: pass a positive number "
            f"of seconds (the default is {_DEFAULT_TIMEOUT_SECONDS:g})"
        )
    # A null temperature is left out rather than sent: the models that refuse an explicit
    # temperature refuse a null one too.
    decoding: dict[str, Any] = {} if temperature is None else {"temperature": temperature}

    def build() -> Model:
        # Bounded connect and read, so an unreachable Bedrock endpoint raises within the window
        # instead of waiting on an SDK default that can change between releases. One attempt,
        # so one live call is one billed request: each SDK retry would be another, uncounted by
        # max_live_calls. The mode is named: left out, it would come from the environment or
        # the AWS profile.
        client_config = BotocoreConfig(
            connect_timeout=timeout_seconds,
            read_timeout=timeout_seconds,
            retries={"mode": "standard", "total_max_attempts": 1},
        )
        return BedrockModel(
            boto_client_config=client_config, region_name=region, model_id=model_id, **decoding
        )

    return DeferredModel(build, {"model_id": model_id, **decoding})

trace_to_session

trace_to_session(trace)

Convert a normalized trace into the vendor session the failure detector reads.

The session id, the one vendor trace's id and every span's trace and session id are the trace id. Spans are taken in seq order. Each becomes an agent-invocation span with the same id and parent, its canonical-JSON input and output as prompt and response, and the distinct names of the tools it called, in first-call order, as its available tools. Each tool call follows as a tool-execution span under it, with its arguments rebuilt with sorted keys and its canonical-JSON result.

When every span records both started_at and ended_at, those times are used, and each tool span takes its agent span's times. Otherwise every span gets placeholder times that encode only the seq order: agent span k (counted from 0) starts 2k seconds and ends 2k+1 seconds after 1970-01-01T00:00:00Z, and each tool span takes its agent span's interval.

The trace is checked before anything is built, so a refused trace never reaches a model.

Parameters:

Name Type Description Default
trace Trace

The normalized trace.

required

Returns:

Type Description
ConvertedSession

The converted session, which time source it used, and the map from each vendor span id

ConvertedSession

back to its Calibrated Agent Evaluation span and tool call.

Raises:

Type Description
SessionConversionError

The trace is probed or has no spans; a span's time does not parse as ISO-8601 or has no timezone (checked even when another span's missing time would send the trace to placeholders); a span ends before it starts; a tool span id the converter would build is already the id of another span; or a span's input, output, tool arguments or tool result is not JSON, or the detector's span types refuse it.

Source code in src/calibrated_agent_evaluation/agentcore/detectors/session.py
def trace_to_session(trace: Trace) -> ConvertedSession:
    """Convert a normalized trace into the vendor session the failure detector reads.

    The session id, the one vendor trace's id and every span's trace and session id are the
    trace id. Spans are taken in ``seq`` order. Each becomes an agent-invocation span with the
    same id and parent, its canonical-JSON input and output as prompt and response, and the
    distinct names of the tools it called, in first-call order, as its available tools. Each
    tool call follows as a tool-execution span under it, with its arguments rebuilt with sorted
    keys and its canonical-JSON result.

    When every span records both ``started_at`` and ``ended_at``, those times are used, and each
    tool span takes its agent span's times. Otherwise every span gets placeholder times that
    encode only the ``seq`` order: agent span ``k`` (counted from 0) starts ``2k`` seconds and
    ends ``2k+1`` seconds after ``1970-01-01T00:00:00Z``, and each tool span takes its agent
    span's interval.

    The trace is checked before anything is built, so a refused trace never reaches a model.

    Args:
        trace: The normalized trace.

    Returns:
        The converted session, which time source it used, and the map from each vendor span id
        back to its Calibrated Agent Evaluation span and tool call.

    Raises:
        SessionConversionError: The trace is probed or has no spans; a span's time does not
            parse as ISO-8601 or has no timezone (checked even when another span's missing time
            would send the trace to placeholders); a span ends before it starts; a tool span
            id the converter would build is already the id of another span; or a span's input,
            output, tool arguments or tool result is not JSON, or the detector's span types
            refuse it.
    """
    _require_owned_spans(trace)
    spans = sorted(trace.spans, key=lambda span: span.seq)
    _require_distinct_tool_span_ids(trace.trace_id, spans)
    times, intervals = _span_times(trace.trace_id, spans)
    vendor_spans: list[SpanUnion] = []
    locations: dict[str, tuple[str, int | None]] = {}
    tool_names: dict[str, str] = {}
    for span, interval in zip(spans, intervals, strict=True):
        vendor_spans.append(_agent_span(trace.trace_id, span, interval, times))
        locations[span.span_id] = (span.span_id, None)
        for index in range(len(span.tool_calls)):
            tool_span = _tool_span(trace.trace_id, span, index, interval, times)
            tool_span_id = _tool_span_id(span.span_id, index)
            vendor_spans.append(tool_span)
            locations[tool_span_id] = (span.span_id, index)
            tool_names[tool_span_id] = tool_span.tool_call.name
    vendor_trace = StrandsTrace(
        spans=vendor_spans, trace_id=trace.trace_id, session_id=trace.trace_id
    )
    return ConvertedSession(
        session=Session(traces=[vendor_trace], session_id=trace.trace_id),
        times=times,
        locations=MappingProxyType(locations),
        tool_names=MappingProxyType(tool_names),
    )

failure_vocabulary

failure_vocabulary()

The installed detector's failure categories in table order, then other.

The template is read on every call. The other entry has no parent and no examples; its definition is the text of the template's guideline line that starts - **Use "other"**:.

Returns:

Type Description
FailureVocabulary

The vocabulary, with the detector version, template version and template hash.

Raises:

Type Description
VocabularyParseError

The category table breaks a row check (see parse_categories), or the guidelines have no line for other or one with no text after its start; the error then names the row other would take, one past the table's last.

AgentCoreError

The installed detector package reports no version, or its template lacks a part the reader needs or holds one of another type.

Source code in src/calibrated_agent_evaluation/agentcore/detectors/vocabulary.py
def failure_vocabulary() -> FailureVocabulary:
    """The installed detector's failure categories in table order, then ``other``.

    The template is read on every call. The ``other`` entry has no parent and no examples; its
    definition is the text of the template's guideline line that starts
    ``- **Use "other"**:``.

    Returns:
        The vocabulary, with the detector version, template version and template hash.

    Raises:
        VocabularyParseError: The category table breaks a row check (see
            :func:`parse_categories`), or the guidelines have no line for ``other`` or one with
            no text after its start; the error then names the row ``other`` would take, one
            past the table's last.
        AgentCoreError: The installed detector package reports no version, or its template
            lacks a part the reader needs or holds one of another type.
    """
    template = _installed_template()
    categories = _text(template, "CATEGORIES")
    guidelines = _text(template, "DEFAULT_ANNOTATION_GUIDELINES")
    rows = parse_categories(categories)
    other = VocabularyEntry(
        name=_OTHER_NAME,
        parent=None,
        definition=_other_guideline(guidelines, row=len(rows) + 1),
        examples="",
    )
    provenance = _provenance_of(template)
    return FailureVocabulary(
        entries=(*rows, other),
        detector_version=provenance.detector_version,
        prompt_template=provenance.prompt_template,
        prompt_sha256=provenance.prompt_sha256,
    )

read_only_guard

read_only_guard(checks)

The guard every doctor client carries.

Source code in src/calibrated_agent_evaluation/agentcore/doctor.py
def read_only_guard(checks: Sequence[Check]) -> ReadOnlyGuard:
    """The guard every doctor client carries."""
    return ReadOnlyGuard(allowed_operations(checks))

run_checks

run_checks(checks, clients, target)

Confirm the identity, then run every check, isolating each one's AWS failures.

Source code in src/calibrated_agent_evaluation/agentcore/doctor.py
def run_checks(checks: Sequence[Check], clients: AwsClients, target: DoctorTarget) -> DoctorReport:
    """Confirm the identity, then run every check, isolating each one's AWS failures."""
    identity = clients.caller_identity()
    results = tuple(_run_one(check, clients, target) for check in checks)
    return DoctorReport(target=target, identity=identity, results=results)

ingest_agentcore

ingest_agentcore(
    source,
    *,
    backend,
    dry_run=False,
    require_owned=False,
    with_scores=False,
    frame=None,
)

Pull AgentCore traces into backend; optionally read evaluator scores as advisory labels.

The traces go through the same gate and report as any pulled source. A mixed-session trace the source re-keyed is noted on its result. When with_scores is set, evaluator results are mapped to advisory llm-judge labels and persisted, skipping any whose identity is already stored (so re-reading the same results adds no duplicate). frame stamps every pulled trace with that sampling frame, as ingest_langfuse does.

Source code in src/calibrated_agent_evaluation/agentcore/ingest.py
def ingest_agentcore(
    source: AgentCoreReadSource,
    *,
    backend: TraceStoreBackend,
    dry_run: bool = False,
    require_owned: bool = False,
    with_scores: bool = False,
    frame: Frame | None = None,
) -> IngestReport:
    """Pull AgentCore traces into ``backend``; optionally read evaluator scores as advisory labels.

    The traces go through the same gate and report as any pulled source. A mixed-session trace the
    source re-keyed is noted on its result. When ``with_scores`` is set, evaluator results are
    mapped to advisory ``llm-judge`` labels and persisted, skipping any whose identity is already
    stored (so re-reading the same results adds no duplicate). ``frame`` stamps every pulled trace
    with that sampling frame, as :func:`~calibrated_agent_evaluation.ingest.ingest_langfuse` does.
    """
    report = ingest_langfuse(
        source,
        backend=backend,
        dry_run=dry_run,
        require_owned=require_owned,
        source_kind="agentcore",
        frame=frame,
    )
    rekeyed = dict(source.rekeyed)  # original trace id -> runtime session id it was stored under
    report = _note_rekeys(report, rekeyed)
    if with_scores and not dry_run:
        ingested_ids = {
            r.trace_id for r in report.results if r.status == "ingested" and r.trace_id is not None
        }
        _ingest_scores(source, backend=backend, ingested_ids=ingested_ids, rekeyed=rekeyed)
    return report

audit_clients

audit_clients(region=None, profile=None)

AWS clients that can make only the audit's reads and the AgentCore ingest's trace reads.

The billed-call budget comes from CALIBRATED_AGENT_EVALUATION_AGENTCORE_MAX_BILLED_CALLS when it is set.

Raises:

Type Description
AgentCoreError

The profile does not exist or cannot be opened, or the budget variable is not a non-negative whole number.

Source code in src/calibrated_agent_evaluation/agentcore/optimization.py
def audit_clients(region: str | None = None, profile: str | None = None) -> AwsClients:
    """AWS clients that can make only the audit's reads and the AgentCore ingest's trace reads.

    The billed-call budget comes from ``CALIBRATED_AGENT_EVALUATION_AGENTCORE_MAX_BILLED_CALLS``
    when it is set.

    Raises:
        AgentCoreError: The profile does not exist or cannot be opened, or the budget variable
            is not a non-negative whole number.
    """
    # Through the package's lazy attribute, which turns a missing AWS SDK into
    # AgentCoreExtraMissing instead of a bare ModuleNotFoundError.
    from calibrated_agent_evaluation import agentcore

    guard = ReadOnlyGuard(AUDIT_READ_OPERATIONS | INGEST_READ_OPERATIONS)
    budget = BilledCallBudget.from_env(os.environ)
    return agentcore.AwsClients.from_profile(
        region=region, profile=profile, handlers=[guard, budget]
    )

read_ab_test

read_ab_test(clients, ab_test_id, *, now=None)

Read an A/B test and each arm's bundle version as a vendor-claim record.

Every variant must name exactly one of a bundle version and a gateway target; the model allows both or neither, so such a response is refused before any bundle version is read. A target arm is kept by name and reads no bundle.

Parameters:

Name Type Description Default
clients AwsClients

The audit's clients.

required
ab_test_id str

The A/B test to read.

required
now datetime | None

The read time, for tests; the current UTC time otherwise.

None

Returns:

Type Description
AbTestClaimRecord

The claim record: the test's state, both arms, AgentCore's per-evaluator figures as

AbTestClaimRecord

returned, whether the two bundle arms are identical, and whether the test is still running.

Raises:

Type Description
AgentCoreError

A call failed, naming the operation and the id, or a variant names both or neither configuration.

Source code in src/calibrated_agent_evaluation/agentcore/optimization.py
@_shaped("GetABTest")
def read_ab_test(
    clients: AwsClients, ab_test_id: str, *, now: datetime | None = None
) -> AbTestClaimRecord:
    """Read an A/B test and each arm's bundle version as a vendor-claim record.

    Every variant must name exactly one of a bundle version and a gateway target; the model
    allows both or neither, so such a response is refused before any bundle version is read. A
    target arm is kept by name and reads no bundle.

    Args:
        clients: The audit's clients.
        ab_test_id: The A/B test to read.
        now: The read time, for tests; the current UTC time otherwise.

    Returns:
        The claim record: the test's state, both arms, AgentCore's per-evaluator figures as
        returned, whether the two bundle arms are identical, and whether the test is still running.

    Raises:
        AgentCoreError: A call failed, naming the operation and the id, or a variant names both
            or neither configuration.
    """
    from calibrated_agent_evaluation._provenance import capture_provenance

    response = _call(
        "GetABTest", ab_test_id, clients.client(_DATA).get_ab_test, abTestId=ab_test_id
    )
    raw_variants = list(response.get("variants") or [])
    for variant in raw_variants:
        config = variant.get("variantConfiguration") or {}
        if ("configurationBundle" in config) == ("target" in config):
            names = (
                "both a bundle version and a target"
                if config
                else "neither a bundle version nor a target"
            )
            raise AgentCoreError(
                f"GetABTest for {ab_test_id!r}: variant {variant.get('name')!r} names {names}; "
                "an arm must name exactly one, so the test is refused"
            )
    variants: list[VariantRef] = []
    for variant in raw_variants:
        config = variant["variantConfiguration"]
        bundle_ref = config.get("configurationBundle")
        target = config.get("target")
        variants.append(
            VariantRef(
                name=variant.get("name"),
                weight=int(variant.get("weight", 0)),
                bundle=(
                    read_bundle_version(
                        clients, str(bundle_ref["bundleArn"]), str(bundle_ref["bundleVersion"])
                    )
                    if bundle_ref
                    else None
                ),
                target_name=str(target["name"]) if target else None,
            )
        )
    results = response.get("results") or {}
    claims = tuple(
        EvaluatorClaim(
            evaluator_arn=str(metric.get("evaluatorArn")),
            control=_stats(metric.get("controlStats") or {}),
            variants=tuple(_stats(v) for v in metric.get("variantResults") or ()),
        )
        for metric in results.get("evaluatorMetrics") or ()
    )
    bundles = [v.bundle for v in variants if v.bundle is not None]
    identical = len(bundles) == 2 and bundles[0].components_sha256 == bundles[1].components_sha256
    execution_status = str(response.get("executionStatus"))
    read_at = (now or datetime.now(tz=UTC)).astimezone(UTC).isoformat()
    return AbTestClaimRecord(
        record_id=f"abtest-{uuid.uuid4().hex[:16]}",
        ab_test_arn=str(response.get("abTestArn")),
        ab_test_id=str(response.get("abTestId") or ab_test_id),
        name=str(response.get("name")),
        status=str(response.get("status")),
        execution_status=execution_status,
        gateway_arn=str(response.get("gatewayArn")),
        variants=tuple(sorted(variants, key=lambda v: v.name)),  # type: ignore[arg-type]
        evaluation_config_arns=_evaluation_config_arns(response.get("evaluationConfig") or {}),
        current_run_id=response.get("currentRunId"),
        error_details=tuple(str(d) for d in response.get("errorDetails") or ()),
        started_at=_iso(response.get("startedAt")),
        stopped_at=_iso(response.get("stoppedAt")),
        max_duration_expires_at=_iso(response.get("maxDurationExpiresAt")),
        analysis_timestamp=_iso(results.get("analysisTimestamp")),
        updated_at=_iso(response.get("updatedAt")) or "",
        evaluator_claims=claims,
        identical_arms=identical,
        provisional=execution_status in _PROVISIONAL,
        read_at=read_at,
        provenance=capture_provenance(now=now),
    )

read_bundle_version

read_bundle_version(clients, bundle_arn, version_id)

Read one configuration bundle version, keeping its components as digests only.

Parameters:

Name Type Description Default
clients AwsClients

The audit's clients.

required
bundle_arn str

The bundle's ARN; its last path segment is the bundle id.

required
version_id str

The version to read.

required

Returns:

Type Description
BundleVersionRef

The version's lineage and a digest of each component, of each top-level key of a

BundleVersionRef

component whose configuration is a JSON object, and of the whole components map.

Raises:

Type Description
AgentCoreError

The call failed, naming GetConfigurationBundleVersion and the version.

Source code in src/calibrated_agent_evaluation/agentcore/optimization.py
@_shaped("GetConfigurationBundleVersion")
def read_bundle_version(clients: AwsClients, bundle_arn: str, version_id: str) -> BundleVersionRef:
    """Read one configuration bundle version, keeping its components as digests only.

    Args:
        clients: The audit's clients.
        bundle_arn: The bundle's ARN; its last path segment is the bundle id.
        version_id: The version to read.

    Returns:
        The version's lineage and a digest of each component, of each top-level key of a
        component whose configuration is a JSON object, and of the whole components map.

    Raises:
        AgentCoreError: The call failed, naming ``GetConfigurationBundleVersion`` and the version.
    """
    bundle_id = bundle_arn.rsplit("/", 1)[-1]
    response = _call(
        "GetConfigurationBundleVersion",
        f"{bundle_id}/{version_id}",
        clients.client(_CONTROL).get_configuration_bundle_version,
        bundleId=bundle_id,
        versionId=version_id,
    )
    components: dict[str, Any] = dict(response.get("components") or {})
    lineage: dict[str, Any] = dict(response.get("lineageMetadata") or {})
    created_by = lineage.get("createdBy") or {}
    return BundleVersionRef(
        bundle_arn=str(response.get("bundleArn") or bundle_arn),
        version_id=str(response.get("versionId") or version_id),
        version_created_at=_iso(response.get("versionCreatedAt")) or "",
        parent_version_ids=tuple(str(v) for v in lineage.get("parentVersionIds") or ()),
        branch_name=lineage.get("branchName"),
        created_by=created_by.get("name") if isinstance(created_by, Mapping) else None,
        components=tuple(
            _component_digest(cid, (components[cid] or {}).get("configuration"))
            for cid in sorted(components)
        ),
        components_sha256=sha256_text(canonical_json(components)),
    )

agent_reported_error_trial

agent_reported_error_trial(exc)

The trial an agent-reported error stands for, or None for any other error.

Source code in src/calibrated_agent_evaluation/agentcore/policy_spans.py
def agent_reported_error_trial(exc: Exception) -> AgentErrorTrial | None:
    """The trial an agent-reported error stands for, or ``None`` for any other error."""
    if isinstance(exc, AgentReportedError) and exc.trace_id:
        return AgentErrorTrial(probe_trace_id=exc.trace_id, error_code=exc.error_code)
    return None

extract_decisions

extract_decisions(
    spans, vocabulary=AGENTCORE_POLICY_VOCABULARY
)

Every per-call Policy decision in one trace's span records, in the order the calls started.

Spans without the decision attribute are skipped, and so are tools/list authorizations and the Gateway's own call spans, which repeat the decision their per-call child records; a call span whose child is missing, or whose value is unknown, is refused, and one that says a log-only gateway decided asynchronously is skipped. A span the store holds more than once is read once.

Raises:

Type Description
PolicySpanMalformed

A span carries the decision attribute and is neither kind; or a per-call span's tool, decision or mode is missing or unknown, or a list attribute is in an unknown encoding. The message names the trace, the span and the attribute.

Source code in src/calibrated_agent_evaluation/agentcore/policy_spans.py
def extract_decisions(
    spans: Iterable[Mapping[str, Any]],
    vocabulary: PolicyVocabulary = AGENTCORE_POLICY_VOCABULARY,
) -> tuple[PolicyDecision, ...]:
    """Every per-call Policy decision in one trace's span records, in the order the calls started.

    Spans without the decision attribute are skipped, and so are ``tools/list`` authorizations
    and the Gateway's own call spans, which repeat the decision their per-call child records; a
    call span whose child is missing, or whose value is unknown, is refused, and one that says a
    log-only gateway decided asynchronously is skipped. A span the store holds more than once is
    read once.

    Raises:
        PolicySpanMalformed: A span carries the decision attribute and is neither kind; or a
            per-call span's tool, decision or mode is missing or unknown, or a list attribute is
            in an unknown encoding. The message names the trace, the span and the attribute.
    """
    # The span store holds two or three records of one span (seen live), so each span counts once.
    by_id: dict[object, Mapping[str, Any]] = {}
    for span in spans:
        if isinstance(span, Mapping):
            by_id.setdefault((span.get("traceId"), span.get("spanId")), span)
    records = list(by_id.values())
    by_id = {span.get("spanId"): span for span in records}
    found: list[tuple[int, PolicyDecision]] = []
    for span in records:
        attrs = _attributes(span)
        if vocabulary.decision_key not in attrs:
            continue
        where = f"span {span.get('spanId')!r} of trace {span.get('traceId')!r}"
        selector = attrs.get(vocabulary.selector_key)
        if selector == vocabulary.tool_list_value:
            continue
        if selector != vocabulary.per_call_value:
            if attrs.get(vocabulary.gateway_call_key) == vocabulary.gateway_call_value:
                _check_gateway_call(span, attrs, records, vocabulary, where)
                continue
            raise PolicySpanMalformed(
                f"{where} carries {vocabulary.decision_key} but is neither a per-call "
                f"authorization ({vocabulary.selector_key}={vocabulary.per_call_value}) nor a "
                f"tools/list one ({vocabulary.tool_list_value}); it is refused rather than counted "
                "or dropped" + _SPAN_FORMAT_HINT
            )
        parent = by_id.get(span.get("parentSpanId"))
        tool = _attributes(parent).get(vocabulary.tool_key) if parent is not None else None
        if not isinstance(tool, str) or not tool:
            raise PolicySpanMalformed(
                f"{where} holds a Policy decision, but its parent Gateway span carries no "
                f"{vocabulary.tool_key}, so the decision can't be tied to a tool"
                + _SPAN_FORMAT_HINT
            )
        decision = vocabulary.decisions.get(str(attrs.get(vocabulary.decision_key)))
        mode = vocabulary.modes.get(str(attrs.get(vocabulary.mode_key)))
        if decision is None or mode is None:
            raise PolicySpanMalformed(
                f"{where} has {vocabulary.decision_key}={attrs.get(vocabulary.decision_key)!r} and "
                f"{vocabulary.mode_key}={attrs.get(vocabulary.mode_key)!r}; the decision must be "
                f"one of {sorted(vocabulary.decisions)} and the mode one of "
                f"{sorted(vocabulary.modes)}" + _SPAN_FORMAT_HINT
            )
        incomplete = attrs.get(vocabulary.incomplete_key, False)
        engine = attrs.get(vocabulary.engine_key)
        reason = attrs.get(vocabulary.reason_key)
        for key, value, wanted in (
            (vocabulary.incomplete_key, incomplete, bool),
            (vocabulary.engine_key, engine, (str, type(None))),
            (vocabulary.reason_key, reason, (str, type(None))),
        ):
            if not isinstance(value, wanted):
                raise PolicySpanMalformed(
                    f"{where} carries {key} as {type(value).__name__}; the decision is refused "
                    "rather than read with the value coerced" + _SPAN_FORMAT_HINT
                )
        try:
            record = PolicyDecision(
                trace_id=str(span.get("traceId") or "").lower(),
                span_id=str(span.get("spanId") or ""),
                tool=tool,
                decision=decision,
                mode=mode,
                determining_policies=_names(span, vocabulary.determining_key),
                log_only_flipping_policies=_names(span, vocabulary.flipping_key),
                log_only_incomplete=incomplete,
                engine=engine,
                reason=reason,
            )
        except ProbeError as exc:
            raise PolicySpanMalformed(f"{where} can't be read as a decision: {exc}") from exc
        started = span.get("startTimeUnixNano")
        found.append((started if isinstance(started, int) else 0, record))
    return tuple(record for _, record in sorted(found, key=lambda pair: pair[0]))

agentcore_source

agentcore_source(clients, config)

Build the trace source for config: the Omni reader, or the Logs Insights fallback.

Omni is used when an Omni domain is configured; otherwise spans are read from CloudWatch's aws/spans and content from the runtime's otel-rt-logs group via Logs Insights.

Source code in src/calibrated_agent_evaluation/agentcore/readers.py
def agentcore_source(clients: AwsClients, config: AgentCoreSourceConfig) -> AgentCoreTraceSource:
    """Build the trace source for ``config``: the Omni reader, or the Logs Insights fallback.

    Omni is used when an Omni domain is configured; otherwise spans are read from CloudWatch's
    ``aws/spans`` and content from the runtime's ``otel-rt-logs`` group via Logs Insights.
    """
    reader: OmniReader | LogsInsightsReader = (
        OmniReader(clients, config) if config.uses_omni else LogsInsightsReader(clients, config)
    )
    return AgentCoreTraceSource(reader, config)

read_scores

read_scores(results, *, job_id=None)

Map a stream of AgentCore evaluation results to advisory labels.

Raises, at the record that cannot be mapped, the error result_to_label raises: AgentCoreScoreIncomplete for a record with no trace or session id or no verdict, and AgentCoreNoPassDirection for an evaluator with no pass direction. Nothing is skipped silently; a caller that wants to skip such records maps each one with result_to_label and counts the ones it skips, as ingest_agentcore does.

Source code in src/calibrated_agent_evaluation/agentcore/scores.py
def read_scores(results: Iterable[dict[str, Any]], *, job_id: str | None = None) -> Iterator[Label]:
    """Map a stream of AgentCore evaluation results to advisory labels.

    Raises, at the record that cannot be mapped, the error :func:`result_to_label` raises:
    :class:`AgentCoreScoreIncomplete` for a record with no trace or session id or no verdict, and
    :class:`AgentCoreNoPassDirection` for an evaluator with no pass direction. Nothing is skipped
    silently; a caller that wants to skip such records maps each one with :func:`result_to_label`
    and counts the ones it skips, as
    :func:`calibrated_agent_evaluation.agentcore.ingest.ingest_agentcore` does.
    """
    for result in results:
        yield result_to_label(result, job_id=job_id)

caller_is_role

caller_is_role(caller_arn, role_arn)

Whether caller_arn is a session of the IAM role role_arn (same account and name).

STS reports a role session as arn:aws:sts::<account>:assumed-role/<name>/<session>; the role itself is arn:aws:iam::<account>:role/<path/><name>.

Source code in src/calibrated_agent_evaluation/agentcore/session.py
def caller_is_role(caller_arn: str, role_arn: str) -> bool:
    """Whether ``caller_arn`` is a session of the IAM role ``role_arn`` (same account and name).

    STS reports a role session as ``arn:aws:sts::<account>:assumed-role/<name>/<session>``; the
    role itself is ``arn:aws:iam::<account>:role/<path/><name>``.
    """
    caller = caller_arn.split(":", 5)
    role = role_arn.split(":", 5)
    if len(caller) != 6 or len(role) != 6:
        return False
    if caller[2] != "sts" or role[2] != "iam" or caller[4] != role[4] or caller[1] != role[1]:
        return False
    kind, _, rest = caller[5].partition("/")
    role_kind, _, role_path = role[5].partition("/")
    if kind != "assumed-role" or role_kind != "role":
        return False
    return rest.partition("/")[0] == role_path.rpartition("/")[2]

require_role

require_role(clients, role_arn)

The caller's ARN, after confirming it is a session of role_arn.

Only sts:GetCallerIdentity is called, so nothing else reaches the account under the wrong identity.

Source code in src/calibrated_agent_evaluation/agentcore/session.py
def require_role(clients: AwsClients, role_arn: str) -> str:
    """The caller's ARN, after confirming it is a session of ``role_arn``.

    Only ``sts:GetCallerIdentity`` is called, so nothing else reaches the account under the wrong
    identity.
    """
    caller = clients.caller_identity()
    if not caller_is_role(caller, role_arn):
        raise WrongAwsIdentity(
            f"running as {caller}, but these calls must run as the role {role_arn}; "
            "switch to a profile that assumes it"
        )
    return caller