calibrated_agent_evaluation.agentcore¶
calibrated_agent_evaluation.agentcore ¶
Amazon Bedrock AgentCore plug-in: opt-in adapters behind the library's existing seams.
Nothing here is a default, and importing this package loads no AWS SDK. Adapter names load on
first access; without the [agentcore] extra that access raises AgentCoreExtraMissing
naming the command that installs it.
The plug-in has six parts:
- the read-only account check behind
calibrated-agent-evaluation doctor --agentcore(run_checks); - the window ingest behind
calibrated-agent-evaluation ingest --source agentcore, which reads a runtime's traces for a time window from AgentCore Observability (AgentCoreTraceSource,ingest_agentcore); - the Runtime executor behind
calibrated-agent-evaluation generate --run --executor agentcore, which sends each generated input to an agent deployed to AgentCore Runtime and stores one trace per input (AgentCoreRuntimeExecutor), reading each run's span tree with the per-run span reader (AgentCoreSpanReader); - dataset publish and pull behind
calibrated-agent-evaluation dataset publishandpull, which publish the test scenarios built from the failures reviewers caught as a version of an AgentCore evaluation dataset, and read a version back (publish,pull); - the Code Interpreter sandbox behind
calibrated-agent-evaluation score --scorer code --sandbox agentcore, which runs an agent's generated code in a managed AgentCore Code Interpreter session (CodeInterpreterSandbox); - the
strands-agents-evalsfailure detector:failure_vocabularyreads the detector's failure categories as typed entries,FailureDetectorruns it on a normalized trace and returns aDetectionor a typed refusal, andDetectorDraftNamernames draft failure-mode clusters from that vocabulary forcalibrated_agent_evaluation.cluster.build_draft. Both call only the strandsModelthe caller passes, refuse a live call pastmax_live_calls(0 by default), and can record and replay their calls from a cassette. The result types and the detector errors are vendor-free, so they import without the extra.
ARM_ARN_ATTRIBUTE
module-attribute
¶
ARM_VARIANT_ATTRIBUTE
module-attribute
¶
DEFAULT_CHECKS
module-attribute
¶
DEFAULT_CHECKS = (
RegionCheck(),
TransactionSearchCheck(),
IndexingCheck(),
SpanLogGroupCheck(),
RuntimeLogGroupCheck(),
OmniDomainCheck(),
ContentExtractionCheck(),
)
AUDIT_READ_OPERATIONS
module-attribute
¶
AUDIT_READ_OPERATIONS = frozenset(
{
(_DATA, "GetABTest"),
(_DATA, "GetRecommendation"),
(_CONTROL, "GetConfigurationBundleVersion"),
(_CONTROL, "GetOnlineEvaluationConfig"),
}
)
INGEST_READ_OPERATIONS
module-attribute
¶
INGEST_READ_OPERATIONS = frozenset(
{
("logs", "StartQuery"),
("logs", "GetQueryResults"),
("logs", "FilterLogEvents"),
("cloudwatchomni", "StartTelemetryQuerySession"),
("cloudwatchomni", "StartTelemetryQuery"),
("cloudwatchomni", "GetTelemetryQueryResults"),
("cloudwatchomni", "StopTelemetryQuerySession"),
}
)
AGENTCORE_POLICY_VOCABULARY
module-attribute
¶
AGENTCORE_POLICY_VOCABULARY = PolicyVocabulary(
selector_key="rpc.method",
per_call_value="AuthorizeAction",
tool_list_value="PartiallyAuthorizeActions",
gateway_call_key="url.path",
gateway_call_value="tools/call",
tool_key="tool.name",
decision_key="aws.agentcore.policy.authorization_decision",
decisions={"ALLOW": "allow", "DENY": "deny"},
asynchronous_decision="LOG_ONLY_ASYNC",
mode_key="aws.agentcore.gateway.policy.mode",
modes={"ENFORCE": "enforce", "LOG_ONLY": "log_only"},
determining_key="aws.agentcore.policy.determining_policies",
flipping_key="aws.agentcore.policy.log_only_decision_flipping_policies",
incomplete_key="aws.agentcore.policy.log_only_eval_incomplete",
engine_key="aws.agentcore.gateway.policy.arn",
reason_key="aws.agentcore.policy.authorization_reason",
)
AgentCoreAppliedBundleMismatch ¶
Bases: AgentCoreError
A trace lacks the bundle attribute or value the operator required, or carries another.
AgentCoreArmHalfStamped ¶
Bases: AgentCoreError
A span carries one of the two arm attributes without the other.
A request whose baggage named one key stamps its spans by halves, so such a trace's arm cannot be read safely; the trace is refused.
AgentCoreArmSessionAmbiguous ¶
Bases: AgentCoreError
A trace's /invocations server spans name two runtime sessions.
AgentCoreArmSessionSplit ¶
Bases: AgentCoreError
One runtime session's traces sit in two variants, so the session's arm is ambiguous.
AgentCoreArmsMissing ¶
Bases: AgentCoreError
An A/B arms read found no trace it can assign to the audited test.
No span in the window carries both arm attributes, or no stamped trace names the audited experiment. Refused rather than stored as a read with every trace unassigned.
AgentCoreArmSourceIdRepeated ¶
Bases: AgentCoreError
Two or more traces arrived under one source id, so none of them can be assigned.
AgentCoreArmsReadChanged ¶
Bases: AgentCoreError
A re-read under a stored read id delivered other traces or assigned one differently.
A stored read never changes; the re-read is refused and nothing from it is written.
AgentCoreContentMissing ¶
Bases: AgentCoreError
A span's message content was not found in the joined otel-rt-logs records.
Raised rather than yielding a span with empty input/output/tool content: an under-joined trace reads as a complete one, which is the silent-loss failure this ingest exists to prevent.
AgentCoreError ¶
Bases: CalibratedAgentEvaluationDataError
An AgentCore integration call or configuration failed.
AgentCoreExtraMissing ¶
Bases: MissingExtraError, AgentCoreError
An AgentCore module was asked for but the [agentcore] extra is not installed.
Source code in src/calibrated_agent_evaluation/agentcore/errors.py
AgentCoreIndexingTooLow ¶
Bases: AgentCoreError
X-Ray indexes below 100% while a read goes through X-Ray or the trace list.
A read that depends on the index would see only the indexed fraction, so the result is an unknown subset. Refused, naming the observed percentage.
AgentCoreMixedArm ¶
Bases: AgentCoreError
One trace's spans name two experiments or two variants.
AgentCoreMixedSession ¶
Bases: AgentCoreError
A single trace carries more than one session.id.
The AgentCore Evaluate API rejects such input, and fusing the sessions would mislabel the
trace. Refused by default; an explicit re-key opts into collapsing them to the runtime session.
AgentCorePartialResult ¶
Bases: AgentCoreError
A CloudWatch Omni query returned partialResultsDetected.
The rows are an unknown subset of the match, so returning them would under-count silently. Refused rather than returned.
AgentCoreScoreIncomplete ¶
Bases: AgentCoreError
An evaluator result lacks a field a Label requires (trace/session id, or a value).
Also raised when an Evaluate call returns no result at all for the trace it targeted.
Refused rather than stored as a label with an empty or inferred field, or a run returning fewer
verdicts than traces, which would enter an unfounded or incomplete set into the store.
AgentCoreSettlingLag ¶
Bases: AgentCoreError
The requested window ends inside the read-after-write settling lag.
A trace written moments ago is not yet fully readable, so the window would return a partial trace. Refused rather than returned mid-write.
AgentCoreTruncatedResult ¶
Bases: AgentCoreError
A CloudWatch Logs Insights query matched more records than it returned.
recordsMatched exceeded the rows retrieved, so the trace set is incomplete. Refused rather
than returned truncated.
AwsCredentialsMissing ¶
Bases: AgentCoreError
No AWS credentials could be resolved for the requested profile.
BilledCallBudgetExceeded ¶
Bases: AgentCoreError
A billed AWS operation would exceed the billed-call budget; refused before sending.
operation is the refused call, (service, operation), or None when not named.
Source code in src/calibrated_agent_evaluation/agentcore/errors.py
DatasetDecodeError ¶
Bases: AgentCoreError
A dataset example has a field the codec does not map; refused rather than dropped.
DatasetInputError ¶
Bases: AgentCoreError
A dataset name or version given to publish or pull is one AgentCore can't accept.
DatasetNameConflict ¶
Bases: AgentCoreError
A dataset with the requested name exists but is not the one this slice was published to.
DatasetPublishError ¶
Bases: AgentCoreError
Publishing a slice could not complete; the message says what state the dataset is left in.
DatasetSchemaUnsupported ¶
Bases: AgentCoreError
A dataset uses a schema type the codec does not support.
DatasetVersionNotFound ¶
Bases: AgentCoreError
The requested dataset or dataset version does not exist (or was deleted).
DetectorCallError ¶
Bases: DetectorError
A model call made for the detector or the namer failed; the cause is chained.
DetectorCallLimitExceeded ¶
Bases: DetectorError
A live model call would exceed max_live_calls; refused before it was sent.
limit is the max_live_calls value that refused it.
Source code in src/calibrated_agent_evaluation/agentcore/errors.py
DetectorConfigError ¶
Bases: DetectorError
The detector or the namer was built with a configuration it cannot run under.
DetectorError ¶
Bases: AgentCoreError
The failure detector or the draft namer refused an input, a configuration or a response.
DetectorOutputError ¶
Bases: DetectorError
The detector's model returned a response that cannot be read faithfully.
DetectorSessionTooLarge ¶
Bases: DetectorError
A trace is too large for the detector's model to read in one call.
DraftNamingError ¶
Bases: DetectorError
The namer's response holds no usable vocabulary name for every cluster.
PolicySpanMalformed ¶
Bases: AgentCoreError
A Gateway span carrying a Policy decision could not be read whole.
Raised for a per-call decision span whose tool, decision or mode is missing or unknown, whose list attribute is in an unknown encoding, and for a span carrying the decision attribute that is neither a per-call nor a tools/list authorization. Guessing would add or drop a decision.
ReadOnlyViolation ¶
Bases: AgentCoreError
A read-only command tried an AWS operation outside its allowlist; refused before sending.
operation is the refused call, (service, operation), or None when not named.
Source code in src/calibrated_agent_evaluation/agentcore/errors.py
SessionConversionError ¶
Bases: DetectorError
A trace cannot be converted into the session the failure detector reads.
VocabularyParseError ¶
Bases: DetectorError
The detector's category table has a row the library cannot read as a vocabulary entry.
row is the number of the table row that failed and check names the check it failed.
Source code in src/calibrated_agent_evaluation/agentcore/errors.py
WrongAwsIdentity ¶
Bases: AgentCoreError
The credentials belong to a different principal than the one required.
ArmRead
dataclass
¶
One arms read: its id, one assignment per source id, its counts and the ARNs it saw.
KeptTrace
dataclass
¶
KeptTrace(
trace_id,
session_ids,
rekey_target,
spans_total,
spans_stamped,
spans_half_stamped,
log_groups=(),
content_missing=False,
experiment_arns=(),
)
What RawTraceLog keeps of one raw trace the ingest's reader yielded.
RawTraceLog ¶
Wraps one of the trace ingest's readers: each raw trace passes through, its counts kept.
The ingest's source drops a trace it refuses for a re-key collision, so the arm read takes what it needs from here: each trace's id, session ids, re-key target and span counts, and the log groups its spans name.
Source code in src/calibrated_agent_evaluation/agentcore/arms.py
read_traces ¶
Yield the wrapped reader's traces unchanged, keeping what the arm read needs.
Source code in src/calibrated_agent_evaluation/agentcore/arms.py
read_scores ¶
collisions ¶
Per runtime session id two or more traces re-keyed to, every trace that did.
A mixed-session trace absent from rekeyed was refused by the ingest because an earlier
trace already took its runtime session id; the trace the ingest delivered under that id is
the one rekeyed maps to it. No error text is read.
Source code in src/calibrated_agent_evaluation/agentcore/arms.py
contentless ¶
Per trace id, the spans of each trace whose message content did not come back.
A trace whose spans name two session ids, and which the source did not re-key, is left out: the ingest refuses it for its sessions whatever its content, so its arm stays unread.
Source code in src/calibrated_agent_evaluation/agentcore/arms.py
groups_read ¶
aws/spans and the log groups the spans read so far name, sorted.
AbTestClaimRecord
pydantic-model
¶
Bases: _AuditModel
What AgentCore said about an A/B test when it was read: a vendor claim, not evidence.
Fields:
-
record_id(str) -
claim_kind(Literal['vendor-claim']) -
ab_test_arn(str) -
ab_test_id(str) -
name(str) -
status(str) -
execution_status(str) -
gateway_arn(str) -
variants(tuple[VariantRef, VariantRef]) -
evaluation_config_arns(tuple[str, ...]) -
current_run_id(str | None) -
error_details(tuple[str, ...]) -
started_at(str | None) -
stopped_at(str | None) -
max_duration_expires_at(str | None) -
analysis_timestamp(str | None) -
updated_at(str) -
evaluator_claims(tuple[EvaluatorClaim, ...]) -
identical_arms(bool) -
provisional(bool) -
read_at(str) -
schema_version(str) -
provenance(ProvenanceStamp)
Validators:
-
_times→started_at,stopped_at,max_duration_expires_at,analysis_timestamp,updated_at,read_at -
_one_of_each_arm
ArmAssignment
pydantic-model
¶
Bases: _AuditModel
Which arm of the audited A/B test one delivered trace ran in, or why it has none.
Fields:
-
trace_id(str) -
otlp_trace_ids(tuple[str, ...]) -
read_id(str) -
runtime_session_id(str | None) -
span_session_ids(tuple[str, ...]) -
ab_test_arn(str) -
runtime(str) -
window_start(str) -
window_end(str) -
omni(bool) -
rekey_sessions(bool) -
shared_id(Literal['repeated-source-id', 'rekey-collision'] | None) -
status(ArmStatus) -
experiment_arn(str | None) -
variant(Literal['C', 'T1'] | None) -
spans_total(int) -
spans_stamped(int) -
spans_half_stamped(int) -
applied_bundle_attribute(str | None) -
applied_bundle_require(dict[str, str]) -
applied_bundle_values(dict[str, str]) -
reason(str | None) -
content_missing(bool) -
schema_version(str)
Validators:
-
_times→window_start,window_end -
_consistent
BundleVersionRef
pydantic-model
¶
Bases: _AuditModel
One configuration bundle version, as read, with its components digested.
Fields:
-
bundle_arn(str) -
version_id(str) -
version_created_at(str) -
parent_version_ids(tuple[str, ...]) -
branch_name(str | None) -
created_by(str | None) -
components(tuple[ComponentDigest, ...]) -
components_sha256(str)
Validators:
-
_time→version_created_at -
_unique_components
CodeInterpreterSandbox ¶
CodeInterpreterSandbox(
*,
region,
clients=None,
cassette=None,
budget=None,
role_arn=None,
identifier=_DEFAULT_IDENTIFIER,
execute=None,
)
A SandboxBackend backed by Bedrock AgentCore Code Interpreter.
Policy refusals (billed-call budget, wrong identity) surface as their own
AgentCoreError subclasses; an
execution/response failure surfaces as
SandboxError (the seam's contract, which
the execution scorer re-wraps into ScorerError).
Source code in src/calibrated_agent_evaluation/agentcore/code_interpreter.py
run_code ¶
Run code in a managed Code Interpreter session and return a normalized result.
Raises:
| Type | Description |
|---|---|
WrongAwsIdentity
|
a role is required and the caller is not it (pre-flight). |
BilledCallBudgetExceeded
|
the billed-call budget is used up (before the call is sent). |
SandboxError
|
the language is unsupported, seeded files were asked for, the call failed, the response was unparseable, or (REPLAY) nothing was recorded for it. |
Source code in src/calibrated_agent_evaluation/agentcore/code_interpreter.py
DatasetPlan
dataclass
¶
DatasetPlan(
name,
region,
schema_type,
n_scenarios,
n_with_assertions,
n_with_trajectory,
digest,
existing_id,
next_version,
unchanged,
)
What a publish would do, worked out from reads only.
creates_dataset
property
¶
Whether the publish creates a new dataset rather than versioning an existing one.
FailureDetector ¶
FailureDetector(
model,
*,
model_id=None,
confidence_threshold="low",
cassette_path=None,
cassette_mode=None,
max_live_calls=0,
now=None,
)
Detects failures in normalized traces with the strands-agents-evals failure detector.
The model stack is built and checked once, when the detector is built, so one live-call cap and one cassette serve every detection it makes, and one cassette can hold the calls for many traces. Each detection makes one model call, with the library's own prompt; the cassette records and replays it under that prompt. Detections on one detector run one at a time.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
Model
|
The strands |
required |
model_id
|
str | None
|
The model id, when the model's configuration has none; if both are given they must be equal. It is recorded on every detection and cassette entry. |
None
|
confidence_threshold
|
Confidence
|
The lowest confidence a reported failure keeps: |
'low'
|
cassette_path
|
str | Path | None
|
A cassette file that records and replays the model calls, or |
None
|
cassette_mode
|
Mode | None
|
The cassette mode; |
None
|
max_live_calls
|
int
|
How many live model calls this detector may make over its lifetime. A cassette replay is not a live call. |
0
|
now
|
datetime | None
|
The time every detection's provenance stamp records, for reproducible results; it
must be timezone-aware. |
None
|
Raises:
| Type | Description |
|---|---|
DetectorConfigError
|
The model, the model id, the cassette, the cassette mode, the
threshold, the live-call limit or |
Source code in src/calibrated_agent_evaluation/agentcore/detectors/detect.py
detect ¶
Detect the failures in one trace with one model call.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
trace
|
Trace
|
A normalized, owned trace with spans. |
required |
Returns:
| Type | Description |
|---|---|
Detection
|
One failure for each category the model named at each span or tool call, at or |
Detection
|
above the confidence threshold, in the converted session's span order and then the |
Detection
|
answer's order; a category named twice at one location is kept twice. The failures |
Detection
|
the threshold dropped are counted in |
Raises:
| Type | Description |
|---|---|
SessionConversionError
|
The trace cannot be converted for the detector. |
DetectorSessionTooLarge
|
The trace is too large for one model call, by the library's own estimate before any call or by the model's own refusal. |
DetectorCallLimitExceeded
|
The call would exceed |
DetectorCallError
|
The model, the cassette or the library failed, before or during the call; or the call could not be prepared, because the installed template is unusable or the trace could not be serialized or sized, and was not made. The original error is chained. |
DetectorConfigError
|
The model's config now names another model id than the one the detector was built for, or cannot be read; nothing was sent. |
DetectorOutputError
|
The library returned without its model call, or the model's
answer cannot be read faithfully: not one JSON object with an |
Source code in src/calibrated_agent_evaluation/agentcore/detectors/detect.py
DetectorDraftNamer ¶
DetectorDraftNamer(
model,
vocabulary,
*,
model_id=None,
max_critiques_per_cluster=_DEFAULT_MAX_CRITIQUES,
cassette_path=None,
cassette_mode=None,
max_live_calls=0,
now=None,
)
Names draft clusters from the detector vocabulary, one model call per naming.
A DraftNamer for build_draft, which it satisfies
structurally. The configuration is checked when the namer is built, before any model call. The
model is always the one passed: None and a model id string are refused, because the detector
library would build a Bedrock model of its own from them.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
Model
|
The strands |
required |
vocabulary
|
FailureVocabulary
|
The names the clusters are named from, usually |
required |
model_id
|
str | None
|
The model id, when the model's config has none; if both are given they must be equal. Every naming result and cassette entry records it. |
None
|
max_critiques_per_cluster
|
int
|
How many of a cluster's critiques the prompt shows, its first ones in member order; the prompt states how many were shown out of how many. |
_DEFAULT_MAX_CRITIQUES
|
cassette_path
|
str | Path | None
|
The cassette file that records and replays the naming calls, or |
None
|
cassette_mode
|
Mode | None
|
The cassette mode; |
None
|
max_live_calls
|
int
|
How many live model calls the namer may make over its lifetime. The default 0 allows none, so only cassette replays run. |
0
|
now
|
datetime | None
|
The time every naming result's provenance stamp records, for tests; |
None
|
Attributes:
| Name | Type | Description |
|---|---|---|
results |
list[NamingResult]
|
One |
Raises:
| Type | Description |
|---|---|
DetectorConfigError
|
The model is missing, a string or not a strands |
AgentCoreError
|
The installed detector package reports no version. |
Source code in src/calibrated_agent_evaluation/agentcore/detectors/namer.py
name_clusters ¶
Name each cluster from the vocabulary, with one model call.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
clusters
|
list[list[str]]
|
The member critiques of each cluster, in cluster order. |
required |
Returns:
| Type | Description |
|---|---|
list[str]
|
One name per cluster, in cluster order: a vocabulary name, with |
list[str]
|
appended to the second and later clusters given the same name. The call's |
list[str]
|
|
list[str]
|
to |
Raises:
| Type | Description |
|---|---|
DraftNamingError
|
|
DetectorCallLimitExceeded
|
The call would be a live call past |
DetectorCallError
|
The model or the cassette failed; the original error is chained. |
DetectorConfigError
|
The model's config now names another model id than the one the namer was built for, or cannot be read; nothing was sent. |
Source code in src/calibrated_agent_evaluation/agentcore/detectors/namer.py
DetectedFailure
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
One category the detector named at one location, with its evidence.
span_id is the Calibrated Agent Evaluation span the failure is on. tool_call_index and
tool_name are set when the detector pointed at one of that span's tool calls. A category the
detector named twice at one location appears twice, each with its own evidence.
Config:
frozen:Trueextra:forbid
Fields:
Detection
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
The detector's findings on one trace.
trace_sha256 is the SHA-256 of the canonical JSON of the trace that was detected.
times says whether the spans' recorded times were sent or placeholders stood in for them.
failures holds the failures at or above the confidence threshold, in span order and then
response order; below_threshold counts the ones the threshold dropped. No failures with
below_threshold 0 means the detector found none.
Config:
frozen:Trueextra:forbid
Fields:
-
trace_id(str) -
trace_sha256(str) -
times(Literal['recorded', 'placeholder']) -
failures(tuple[DetectedFailure, ...]) -
below_threshold(int) -
provenance(DetectorProvenance)
oracle_view ¶
The detection as JSON, without the values that describe the run's environment.
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
|
dict[str, Any]
|
|
dict[str, Any]
|
numeric-library versions |
dict[str, Any]
|
other field is kept. |
Source code in src/calibrated_agent_evaluation/agentcore/detectors/types.py
DetectorProvenance
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
What produced a detection or a set of cluster names.
model_config_sent is the model's config without its model_id, as sent.
confidence_threshold is the threshold the detection was filtered at, None for cluster
names. stamp.dep_versions records the strands-agents-evals version beside the
numeric libraries'.
Config:
frozen:Trueextra:forbidprotected_namespaces:()
Fields:
-
detector_package(Literal['strands-agents-evals']) -
detector_version(str) -
prompt_template(str) -
prompt_sha256(str) -
model_id(str) -
model_config_sent(dict[str, Any]) -
confidence_threshold(Confidence | None) -
stamp(ProvenanceStamp)
FailureVocabulary
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
The detector's categories in table order, then other, and the template they came from.
detector_version is the installed strands-agents-evals version, prompt_template
the template version and prompt_sha256 the SHA-256 of the prompt the detector sends for
an empty session.
Config:
frozen:Trueextra:forbid
Fields:
-
entries(tuple[VocabularyEntry, ...]) -
detector_version(str) -
prompt_template(str) -
prompt_sha256(str)
names ¶
NamingResult
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
The names one draft-naming call gave a set of clusters.
names holds one name per cluster, a repeated name suffixed #2, #3. shown
holds, per cluster, how many critiques the prompt showed and how many the cluster has, at
most max_critiques_per_cluster shown. namer_prompt_sha256 hashes the part of the
namer prompt that holds no cluster data.
Config:
frozen:Trueextra:forbid
Fields:
-
names(tuple[str, ...]) -
shown(tuple[tuple[int, int], ...]) -
namer_prompt_version(Literal['namer-v1']) -
namer_prompt_sha256(str) -
max_critiques_per_cluster(int) -
provenance(DetectorProvenance)
oracle_view ¶
The naming result as JSON, without the values that describe the run's environment.
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
|
dict[str, Any]
|
|
dict[str, Any]
|
numeric-library versions |
dict[str, Any]
|
other field is kept. |
Source code in src/calibrated_agent_evaluation/agentcore/detectors/types.py
VocabularyEntry
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
One failure category the detector may name.
parent is the category's group with its bold markup removed, None only for
other. definition and examples are the table's cells with their markup (such as
<br>) kept as written; examples is empty when the table gives none, and for
other, whose definition is the template's guideline for using it.
Config:
frozen:Trueextra:forbid
Fields:
DoctorTarget
pydantic-model
¶
Bases: _DoctorModel
The account slice to check, from the CLI flags.
Fields:
-
region(str) -
profile(str | None) -
runtime_name(str | None) -
omni_domain(str | None)
BilledCallBudget ¶
Counts billed operations and refuses the first one past limit; no limit when None.
Source code in src/calibrated_agent_evaluation/agentcore/guards.py
remaining
property
¶
Billed calls left before the limit, or None with no limit; charges nothing.
from_env
classmethod
¶
The budget the environment sets; no limit when unset.
The variable is CALIBRATED_AGENT_EVALUATION_AGENTCORE_MAX_BILLED_CALLS.
Source code in src/calibrated_agent_evaluation/agentcore/guards.py
charge ¶
Count one operation against the budget, or refuse it past the limit.
The explicit entry point for callers that do not go through a botocore before-call
handler (an adapter that builds its own client, e.g. the Code Interpreter sandbox). A
non-billed operation is ignored; a billed one past limit raises before it is sent.
Source code in src/calibrated_agent_evaluation/agentcore/guards.py
ReadOnlyGuard ¶
AgentCorePolicyDecisionSource ¶
AgentCorePolicyDecisionSource(
clients,
*,
run_records,
settle_deadline_seconds=None,
poll_interval_seconds=POLL_INTERVAL_SECONDS,
clock=None,
sleep=None,
monotonic=None,
)
Reads each trial's Policy decisions from aws/spans by the trial's W3C trace ID.
run_records is read when the decisions are read, never when the source is built: it maps
each trial's probe trace ID to the run that sent it. Each poll reads every trial not yet
settled through the Logs Insights index and straight from the log store. A trial settles when
the two hold the same spans on two reads in a row, and is not read again. A trial with no span
in either may simply not have reached the store yet, so it settles only at the bound, and then
counts as a trial with no decision. The wait is bounded
by settle_deadline_seconds, by default settle_deadline_for the number of trials;
a trial whose last read before the bound matched still gets its confirming read. Every read is
read-only, and none is billed.
Source code in src/calibrated_agent_evaluation/agentcore/policy_spans.py
check_readable ¶
One read-only Logs Insights query on aws/spans over the last second.
A truncated result counts as readable, since the query ran; any other failure, a missing
logs:StartQuery permission among them, is raised before any trial is sent.
Source code in src/calibrated_agent_evaluation/agentcore/policy_spans.py
read_decisions ¶
One answer per trial, keyed by probe trace ID, once every trial's spans have settled.
Raises:
| Type | Description |
|---|---|
ProbeError
|
A trial has no run record, or its record's W3C trace ID or times are unreadable. |
AgentCoreSettlingLag
|
Some trial's spans were still arriving at the deadline. |
PolicySpanMalformed
|
A decision span can't be read whole. |
Source code in src/calibrated_agent_evaluation/agentcore/policy_spans.py
357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 | |
OmniReader ¶
Reads a runtime's traces and evaluator scores through one CloudWatch Omni SQL client.
Source code in src/calibrated_agent_evaluation/agentcore/readers.py
read_traces ¶
Yield each trace in the window, with its spans and its content log records.
Source code in src/calibrated_agent_evaluation/agentcore/readers.py
read_scores ¶
Yield each successful evaluator-result record in the window (empty when none).
Reads the gen_ai.evaluation.result records (whichever log groups the Omni dataset
forwards them from) and maps each to an Evaluate-result dict; error-event records (a failed
evaluation, no score) are skipped, not yielded.
Source code in src/calibrated_agent_evaluation/agentcore/readers.py
RunRecord
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
One record of the run manifest: where one run went, what it carried and how it ended.
Attributes:
| Name | Type | Description |
|---|---|---|
schema_version |
str
|
The run record format, |
trace_id |
str
|
The ID of the run's |
w3c_trace_id |
str
|
The W3C trace ID the run's invocations carried. |
session_id |
str
|
The runtime session the run used. |
runtime_arn |
str
|
The runtime invoked. |
qualifier |
str
|
The endpoint invoked. |
runtime_version |
str | None
|
The runtime version that served the run, or |
runtime_version_source |
VersionSource | None
|
|
config_bundle_arn |
str | None
|
The configuration bundle the run named, or |
config_bundle_version |
str | None
|
That bundle's version, or |
payload_format |
str
|
The request payload format. |
started_at |
str
|
When the run started, ISO-8601 with its UTC offset. |
ended_at |
str
|
When the run ended, in the same form. |
duration_ms |
int
|
How long the run took, in milliseconds. |
attempts |
int
|
How many invocation attempts the run sent, retries included. |
status_code |
int | None
|
The last response's HTTP status, or |
content_type |
str | None
|
The last response's content type, or |
response_traceparent |
str | None
|
The |
replayed |
bool
|
Whether the run was replayed from a cassette instead of sent. |
session_stop |
SessionStop
|
How the run's session ended. |
error_code |
str | None
|
Why the run failed: the AWS error code for an AWS service error, else the
type name of the error that ended it. |
Config:
frozen:Trueextra:forbiddefer_build:True
Fields:
-
schema_version(str) -
trace_id(str) -
w3c_trace_id(str) -
session_id(str) -
runtime_arn(str) -
qualifier(str) -
runtime_version(str | None) -
runtime_version_source(VersionSource | None) -
config_bundle_arn(str | None) -
config_bundle_version(str | None) -
payload_format(str) -
started_at(str) -
ended_at(str) -
duration_ms(int) -
attempts(int) -
status_code(int | None) -
content_type(str | None) -
response_traceparent(str | None) -
replayed(bool) -
session_stop(SessionStop) -
error_code(str | None)
RunSpanQuery
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
The run whose spans to read, named by the IDs the executor generated for it.
Attributes:
| Name | Type | Description |
|---|---|---|
trace_id |
str
|
The Calibrated Agent Evaluation trace ID of the run's |
w3c_trace_id |
str
|
The W3C trace ID every invocation of the run carried in |
session_id |
str
|
The runtime session ID every invocation of the run was sent with. |
runtime_arn |
str
|
The runtime invoked. |
qualifier |
str
|
The endpoint invoked. |
started_at |
str
|
When the invocation started: ISO-8601 in UTC, with its offset. |
ended_at |
str
|
When the invocation ended, in the same form. |
Config:
frozen:Trueextra:forbiddefer_build:True
Fields:
-
trace_id(str) -
w3c_trace_id(str) -
session_id(str) -
runtime_arn(str) -
qualifier(str) -
started_at(str) -
ended_at(str)
RunSpanReader ¶
Bases: Protocol
Reads the span tree of one AgentCore Runtime run, under the contract in this module.
read_spans ¶
The run's whole span tree, every span keyed to query.trace_id.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
query
|
RunSpanQuery
|
The run to read. |
required |
Returns:
| Type | Description |
|---|---|
list[Span]
|
The run's spans: never empty, never partial. |
Raises:
| Type | Description |
|---|---|
AgentCoreError
|
A typed subclass when the tree can't be read whole, or a guard's refusal, as it is or chained. |
Source code in src/calibrated_agent_evaluation/agentcore/run_spans.py
RuntimeTarget
pydantic-model
¶
Bases: _RuntimeModel
The AgentCore Runtime agent to invoke, and how to ask it.
Attributes:
| Name | Type | Description |
|---|---|---|
runtime_arn |
str
|
The runtime's ARN,
|
qualifier |
str
|
The endpoint every invocation goes to: a letter, then up to 47 letters, digits and underscores. |
payload_format |
str
|
How a prompt becomes a request body: one of |
config_bundle_arn |
str | None
|
The configuration bundle the agent runs with. Name it together with its version, or name neither; an empty string counts as not named. |
config_bundle_version |
str | None
|
That bundle's version, a lowercase UUID. |
Fields:
-
runtime_arn(str) -
qualifier(str) -
payload_format(str) -
config_bundle_arn(str | None) -
config_bundle_version(str | None)
Validators:
-
_is_a_runtime_arn→runtime_arn -
_is_an_endpoint_name→qualifier -
_is_a_known_format→payload_format -
_empty_is_not_named→config_bundle_arn,config_bundle_version -
_is_a_bundle_arn→config_bundle_arn -
_is_a_bundle_version→config_bundle_version -
_bundle_is_whole
AgentReportedError ¶
Bases: RuntimeInvocationError
The agent's reply reported an error, so the run has no answer.
A caller that treats such a run as a result rather than a failure (a policy battery, where a
denied tool call can come back as an agent error) tells it apart from every other refusal by
type. error_code is the report's error_type when that is an identifier of at most 100
characters, else "agent_error"; the error text is never kept. trace_id is the run's
trace ID when the raiser knows it, else None.
Source code in src/calibrated_agent_evaluation/agentcore/runtime_errors.py
RuntimeInvocationError ¶
Bases: ExecutionError, AgentCoreError
A run against an AgentCore Runtime agent was refused or failed.
An ExecutionError and an AgentCoreError, so a caller of any executor and a caller of
the AgentCore plug-in both catch it. The underlying error is chained as __cause__. The
message names identifiers and metadata only (error codes, operations, session and trace IDs,
paths), never a payload, a response body or the agent's error text, with one exception: when
a span reader fails with an AgentCoreError other than a billed-call or read-only guard's
refusal, the message quotes that error's text after The reader's message:. The span reader
contract keeps a reader's typed refusals to IDs, log groups, counts and times. An
AgentCoreError from a caller's own guard on the reader's clients looks the same, so its
text is quoted too, whatever it holds.
AgentCoreRuntimeExecutor ¶
AgentCoreRuntimeExecutor(
target,
*,
span_reader=None,
clients=None,
profile=None,
payload_builder=None,
builder_name=None,
cassette_path=None,
cassette_mode=None,
log_dir=None,
frame="production-representative",
read_timeout_seconds=DEFAULT_READ_TIMEOUT_SECONDS,
attempt_limit=DEFAULT_ATTEMPT_LIMIT,
sleep=time.sleep,
clock=utc_now,
)
An ExecutorBackend that runs each task against an AgentCore Runtime agent.
Use it as a context manager. Leaving the context stops every session a run opened whose stop did not succeed, logging each result, closes the AWS clients the executor built (and the invocation clients it derived from yours), and removes its temporary log directory. A failure on the way out is raised when the block succeeded; when the block raised, the block's error is raised, with the failure logged as a warning and noted on it.
Tasks run one after another; an executor is not safe to share across threads.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
target
|
RuntimeTarget
|
The runtime, endpoint qualifier, payload format and configuration bundle to use. |
required |
span_reader
|
RunSpanReader | None
|
Reads each run's span tree: any object with the |
None
|
clients
|
AwsClients | None
|
AWS clients to use instead of building them, for the runtime ARN's region. The
executor installs no billed-call budget on them. Give these or |
None
|
profile
|
str | None
|
The AWS profile to build clients from; the default credential chain when
|
None
|
payload_builder
|
PayloadBuilder | None
|
Builds the request body for the |
None
|
builder_name
|
str | None
|
Names |
None
|
cassette_path
|
str | Path | None
|
A cassette to record runs into or replay them from. |
None
|
cassette_mode
|
Mode | str | None
|
Whether the cassette records, replays or does both; pass it only with
|
None
|
log_dir
|
str | Path | None
|
The directory whose run manifest gets each run's record; a temporary directory,
removed on exit, when |
None
|
frame
|
Frame
|
The sampling frame every returned |
'production-representative'
|
read_timeout_seconds
|
float
|
How long an invocation waits for each read of its response: a
positive, finite number of seconds of at most |
DEFAULT_READ_TIMEOUT_SECONDS
|
attempt_limit
|
int
|
The most attempts one invocation may send, the first one included. |
DEFAULT_ATTEMPT_LIMIT
|
sleep
|
Callable[[float], object]
|
Waits between retried attempts. |
sleep
|
clock
|
Callable[[], datetime]
|
The current time, timezone-aware; it times every run. |
utc_now
|
Raises:
| Type | Description |
|---|---|
RuntimeInvocationError
|
|
Source code in src/calibrated_agent_evaluation/agentcore/runtime_executor.py
246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 | |
run_records
property
¶
The record of every task run this executor finished, whatever its outcome, in order.
A record is listed even when the run manifest could not take it; manifest_errors
says which writes failed.
manifest_errors
property
¶
Why each run record that could not be added to the run manifest was not, in order.
One error per failed write, each naming the manifest's path and what to do next. Empty
when every record in run_records is in the manifest. A successful run whose record
can't be written raises its error; a failed run keeps its own error, with this one noted.
close ¶
Stop the sessions left open, close the clients built here, remove the temp log dir.
Each session whose stop did not succeed, a failed or interrupted one included, gets one
more StopRuntimeSession, its result logged with its session ID. Every step runs even
when an earlier one fails.
Raises:
| Type | Description |
|---|---|
RuntimeInvocationError
|
The client to stop the open sessions could not be built, an AWS client could not be closed, or the temporary log directory could not be removed. The first failure is raised; each later one is logged as a warning and noted on it. |
Source code in src/calibrated_agent_evaluation/agentcore/runtime_executor.py
run ¶
Run one task against the runtime and return its validated, owned Trace.
The run's record goes to the run manifest (agentcore_run_record.json and .csv)
in log_dir, else in the executor's log directory, whatever the outcome. With a
cassette, a recorded run is replayed, and a live run is recorded once its Trace is
valid.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
task
|
Any
|
required | |
log_dir
|
str | Path | None
|
The directory whose run manifest gets this run's record, instead of the executor's log directory. |
None
|
Returns:
| Type | Description |
|---|---|
list[Trace]
|
One |
Raises:
| Type | Description |
|---|---|
RuntimeInvocationError
|
Before any AWS call and with no run record: the task is of
another type, its prompt is not a non-empty string, its |
Source code in src/calibrated_agent_evaluation/agentcore/runtime_executor.py
end_conversation ¶
Stop the runtime session a conversation ran on, once its last turn has run.
Call it when the conversation ends, fails or is interrupted. A conversation with no open
session (unknown, already ended, or replayed from a cassette) is let go and nothing is
stopped. A session whose stop did not succeed stays held, and close tries it once
more.
Each turn's run record says held_open; the stop's own result is returned, and a stop
that did not succeed is logged as a warning.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
conversation_id
|
str
|
The conversation's ID, as its tasks' |
required |
Returns:
| Type | Description |
|---|---|
SessionStop | None
|
How the stop ended, or |
Raises:
| Type | Description |
|---|---|
RuntimeInvocationError
|
The client to stop the session could not be built. |
Source code in src/calibrated_agent_evaluation/agentcore/runtime_executor.py
AgentCoreRuntimeTask
dataclass
¶
One input to send to the AgentCore Runtime agent.
Attributes:
| Name | Type | Description |
|---|---|---|
prompt |
str
|
The test input, a non-empty string; |
trace_id |
str | None
|
The ID the returned |
taxonomy_cell |
dict[str, str]
|
The taxonomy cell the input was written for, carried onto the |
conversation |
ConversationTurn | None
|
For one turn of a simulated conversation, which conversation and turn. Every
turn of a conversation is sent on one runtime session, left open between turns and
stopped by |
AgentCoreTaskBuilder ¶
Turns synthesized inputs into AgentCoreRuntimeTasks, one per input.
Each task's prompt is the input's text and its taxonomy_cell the input's cell, so every
returned trace can be traced to the cell it was written for.
build_all ¶
One task per input, in order.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
inputs
|
list[SynthInput]
|
The synthesized inputs. |
required |
Returns:
| Type | Description |
|---|---|
list[AgentCoreRuntimeTask]
|
The tasks. |
Raises:
| Type | Description |
|---|---|
SynthError
|
|
Source code in src/calibrated_agent_evaluation/agentcore/runtime_executor.py
build ¶
The task for a single input. Use build_all for a list.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
inputs
|
list[SynthInput]
|
Exactly one synthesized input. |
required |
Returns:
| Type | Description |
|---|---|
AgentCoreRuntimeTask
|
The task. |
Raises:
| Type | Description |
|---|---|
SynthError
|
|
Source code in src/calibrated_agent_evaluation/agentcore/runtime_executor.py
build_conversation_turn ¶
The task for one agent turn of a simulated conversation.
A runtime keeps the conversation's memory in its session, so the turn sends only the
latest customer message; turn keeps every turn of the conversation on one session.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
item
|
SynthInput
|
The synthesized input the conversation started from. |
required |
history
|
list[dict[str, Any]]
|
The conversation so far, ending with the customer's message for this turn. |
required |
turn
|
ConversationTurn
|
Which conversation and turn this is. |
required |
Returns:
| Type | Description |
|---|---|
AgentCoreRuntimeTask
|
The turn's task. |
Raises:
| Type | Description |
|---|---|
SynthError
|
|
Source code in src/calibrated_agent_evaluation/agentcore/runtime_executor.py
RuntimeProbeAdapter ¶
RuntimeProbeAdapter(
target,
*,
clients,
log_dir,
read_timeout_seconds=DEFAULT_READ_TIMEOUT_SECONDS,
attempt_limit=DEFAULT_ATTEMPT_LIMIT,
sleep=time.sleep,
clock=utc_now,
)
A probe adapter that sends each prompt to an AgentCore Runtime agent as one run.
Pass it to probe_with, with
target_label as the probe's target. Use it as a context manager: leaving the context
stops every runtime session a prompt opened whose stop did not succeed, logging each result, and
closes the invocation view it derived from your clients. Your clients stay open; they are yours
to close. A failure on the way out is raised when the block succeeded. When the block raised,
its own error is the one raised, and the failure is logged as a warning and noted on it.
Prompts run one after another; an adapter is not safe to share across threads.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
target
|
RuntimeTarget
|
The runtime, endpoint qualifier, payload format and configuration bundle to
probe. The format is one of the probe's request shapes: |
required |
clients
|
AwsClients
|
The AWS clients every call goes through, built for the region in the runtime ARN, with any billed-call budget already installed. The adapter adds no guard. |
required |
log_dir
|
Path
|
The directory whose run manifest ( |
required |
read_timeout_seconds
|
float
|
How long an invocation waits for each read of its reply: a
positive, finite number of seconds of at most |
DEFAULT_READ_TIMEOUT_SECONDS
|
attempt_limit
|
int
|
The most attempts one prompt's invocation may send, the first included. |
DEFAULT_ATTEMPT_LIMIT
|
sleep
|
Callable[[float], object]
|
Waits between retried attempts. |
sleep
|
clock
|
Callable[[], datetime]
|
The current time, timezone-aware; it times every prompt's run. |
utc_now
|
Raises:
| Type | Description |
|---|---|
RuntimeInvocationError
|
Raised before any client is built: the payload format is
|
Source code in src/calibrated_agent_evaluation/agentcore/runtime_probe.py
target_label
property
¶
What this adapter probes, <runtime ARN>@<qualifier>, to pass as target.
Every trace ID this adapter names carries the endpoint fingerprint of this label. Passed
to probe_with as target, it names the
target in the authorization refusal, and every trace and the finding carry that same
fingerprint.
run_records
property
¶
The record of every prompt this adapter sent, whatever its outcome, in order.
A record is listed even when the run manifest could not take it; manifest_errors
says which writes failed.
manifest_errors
property
¶
Why each run record that could not be added to the run manifest was not, in order.
One error per failed write, each naming the manifest's path and what to do next. Empty
when every record in run_records is in the manifest. A prompt that succeeded but
whose record can't be written raises its error; a failed prompt keeps its own error, with
this one noted.
close ¶
Stop the sessions left open, then close the invocation view derived from your clients.
Each session whose stop did not succeed, a failed or interrupted one included, gets one
more StopRuntimeSession, and its result is logged at INFO with its session ID. The
view is closed even when stopping fails. Your clients are left open.
Raises:
| Type | Description |
|---|---|
RuntimeInvocationError
|
The client to stop the open sessions could not be built, or the view could not be closed, naming the error's type and never its text. When both fail, the first is raised, and the second is logged as a warning and noted on it. |
Source code in src/calibrated_agent_evaluation/agentcore/runtime_probe.py
send ¶
Send one probe prompt as a run of its own, and return the agent's reply.
The prompt gets a new runtime session and W3C trace context. Its body is built in the target's payload format before any AWS call. Before the first invocation the runtime is checked with read-only calls, and only a check that succeeded is kept for later prompts. A busy session or throttling is retried with the same IDs and body. Once a request may have reached the service, the session is stopped before this returns or raises. The run record goes to the run manifest of the log directory, whatever the outcome.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
prompt
|
str
|
The probe prompt, a non-empty string. |
required |
Returns:
| Type | Description |
|---|---|
ProbeResponse
|
The reply: status 200, its final text, the tools its stream events name (each |
ProbeResponse
|
string |
ProbeResponse
|
its decoded objects as the events, and |
ProbeResponse
|
its trace ID, |
Raises:
| Type | Description |
|---|---|
RuntimeInvocationError
|
The prompt is not a string or is empty, refused before any AWS call and with no run record; or the prompt was refused or failed, named by the step and the session ID: the body could not be built; the clients could not be built; the preflight failed, or found a runtime that only accepts OAuth or does not serve HTTP; the invocation was refused by a guard, never sent, refused by the service, still busy or throttled on its last attempt, or failed after it was sent; or the reply could not be decoded, reported an agent error, or held no answer. A prompt whose run record could not be written is refused naming the manifest's path; when the prompt had already failed, its own error is raised with that path noted. |
Source code in src/calibrated_agent_evaluation/agentcore/runtime_probe.py
AwsClients ¶
Builds and caches one client per AWS service from an injected boto3 session.
Every handler in handlers runs before each API call on every client this builds, ahead
of any other before-call handler, including botocore's Stubber. Use it as a context
manager so the clients' connection pools are closed. Not safe to share across threads; the
clients it hands out are.
Source code in src/calibrated_agent_evaluation/agentcore/session.py
boto_session
property
¶
The underlying boto3 session, for an adapter that must build its own vendor client.
Named boto_session (not session) to stay clear of the unrelated review session
and of subclasses that keep their own session attribute. The Code Interpreter SDK
(code_session) constructs its own bedrock-agentcore client from a session rather
than taking one of ours, so it needs the session itself; identity and billed-call limits are
enforced by the adapter around it (require_role + the budget).
billed_call_budget
property
¶
The first billed-call budget among the handlers, or None (no limit).
Reading it charges nothing, so a caller can check remaining before a run.
close ¶
Close every client built so far; later client() calls build fresh ones.
from_profile
classmethod
¶
Clients for a named profile (or the default credential chain) in region.
Without region, the profile's configured region is used.
Source code in src/calibrated_agent_evaluation/agentcore/session.py
with_config ¶
A second set of clients with config merged over this one's settings.
The new set shares this one's class, boto3 session and handlers, so every guard installed
here runs on its calls too, and a billed-call budget counts both sets' calls together.
Only the settings config names change; the rest (timeouts it leaves out, proxies, the
user agent) carry over. This set's clients are left as they are. The new set builds and
caches its own clients, and closing one set leaves the other's open, so close each.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
Config
|
The settings to change, such as a single-attempt retry policy. |
required |
Returns:
| Type | Description |
|---|---|
Self
|
The new set of clients, with no client built yet. |
Source code in src/calibrated_agent_evaluation/agentcore/session.py
client ¶
The cached client for service_name, built on first use.
region builds the client in another region (for an API that answers only from a few
endpoint regions, such as the Price List API), with the same handlers and settings; it is
cached per service and region. Without it, the client is built in this set's region.
Source code in src/calibrated_agent_evaluation/agentcore/session.py
caller_identity ¶
The caller's ARN, which also proves credentials resolve before any other call.
Source code in src/calibrated_agent_evaluation/agentcore/session.py
SettlePolicy ¶
Decides whether a window has settled, waiting a bounded interval when it has not.
Source code in src/calibrated_agent_evaluation/agentcore/settle.py
max_wait_seconds
instance-attribute
¶
from_env
classmethod
¶
A policy whose lag is the one the environment sets, when set.
The variable is CALIBRATED_AGENT_EVALUATION_AGENTCORE_SETTLE_SECONDS.
Source code in src/calibrated_agent_evaluation/agentcore/settle.py
is_settled ¶
Whether the window ended at least lag_seconds before now.
Source code in src/calibrated_agent_evaluation/agentcore/settle.py
refuse_if_unsettled ¶
Raise AgentCoreSettlingLag when the window ends inside the lag.
Source code in src/calibrated_agent_evaluation/agentcore/settle.py
wait_for_window ¶
Wait up to max_wait_seconds for the window to settle, else refuse.
Returns as soon as the window has settled. Raises AgentCoreSettlingLag if it
cannot settle within the bounded wait, so a window too recent to ever settle in time is a
loud refusal, not a silent partial read.
Source code in src/calibrated_agent_evaluation/agentcore/settle.py
AgentCoreSourceConfig
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
What to read, resolved from the CLI flags.
omni selects the CloudWatch Omni SQL path; otherwise spans are read from Logs Insights.
(Omni queries need no domain — verified live — so this is a plain flag, not a domain name:
a domain value would only pick the path while never reaching a call, so a typo would read the
default silently.) rekey_sessions opts into collapsing a mixed-session trace to the runtime
session id. with_scores additionally reads AgentCore evaluator results as advisory labels.
Config:
frozen:Trueextra:forbid
Fields:
-
runtime(str) -
window(TimeWindow) -
omni(bool) -
rekey_sessions(bool) -
with_scores(bool)
Validators:
-
_safe_runtime_name→runtime
AgentCoreTraceSource ¶
A read-only TraceSource for AgentCore.
Satisfies the TraceSource protocol (pull only); it is not a store and exposes no write.
The AWS reads are the injected reader; this class owns the Calibrated Agent Evaluation-side
orchestration: wait for the window to settle, resolve session identity, convert to OTLP, and
refuse (per trace) a trace whose content did not come back.
Source code in src/calibrated_agent_evaluation/agentcore/sources.py
pull ¶
Yield each AgentCore trace as a SourceTrace carrying canonical OTLP-JSON.
Source code in src/calibrated_agent_evaluation/agentcore/sources.py
read_scores ¶
The reader's evaluator-result records (mapped to labels by the scores module).
TimeWindow
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
The read window, as timezone-aware ISO-8601 bounds.
Timezone-aware is required: a naive bound compared against AWS timestamps silently shifts the window by the local offset, so a naive value is refused rather than coerced.
Config:
frozen:Trueextra:forbid
Fields:
Validators:
AgentCoreSpanReader ¶
AgentCoreSpanReader(
clients,
*,
use_omni=False,
settle_deadline_seconds=_DEFAULT_SETTLE_DEADLINE_SECONDS,
poll_interval_seconds=_DEFAULT_POLL_INTERVAL_SECONDS,
)
Reads one AgentCore Runtime run's span tree by id, satisfying the executor's reader contract.
Built from the same AwsClients as the
executor, so its billed reads count against the shared budget and a guard's refusal (for example
BilledCallBudgetExceeded) propagates to
the caller. The read path is the Logs Insights group aws/spans by default, or CloudWatch
Omni when use_omni=True (Omni needs no domain, so the path is an explicit flag, as in the
window ingest). The Logs Insights path needs logs:StartQuery, logs:GetQueryResults and
logs:FilterLogEvents on aws/spans and on the runtime's own log group.
Source code in src/calibrated_agent_evaluation/agentcore/span_reader.py
read_spans ¶
The run's whole normalized span tree, every span keyed to query.trace_id.
Polls the by-id read until a stable read (the same span count and content-record count on
two polls in a row) assembles into a whole tree. On the Logs Insights path a read counts
only when it holds the same spans and as many content records as the log store already
holds for the run, since the index serves a run's records in batches minutes apart and
two equal reads can both be missing one. A stable read that is not yet whole (no
agent span tree yet, an agent with no content, or a completed agent missing its input or
output) is polled again; at the deadline the reader raises that refusal
(AgentCoreContentMissing, or an
AgentCoreError naming that no agent tree formed). A run that never appears, or is still
changing at the deadline, raises AgentCoreSettlingLag. A run carrying more than one
session id raises AgentCoreMixedSession. An errored agent span may lack its answer.
A partial or truncated read raises its typed error (from the reused drivers). Never returns
a partial or empty list.
Source code in src/calibrated_agent_evaluation/agentcore/span_reader.py
227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 | |
adot_to_otlp ¶
Assemble OTLP span dicts into one canonical OTLP-JSON document, content re-homed.
span_records are already-decoded OTLP span dicts (spanId/traceId/parentSpanId/
attributes/...). log_records are the decoded otel-rt-logs content records, each a
plain dict with spanId, eventName and body (and an optional timestamp used only for
ordering). Content is re-homed from the chat records onto each invoke_agent and
execute_tool span; a content-bearing span that recovers nothing raises
AgentCoreContentMissing naming the trace and span.
Source code in src/calibrated_agent_evaluation/agentcore/adot.py
xray_trace_id_to_otel ¶
Normalize an X-Ray 1-<8 hex>-<24 hex> trace id to the 32-hex OTel id.
An already-32-hex id is returned lowercased unchanged. A value in neither form raises rather than returning a corrupt id, so a malformed id never silently points at a nonexistent trace.
Source code in src/calibrated_agent_evaluation/agentcore/adot.py
merge_arms_read ¶
The assignments of read to store, given what is already stored for its test.
A read under a new read id is stored whole; one that repeats a stored read exactly stores nothing. A stored read never changes.
Raises:
| Type | Description |
|---|---|
AgentCoreArmsReadChanged
|
The read repeats a stored read id and adds, drops or reassigns a source id; nothing of it may be stored. |
Source code in src/calibrated_agent_evaluation/agentcore/arms.py
read_arms ¶
read_arms(
source,
*,
ab_test_arn,
runtime,
window,
omni=False,
rekey_sessions=False,
rekey_collisions=None,
contentless=None,
applied_bundle_attribute=None,
applied_bundle_require=(),
bundle_versions=None,
)
Read which arm of ab_test_arn each trace the source delivers ran in.
The source is pulled to the end. A source id delivered more than once, and each re-key
collision rekey_collisions reports, is one refused assignment. Every other trace is
unread (delivered as an error or a normalized trace), refused for a half-stamped span,
unassigned (no span carries both attributes), refused (two arms, a variant other than C or
T1, two runtime sessions, a failed applied-bundle condition), set aside as another
experiment's, or assigned. A runtime session whose traces sit in two arms is then refused.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
TraceSource
|
Any trace source; for the AgentCore ingest's, the read path and re-key setting used at ingest. |
required |
ab_test_arn
|
str
|
The audited test's ARN. |
required |
runtime
|
str
|
The runtime read, recorded with the read. |
required |
window
|
_Window
|
The read window, with |
required |
omni
|
bool
|
Whether the source read through Omni; recorded, not used. |
False
|
rekey_sessions
|
bool
|
Whether the source re-keyed mixed-session traces; recorded, not used. |
False
|
rekey_collisions
|
Callable[[], Mapping[str, Sequence[KeptTrace]]] | None
|
Called once after the pull: per runtime session id, every trace that
re-keyed to it, as |
None
|
contentless
|
Callable[[], Mapping[str, Mapping[str, Any]]] | None
|
Called once after the pull: per trace id, the spans (as an OTLP-JSON
document with no content) of each trace the source delivered as an error because its
message content did not come back, as |
None
|
applied_bundle_attribute
|
str | None
|
A span attribute that must equal the arm's bundle version. |
None
|
applied_bundle_require
|
Iterable[tuple[str, str]]
|
|
()
|
bundle_versions
|
Mapping[str, str] | None
|
Each arm's bundle version id, for |
None
|
Returns:
| Type | Description |
|---|---|
ArmRead
|
The read: its id, one assignment per source id, and its counts. |
Raises:
| Type | Description |
|---|---|
AgentCoreArmsMissing
|
No span is stamped, or no stamped span names the audited test. |
Exception
|
Whatever the source raises before its first trace, unchanged. |
Source code in src/calibrated_agent_evaluation/agentcore/arms.py
516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 | |
runtime_session_id ¶
The runtime session a trace ran in: the session.id of its /invocations server span.
An agent's own spans may carry another session.id; only the runtime's request span names
the session the client sent. None when the trace has no such span or it carries no id.
Raises:
| Type | Description |
|---|---|
AgentCoreArmSessionAmbiguous
|
Two |
Source code in src/calibrated_agent_evaluation/agentcore/arms.py
span_stamp ¶
stamped with both arm attributes, half with one of them, none with neither.
Source code in src/calibrated_agent_evaluation/agentcore/arms.py
dataset_guard ¶
The allowlist every publish and pull client carries: the dataset operations, nothing else.
It is a ReadOnlyGuard used as a plain allowlist: the listed operations include the
dataset writes publish makes, and every other operation is refused before it is sent.
Source code in src/calibrated_agent_evaluation/agentcore/datasets.py
plan_publish ¶
The publish plan for slice_ under name; checks that need no call come first.
Source code in src/calibrated_agent_evaluation/agentcore/datasets.py
publish ¶
publish(
clients,
slice_,
plan,
*,
poll_seconds=2.0,
max_wait_seconds=120.0,
sleep=time.sleep,
clock=time.monotonic,
)
Create or version the dataset the plan names and return where the slice now lives.
Source code in src/calibrated_agent_evaluation/agentcore/datasets.py
185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 | |
pull ¶
The slice one published version of a dataset holds, decoded strictly.
Source code in src/calibrated_agent_evaluation/agentcore/datasets.py
content_digest ¶
A SHA-256 over the slice's encoded examples, so an unchanged slice can be recognized.
Source code in src/calibrated_agent_evaluation/agentcore/datasets_codec.py
decode_example ¶
The scenario an AgentCore example document holds, refusing any field that does not map.
Source code in src/calibrated_agent_evaluation/agentcore/datasets_codec.py
encode_example ¶
The AgentCore example document for scenario; empty optional fields are left out.
Source code in src/calibrated_agent_evaluation/agentcore/datasets_codec.py
detector_bedrock_model ¶
detector_bedrock_model(
model_id,
*,
region=None,
timeout_seconds=_DEFAULT_TIMEOUT_SECONDS,
temperature=None,
)
A Bedrock model for the detector or the namer, built only when a live call needs it.
The BedrockModel and its client are built on the first live call, with connect and read
timeouts of timeout_seconds each, so a cassette replay builds no client, resolves no AWS
credentials and opens no connection. The client sends each request once, whatever retries the
environment or the AWS profile configure, so one live call is one billed ConverseStream
request; a throttled or failed call raises instead of being retried.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model_id
|
str
|
The Bedrock model id, for example |
required |
region
|
str | None
|
The AWS region of the Bedrock runtime; |
None
|
timeout_seconds
|
float
|
The connect and read timeout of the Bedrock client, in seconds. |
_DEFAULT_TIMEOUT_SECONDS
|
temperature
|
float | None
|
The sampling temperature; |
None
|
Returns:
| Type | Description |
|---|---|
Model
|
A strands |
Model
|
|
Raises:
| Type | Description |
|---|---|
DetectorConfigError
|
|
Source code in src/calibrated_agent_evaluation/agentcore/detectors/models.py
trace_to_session ¶
Convert a normalized trace into the vendor session the failure detector reads.
The session id, the one vendor trace's id and every span's trace and session id are the
trace id. Spans are taken in seq order. Each becomes an agent-invocation span with the
same id and parent, its canonical-JSON input and output as prompt and response, and the
distinct names of the tools it called, in first-call order, as its available tools. Each
tool call follows as a tool-execution span under it, with its arguments rebuilt with sorted
keys and its canonical-JSON result.
When every span records both started_at and ended_at, those times are used, and each
tool span takes its agent span's times. Otherwise every span gets placeholder times that
encode only the seq order: agent span k (counted from 0) starts 2k seconds and
ends 2k+1 seconds after 1970-01-01T00:00:00Z, and each tool span takes its agent
span's interval.
The trace is checked before anything is built, so a refused trace never reaches a model.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
trace
|
Trace
|
The normalized trace. |
required |
Returns:
| Type | Description |
|---|---|
ConvertedSession
|
The converted session, which time source it used, and the map from each vendor span id |
ConvertedSession
|
back to its Calibrated Agent Evaluation span and tool call. |
Raises:
| Type | Description |
|---|---|
SessionConversionError
|
The trace is probed or has no spans; a span's time does not parse as ISO-8601 or has no timezone (checked even when another span's missing time would send the trace to placeholders); a span ends before it starts; a tool span id the converter would build is already the id of another span; or a span's input, output, tool arguments or tool result is not JSON, or the detector's span types refuse it. |
Source code in src/calibrated_agent_evaluation/agentcore/detectors/session.py
failure_vocabulary ¶
The installed detector's failure categories in table order, then other.
The template is read on every call. The other entry has no parent and no examples; its
definition is the text of the template's guideline line that starts
- **Use "other"**:.
Returns:
| Type | Description |
|---|---|
FailureVocabulary
|
The vocabulary, with the detector version, template version and template hash. |
Raises:
| Type | Description |
|---|---|
VocabularyParseError
|
The category table breaks a row check (see
|
AgentCoreError
|
The installed detector package reports no version, or its template lacks a part the reader needs or holds one of another type. |
Source code in src/calibrated_agent_evaluation/agentcore/detectors/vocabulary.py
read_only_guard ¶
run_checks ¶
Confirm the identity, then run every check, isolating each one's AWS failures.
Source code in src/calibrated_agent_evaluation/agentcore/doctor.py
ingest_agentcore ¶
ingest_agentcore(
source,
*,
backend,
dry_run=False,
require_owned=False,
with_scores=False,
frame=None,
)
Pull AgentCore traces into backend; optionally read evaluator scores as advisory labels.
The traces go through the same gate and report as any pulled source. A mixed-session trace the
source re-keyed is noted on its result. When with_scores is set, evaluator results are
mapped to advisory llm-judge labels and persisted, skipping any whose identity is already
stored (so re-reading the same results adds no duplicate). frame stamps every pulled trace
with that sampling frame, as ingest_langfuse does.
Source code in src/calibrated_agent_evaluation/agentcore/ingest.py
audit_clients ¶
AWS clients that can make only the audit's reads and the AgentCore ingest's trace reads.
The billed-call budget comes from CALIBRATED_AGENT_EVALUATION_AGENTCORE_MAX_BILLED_CALLS
when it is set.
Raises:
| Type | Description |
|---|---|
AgentCoreError
|
The profile does not exist or cannot be opened, or the budget variable is not a non-negative whole number. |
Source code in src/calibrated_agent_evaluation/agentcore/optimization.py
read_ab_test ¶
Read an A/B test and each arm's bundle version as a vendor-claim record.
Every variant must name exactly one of a bundle version and a gateway target; the model allows both or neither, so such a response is refused before any bundle version is read. A target arm is kept by name and reads no bundle.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
clients
|
AwsClients
|
The audit's clients. |
required |
ab_test_id
|
str
|
The A/B test to read. |
required |
now
|
datetime | None
|
The read time, for tests; the current UTC time otherwise. |
None
|
Returns:
| Type | Description |
|---|---|
AbTestClaimRecord
|
The claim record: the test's state, both arms, AgentCore's per-evaluator figures as |
AbTestClaimRecord
|
returned, whether the two bundle arms are identical, and whether the test is still running. |
Raises:
| Type | Description |
|---|---|
AgentCoreError
|
A call failed, naming the operation and the id, or a variant names both or neither configuration. |
Source code in src/calibrated_agent_evaluation/agentcore/optimization.py
240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 | |
read_bundle_version ¶
Read one configuration bundle version, keeping its components as digests only.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
clients
|
AwsClients
|
The audit's clients. |
required |
bundle_arn
|
str
|
The bundle's ARN; its last path segment is the bundle id. |
required |
version_id
|
str
|
The version to read. |
required |
Returns:
| Type | Description |
|---|---|
BundleVersionRef
|
The version's lineage and a digest of each component, of each top-level key of a |
BundleVersionRef
|
component whose configuration is a JSON object, and of the whole components map. |
Raises:
| Type | Description |
|---|---|
AgentCoreError
|
The call failed, naming |
Source code in src/calibrated_agent_evaluation/agentcore/optimization.py
agent_reported_error_trial ¶
The trial an agent-reported error stands for, or None for any other error.
Source code in src/calibrated_agent_evaluation/agentcore/policy_spans.py
extract_decisions ¶
Every per-call Policy decision in one trace's span records, in the order the calls started.
Spans without the decision attribute are skipped, and so are tools/list authorizations
and the Gateway's own call spans, which repeat the decision their per-call child records; a
call span whose child is missing, or whose value is unknown, is refused, and one that says a
log-only gateway decided asynchronously is skipped. A span the store holds more than once is
read once.
Raises:
| Type | Description |
|---|---|
PolicySpanMalformed
|
A span carries the decision attribute and is neither kind; or a per-call span's tool, decision or mode is missing or unknown, or a list attribute is in an unknown encoding. The message names the trace, the span and the attribute. |
Source code in src/calibrated_agent_evaluation/agentcore/policy_spans.py
156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 | |
agentcore_source ¶
Build the trace source for config: the Omni reader, or the Logs Insights fallback.
Omni is used when an Omni domain is configured; otherwise spans are read from CloudWatch's
aws/spans and content from the runtime's otel-rt-logs group via Logs Insights.
Source code in src/calibrated_agent_evaluation/agentcore/readers.py
read_scores ¶
Map a stream of AgentCore evaluation results to advisory labels.
Raises, at the record that cannot be mapped, the error result_to_label raises:
AgentCoreScoreIncomplete for a record with no trace or session id or no verdict, and
AgentCoreNoPassDirection for an evaluator with no pass direction. Nothing is skipped
silently; a caller that wants to skip such records maps each one with result_to_label
and counts the ones it skips, as
ingest_agentcore does.
Source code in src/calibrated_agent_evaluation/agentcore/scores.py
caller_is_role ¶
Whether caller_arn is a session of the IAM role role_arn (same account and name).
STS reports a role session as arn:aws:sts::<account>:assumed-role/<name>/<session>; the
role itself is arn:aws:iam::<account>:role/<path/><name>.
Source code in src/calibrated_agent_evaluation/agentcore/session.py
require_role ¶
The caller's ARN, after confirming it is a session of role_arn.
Only sts:GetCallerIdentity is called, so nothing else reaches the account under the wrong
identity.