calibrated_agent_evaluation.scenarios¶
calibrated_agent_evaluation.scenarios ¶
Test scenarios built from caught failures, grouped into dataset slices.
A scenario is a test case for a future run: an input, the expected answer and, optionally, the
expected tool order and plain-language assertions. Scenarios are built from human FAIL labels plus
a hand-written answer key and grouped into a DatasetSlice. Simulated scenarios (a persona
and a goal instead of scripted turns) use the same model, with assertions as their only ground
truth. This package is vendor-free; the AgentCore codec and publish/pull live in
agentcore.
Scenario
module-attribute
¶
BuildResult
dataclass
¶
The built slice (None when no trace qualified) and what was left out, by reason.
ScenarioError ¶
Bases: CalibratedAgentEvaluationDataError
A scenario, slice file or answer key is invalid, unreadable, or can't be built as asked.
ActorProfile
pydantic-model
¶
AnswerEntry
pydantic-model
¶
Bases: _Model
The expected tool order and assertions an answer key gives one failure mode or trace.
Fields:
-
assertions(list[str]) -
expected_trajectory(list[str])
Validators:
-
_check_entries→assertions,expected_trajectory
AnswerKey
pydantic-model
¶
Bases: _Model
The hand-written answer key: entries per failure mode, with optional per-trace overrides.
Fields:
-
by_failure_mode(dict[str, AnswerEntry]) -
by_trace(dict[str, AnswerEntry])
CalibratedAgentEvaluationScenarioMeta
pydantic-model
¶
Bases: _Model
The scenario fields only the library uses; None and empty mean "not recorded".
Fields:
-
source_trace_id(str | None) -
frame(Frame | None) -
taxonomy_cell(dict[str, str]) -
failure_modes(list[str]) -
expected(dict[str, Any] | None)
is_empty ¶
Whether every field is unset, so an encoder can leave the block out.
DatasetSlice
pydantic-model
¶
Bases: _Model
Scenarios of one form with unique ids, and the version they were published as, if any.
Fields:
-
schema_version(str) -
form(Form) -
scenarios(list[Scenario]) -
published(PublishedVersion | None)
Validators:
-
_one_form_unique_ids
PredefinedScenario
pydantic-model
¶
Bases: _ScenarioBase
A scripted conversation: fixed user turns, replayed as written.
Fields:
-
scenario_id(str) -
assertions(list[str]) -
metadata(dict[str, Any]) -
calibrated_agent_evaluation(CalibratedAgentEvaluationScenarioMeta) -
form(Literal['predefined']) -
turns(list[Turn]) -
expected_trajectory(list[str])
Validators:
-
_check_assertions→assertions -
_reserved_key→metadata -
_check_trajectory→expected_trajectory
PublishedVersion
pydantic-model
¶
Bases: _Model
Where a slice was published: the AgentCore dataset and the immutable version it became.
Fields:
-
service(Literal['agentcore']) -
dataset_arn(str) -
dataset_id(str) -
dataset_name(str) -
version(str) -
region(str) -
content_digest(str) -
published_at(str)
SimulatedScenario
pydantic-model
¶
Bases: _ScenarioBase
A conversation an LLM-played user drives toward a goal, so only assertions can judge it.
Fields:
-
scenario_id(str) -
assertions(list[str]) -
metadata(dict[str, Any]) -
calibrated_agent_evaluation(CalibratedAgentEvaluationScenarioMeta) -
form(Literal['simulated']) -
actor_profile(ActorProfile) -
input(str | dict[str, Any]) -
max_turns(int) -
scenario_description(str)
Validators:
-
_check_assertions→assertions -
_reserved_key→metadata -
_check_input→input
Turn
pydantic-model
¶
build_slice ¶
A predefined slice of one single-turn scenario per trace every rater labeled FAIL, by id.
Source code in src/calibrated_agent_evaluation/scenarios/build.py
expected_text ¶
A structured expected output as the compact, sorted-key JSON text a turn carries.
Source code in src/calibrated_agent_evaluation/scenarios/build.py
starter_key ¶
An answer key listing each failure mode in scope with empty lists, for a human to fill.
Source code in src/calibrated_agent_evaluation/scenarios/build.py
read_answer_key ¶
The answer key in the file at path; unknown fields and bad JSON are refused.
read_slice ¶
The slice in the file at path; a missing, unparsable or invalid file is refused.
The schema version is checked first, so a file from another version is named as such rather than reported through whichever field changed shape.
Source code in src/calibrated_agent_evaluation/scenarios/slicefile.py
write_slice ¶
Write slice_ to path, refusing to replace an existing file unless force.