Skip to content

calibrated_agent_evaluation.scenarios

calibrated_agent_evaluation.scenarios

Test scenarios built from caught failures, grouped into dataset slices.

A scenario is a test case for a future run: an input, the expected answer and, optionally, the expected tool order and plain-language assertions. Scenarios are built from human FAIL labels plus a hand-written answer key and grouped into a DatasetSlice. Simulated scenarios (a persona and a goal instead of scripted turns) use the same model, with assertions as their only ground truth. This package is vendor-free; the AgentCore codec and publish/pull live in agentcore.

Scenario module-attribute

Scenario = Annotated[
    PredefinedScenario | SimulatedScenario,
    Field(discriminator="form"),
]

BuildResult dataclass

BuildResult(slice, left_out=dict(), unused_modes=list())

The built slice (None when no trace qualified) and what was left out, by reason.

slice instance-attribute

slice

left_out class-attribute instance-attribute

left_out = field(default_factory=dict)

unused_modes class-attribute instance-attribute

unused_modes = field(default_factory=list)

n_scenarios property

n_scenarios

How many scenarios the slice holds.

ScenarioError

Bases: CalibratedAgentEvaluationDataError

A scenario, slice file or answer key is invalid, unreadable, or can't be built as asked.

ActorProfile pydantic-model

Bases: _Model

Who a simulated user is and what they want.

Fields:

context pydantic-field

context

goal pydantic-field

goal

traits pydantic-field

traits

AnswerEntry pydantic-model

Bases: _Model

The expected tool order and assertions an answer key gives one failure mode or trace.

Fields:

Validators:

assertions pydantic-field

assertions

expected_trajectory pydantic-field

expected_trajectory

AnswerKey pydantic-model

Bases: _Model

The hand-written answer key: entries per failure mode, with optional per-trace overrides.

Fields:

by_failure_mode pydantic-field

by_failure_mode

by_trace pydantic-field

by_trace

CalibratedAgentEvaluationScenarioMeta pydantic-model

Bases: _Model

The scenario fields only the library uses; None and empty mean "not recorded".

Fields:

source_trace_id pydantic-field

source_trace_id = None

frame pydantic-field

frame = None

taxonomy_cell pydantic-field

taxonomy_cell

failure_modes pydantic-field

failure_modes

expected pydantic-field

expected = None

is_empty

is_empty()

Whether every field is unset, so an encoder can leave the block out.

Source code in src/calibrated_agent_evaluation/scenarios/types.py
def is_empty(self) -> bool:
    """Whether every field is unset, so an encoder can leave the block out."""
    return self == CalibratedAgentEvaluationScenarioMeta()

DatasetSlice pydantic-model

Bases: _Model

Scenarios of one form with unique ids, and the version they were published as, if any.

Fields:

Validators:

  • _one_form_unique_ids

schema_version pydantic-field

schema_version = SLICE_SCHEMA_VERSION

form pydantic-field

form

scenarios pydantic-field

scenarios

published pydantic-field

published = None

PredefinedScenario pydantic-model

Bases: _ScenarioBase

A scripted conversation: fixed user turns, replayed as written.

Fields:

Validators:

  • _check_assertions → assertions
  • _reserved_key → metadata
  • _check_trajectory → expected_trajectory

form pydantic-field

form = 'predefined'

turns pydantic-field

turns

expected_trajectory pydantic-field

expected_trajectory

PublishedVersion pydantic-model

Bases: _Model

Where a slice was published: the AgentCore dataset and the immutable version it became.

Fields:

service pydantic-field

service = 'agentcore'

dataset_arn pydantic-field

dataset_arn

dataset_id pydantic-field

dataset_id

dataset_name pydantic-field

dataset_name

version pydantic-field

version

region pydantic-field

region

content_digest pydantic-field

content_digest

published_at pydantic-field

published_at

SimulatedScenario pydantic-model

Bases: _ScenarioBase

A conversation an LLM-played user drives toward a goal, so only assertions can judge it.

Fields:

Validators:

  • _check_assertions → assertions
  • _reserved_key → metadata
  • _check_input → input

form pydantic-field

form = 'simulated'

actor_profile pydantic-field

actor_profile

input pydantic-field

input

max_turns pydantic-field

max_turns = 10

scenario_description pydantic-field

scenario_description = ''

Turn pydantic-model

Bases: _Model

One scripted user turn and, when known, the answer the agent should give to it.

Fields:

Validators:

input pydantic-field

input

expected_response pydantic-field

expected_response = None

build_slice

build_slice(traces, labels, key=None, failure_mode=None)

A predefined slice of one single-turn scenario per trace every rater labeled FAIL, by id.

Source code in src/calibrated_agent_evaluation/scenarios/build.py
def build_slice(
    traces: Sequence[Trace],
    labels: Mapping[str, Sequence[Label]],
    key: AnswerKey | None = None,
    failure_mode: str | None = None,
) -> BuildResult:
    """A predefined slice of one single-turn scenario per trace every rater labeled FAIL, by id."""
    if key is None:
        key = AnswerKey()
    in_scope = {t.trace_id for t in traces}
    unknown = sorted(set(key.by_trace) - in_scope)
    if unknown:
        raise ScenarioError(
            f"the answer key's by_trace names trace(s) not in scope: {unknown}; fix the ids or "
            "widen the scope"
        )
    left_out: dict[str, list[str]] = {}
    scenarios: list[Scenario] = []
    ambiguous: dict[str, list[str]] = {}
    for failure in _failures(traces, labels, failure_mode, left_out):
        trace = failure.trace
        # The same rule as `calibrated-agent-evaluation score`: each rater's newest label that
        # records an expected.
        resolved = resolve_expected(
            newest_per_rater(lb for lb in failure.all_fails if lb.expected is not None).values()
        )
        if resolved.conflicting:
            left_out.setdefault(CONFLICTING_EXPECTED, []).append(trace.trace_id)
            continue
        if not trace.input:
            left_out.setdefault(NO_INPUT, []).append(trace.trace_id)
            continue
        answers = _answers_for(trace.trace_id, failure.modes, key, ambiguous)
        scenarios.append(
            PredefinedScenario(
                scenario_id=trace.trace_id,
                turns=[
                    Turn(
                        input=trace.input,
                        expected_response=None
                        if resolved.value is None
                        else expected_text(resolved.value),
                    )
                ],
                expected_trajectory=answers.expected_trajectory,
                assertions=answers.assertions,
                calibrated_agent_evaluation=CalibratedAgentEvaluationScenarioMeta(
                    source_trace_id=trace.trace_id,
                    frame=trace.frame,
                    taxonomy_cell=dict(trace.taxonomy_cell),
                    failure_modes=failure.modes,
                    expected=resolved.value,
                ),
            )
        )
    if ambiguous:
        detail = "; ".join(f"{tid}: {modes}" for tid, modes in sorted(ambiguous.items()))
        raise ScenarioError(
            "the answer key gives different expected tool orders for failure modes that share a "
            f"trace ({detail}); set the trace's expected_trajectory under by_trace"
        )
    used = {m for s in scenarios for m in s.calibrated_agent_evaluation.failure_modes}
    unused = [m for m in key.by_failure_mode if m not in used]
    # Sorted, so the same store content builds the same slice whatever order a backend reads in.
    scenarios.sort(key=lambda s: s.scenario_id)
    built = DatasetSlice(form="predefined", scenarios=scenarios) if scenarios else None
    return BuildResult(slice=built, left_out=left_out, unused_modes=unused)

expected_text

expected_text(expected)

A structured expected output as the compact, sorted-key JSON text a turn carries.

Source code in src/calibrated_agent_evaluation/scenarios/build.py
def expected_text(expected: dict[str, Any]) -> str:
    """A structured expected output as the compact, sorted-key JSON text a turn carries."""
    return json.dumps(expected, sort_keys=True, separators=(",", ":"), ensure_ascii=False)

starter_key

starter_key(traces, labels, failure_mode=None)

An answer key listing each failure mode in scope with empty lists, for a human to fill.

Source code in src/calibrated_agent_evaluation/scenarios/build.py
def starter_key(
    traces: Sequence[Trace], labels: Mapping[str, Sequence[Label]], failure_mode: str | None = None
) -> AnswerKey:
    """An answer key listing each failure mode in scope with empty lists, for a human to fill."""
    return AnswerKey(
        by_failure_mode={
            m: AnswerEntry() for m in failure_modes_in_scope(traces, labels, failure_mode)
        }
    )

read_answer_key

read_answer_key(path)

The answer key in the file at path; unknown fields and bad JSON are refused.

Source code in src/calibrated_agent_evaluation/scenarios/slicefile.py
def read_answer_key(path: str | Path) -> AnswerKey:
    """The answer key in the file at ``path``; unknown fields and bad JSON are refused."""
    return _read(Path(path), AnswerKey, "answer key")

read_slice

read_slice(path)

The slice in the file at path; a missing, unparsable or invalid file is refused.

The schema version is checked first, so a file from another version is named as such rather than reported through whichever field changed shape.

Source code in src/calibrated_agent_evaluation/scenarios/slicefile.py
def read_slice(path: str | Path) -> DatasetSlice:
    """The slice in the file at ``path``; a missing, unparsable or invalid file is refused.

    The schema version is checked first, so a file from another version is named as such rather
    than reported through whichever field changed shape.
    """
    return _read(Path(path), DatasetSlice, "slice file", version=SLICE_SCHEMA_VERSION)

write_slice

write_slice(path, slice_, *, force=False)

Write slice_ to path, refusing to replace an existing file unless force.

Source code in src/calibrated_agent_evaluation/scenarios/slicefile.py
def write_slice(path: str | Path, slice_: DatasetSlice, *, force: bool = False) -> None:
    """Write ``slice_`` to ``path``, refusing to replace an existing file unless ``force``."""
    write_json_model(Path(path), slice_, force=force)