Simulator

Placeholder — not implemented

This package defines the intended interfaces for conversation generation, scenario orchestration, and test-package generation, but every method raises NotImplementedError. It is documented here so the planned shape is discoverable; do not build against it yet.

`uaef.simulator.interfaces`

Conversation Simulator Interface Definitions.

This module defines the input and output specifications for the conversation simulator. These interfaces provide the API contract for future simulator implementations.

PLACEHOLDER: Full implementation deferred to future work.

`AgentConfig`

Bases: BaseModel

Configuration for the agent being tested.

Defines the framework, model, tools, and system prompt for the agent that will participate in the simulated conversation.

Attributes:

Name	Type	Description
`framework`	`str`	Agent framework (e.g., "langgraph", "bedrock", "langchain")
`model`	`str`	LLM model identifier (e.g., "claude-3-sonnet", "gpt-4")
`tools`	`List[str]`	List of available tool names
`system_prompt`	`str`	System prompt for the agent
`parameters`	`Dict[str, Any]`	Additional agent parameters (temperature, max_tokens, etc.)

Source code in uaef/simulator/interfaces.py

class AgentConfig(BaseModel):
    """
    Configuration for the agent being tested.

    Defines the framework, model, tools, and system prompt for the agent
    that will participate in the simulated conversation.

    Attributes:
        framework: Agent framework (e.g., "langgraph", "bedrock", "langchain")
        model: LLM model identifier (e.g., "claude-3-sonnet", "gpt-4")
        tools: List of available tool names
        system_prompt: System prompt for the agent
        parameters: Additional agent parameters (temperature, max_tokens, etc.)
    """

    framework: str = Field(..., description="Agent framework")
    model: str = Field(..., description="LLM model identifier")
    tools: List[str] = Field(
        default_factory=list,
        description="List of available tool names"
    )
    system_prompt: str = Field(..., description="System prompt for the agent")
    parameters: Dict[str, Any] = Field(
        default_factory=dict,
        description="Additional agent parameters"
    )

`CommunicationStyle`

Bases: str, Enum

Communication style for user behavior.

Source code in uaef/simulator/interfaces.py

class CommunicationStyle(str, Enum):
    """Communication style for user behavior."""

    DIRECT = "direct"
    VERBOSE = "verbose"
    TERSE = "terse"
    TECHNICAL = "technical"
    NON_TECHNICAL = "non_technical"

`ComplexityLevel`

Bases: str, Enum

Complexity level for scenario generation.

Source code in uaef/simulator/interfaces.py

class ComplexityLevel(str, Enum):
    """Complexity level for scenario generation."""

    SIMPLE = "simple"
    MODERATE = "moderate"
    COMPLEX = "complex"
    EXPERT = "expert"

`GeneratedConversation`

Bases: BaseModel

A generated conversation trace.

Contains the complete conversation as a list of Message objects with role, content, tool calls, and timestamps.

Attributes:

Name	Type	Description
`conversation_id`	`str`	Unique identifier for the conversation
`scenario_id`	`str`	ID of the scenario this conversation is based on
`messages`	`List[Message]`	List of messages in the conversation
`turn_count`	`int`	Number of turns in the conversation
`completed`	`bool`	Whether the conversation completed successfully
`completion_reason`	`str`	Reason for conversation completion
`metadata`	`Dict[str, Any]`	Additional metadata about the conversation

Source code in uaef/simulator/interfaces.py

class GeneratedConversation(BaseModel):
    """
    A generated conversation trace.

    Contains the complete conversation as a list of Message objects with
    role, content, tool calls, and timestamps.

    Attributes:
        conversation_id: Unique identifier for the conversation
        scenario_id: ID of the scenario this conversation is based on
        messages: List of messages in the conversation
        turn_count: Number of turns in the conversation
        completed: Whether the conversation completed successfully
        completion_reason: Reason for conversation completion
        metadata: Additional metadata about the conversation
    """

    conversation_id: str = Field(
        ...,
        description="Unique identifier for the conversation"
    )
    scenario_id: str = Field(
        ...,
        description="ID of the scenario this conversation is based on"
    )
    messages: List[Message] = Field(
        ...,
        description="List of messages in the conversation"
    )
    turn_count: int = Field(..., ge=0, description="Number of turns in the conversation")
    completed: bool = Field(..., description="Whether the conversation completed successfully")
    completion_reason: str = Field(
        ...,
        description="Reason for conversation completion"
    )
    metadata: Dict[str, Any] = Field(
        default_factory=dict,
        description="Additional metadata about the conversation"
    )

`GeneratedGroundTruth`

Bases: BaseModel

Ground truth for the entire conversation.

Contains expected outputs, tool calls, and arguments for each turn to enable automated evaluation.

Attributes:

Name	Type	Description
`conversation_id`	`str`	ID of the conversation this ground truth is for
`turn_ground_truths`	`List[TurnGroundTruth]`	Ground truth for each turn
`overall_success_criteria`	`List[str]`	Overall success criteria for the conversation
`metadata`	`Dict[str, Any]`	Additional metadata about the ground truth

Source code in uaef/simulator/interfaces.py

class GeneratedGroundTruth(BaseModel):
    """
    Ground truth for the entire conversation.

    Contains expected outputs, tool calls, and arguments for each turn
    to enable automated evaluation.

    Attributes:
        conversation_id: ID of the conversation this ground truth is for
        turn_ground_truths: Ground truth for each turn
        overall_success_criteria: Overall success criteria for the conversation
        metadata: Additional metadata about the ground truth
    """

    conversation_id: str = Field(
        ...,
        description="ID of the conversation this ground truth is for"
    )
    turn_ground_truths: List[TurnGroundTruth] = Field(
        ...,
        description="Ground truth for each turn"
    )
    overall_success_criteria: List[str] = Field(
        default_factory=list,
        description="Overall success criteria for the conversation"
    )
    metadata: Dict[str, Any] = Field(
        default_factory=dict,
        description="Additional metadata about the ground truth"
    )

`GeneratedScenario`

Bases: BaseModel

A generated test scenario.

Contains the scenario definition including ID, name, description, initial context, user goal, success criteria, and expected turns.

Attributes:

Name	Type	Description
`scenario_id`	`str`	Unique identifier for the scenario
`name`	`str`	Human-readable name for the scenario
`description`	`str`	Detailed description of the scenario
`initial_context`	`str`	Initial context for the conversation
`user_goal`	`str`	The goal the user is trying to achieve
`success_criteria`	`List[str]`	Criteria that define successful completion
`expected_turns`	`int`	Expected number of conversation turns
`metadata`	`Dict[str, Any]`	Additional metadata about the scenario

Source code in uaef/simulator/interfaces.py

class GeneratedScenario(BaseModel):
    """
    A generated test scenario.

    Contains the scenario definition including ID, name, description,
    initial context, user goal, success criteria, and expected turns.

    Attributes:
        scenario_id: Unique identifier for the scenario
        name: Human-readable name for the scenario
        description: Detailed description of the scenario
        initial_context: Initial context for the conversation
        user_goal: The goal the user is trying to achieve
        success_criteria: Criteria that define successful completion
        expected_turns: Expected number of conversation turns
        metadata: Additional metadata about the scenario
    """

    scenario_id: str = Field(..., description="Unique identifier for the scenario")
    name: str = Field(..., description="Human-readable name for the scenario")
    description: str = Field(..., description="Detailed description of the scenario")
    initial_context: str = Field(..., description="Initial context for the conversation")
    user_goal: str = Field(..., description="Goal the user is trying to achieve")
    success_criteria: List[str] = Field(
        ...,
        description="Criteria that define successful completion"
    )
    expected_turns: int = Field(
        ...,
        ge=1,
        description="Expected number of conversation turns"
    )
    metadata: Dict[str, Any] = Field(
        default_factory=dict,
        description="Additional metadata about the scenario"
    )

`GenerationParameters`

Bases: BaseModel

Parameters for conversation generation.

Controls the generation process including max turns, adversarial mode, and ambiguity level.

Attributes:

Name	Type	Description
`max_turns`	`int`	Maximum number of conversation turns
`adversarial_mode`	`bool`	Whether to generate adversarial user behavior
`ambiguity_level`	`float`	Level of ambiguity in user messages (0-1)
`include_edge_cases`	`bool`	Whether to include edge cases
`seed`	`Optional[int]`	Random seed for reproducibility

Source code in uaef/simulator/interfaces.py

class GenerationParameters(BaseModel):
    """
    Parameters for conversation generation.

    Controls the generation process including max turns, adversarial mode,
    and ambiguity level.

    Attributes:
        max_turns: Maximum number of conversation turns
        adversarial_mode: Whether to generate adversarial user behavior
        ambiguity_level: Level of ambiguity in user messages (0-1)
        include_edge_cases: Whether to include edge cases
        seed: Random seed for reproducibility
    """

    max_turns: int = Field(
        default=10,
        ge=1,
        description="Maximum number of conversation turns"
    )
    adversarial_mode: bool = Field(
        default=False,
        description="Whether to generate adversarial user behavior"
    )
    ambiguity_level: float = Field(
        default=0.3,
        ge=0.0,
        le=1.0,
        description="Level of ambiguity in user messages (0-1)"
    )
    include_edge_cases: bool = Field(
        default=False,
        description="Whether to include edge cases"
    )
    seed: Optional[int] = Field(
        None,
        description="Random seed for reproducibility"
    )

`KnowledgeLevel`

Bases: str, Enum

Knowledge level of the simulated user.

Source code in uaef/simulator/interfaces.py

class KnowledgeLevel(str, Enum):
    """Knowledge level of the simulated user."""

    NOVICE = "novice"
    INTERMEDIATE = "intermediate"
    EXPERT = "expert"

`ScenarioConfig`

Bases: BaseModel

Configuration for scenario generation.

Defines the domain, user goal, success criteria, and complexity level for generating a test scenario.

Attributes:

Name	Type	Description
`domain`	`str`	The domain or topic area (e.g., "customer_support", "data_analysis")
`user_goal`	`str`	The goal the user is trying to achieve
`success_criteria`	`List[str]`	List of criteria that define successful completion
`complexity_level`	`ComplexityLevel`	How complex the scenario should be
`initial_context`	`Optional[str]`	Optional initial context or background information
`constraints`	`Dict[str, Any]`	Optional constraints or limitations for the scenario

Source code in uaef/simulator/interfaces.py

class ScenarioConfig(BaseModel):
    """
    Configuration for scenario generation.

    Defines the domain, user goal, success criteria, and complexity level
    for generating a test scenario.

    Attributes:
        domain: The domain or topic area (e.g., "customer_support", "data_analysis")
        user_goal: The goal the user is trying to achieve
        success_criteria: List of criteria that define successful completion
        complexity_level: How complex the scenario should be
        initial_context: Optional initial context or background information
        constraints: Optional constraints or limitations for the scenario
    """

    domain: str = Field(..., description="Domain or topic area for the scenario")
    user_goal: str = Field(..., description="Goal the user is trying to achieve")
    success_criteria: List[str] = Field(
        ...,
        description="Criteria that define successful completion"
    )
    complexity_level: ComplexityLevel = Field(
        default=ComplexityLevel.MODERATE,
        description="Complexity level for the scenario"
    )
    initial_context: Optional[str] = Field(
        None,
        description="Initial context or background information"
    )
    constraints: Dict[str, Any] = Field(
        default_factory=dict,
        description="Constraints or limitations for the scenario"
    )

`TestPackage`

Bases: BaseModel

Complete test package for evaluation.

Bundles scenario, conversation, ground truth, and metadata into a single package ready for automated evaluation.

Attributes:

Name	Type	Description
`package_id`	`str`	Unique identifier for the test package
`scenario`	`GeneratedScenario`	The generated scenario
`conversation`	`GeneratedConversation`	The generated conversation
`ground_truth`	`GeneratedGroundTruth`	The generated ground truth
`generation_timestamp`	`datetime`	When the package was generated
`generation_parameters`	`GenerationParameters`	Parameters used for generation
`metadata`	`Dict[str, Any]`	Additional metadata about the test package

Source code in uaef/simulator/interfaces.py

class TestPackage(BaseModel):
    """
    Complete test package for evaluation.

    Bundles scenario, conversation, ground truth, and metadata into a
    single package ready for automated evaluation.

    Attributes:
        package_id: Unique identifier for the test package
        scenario: The generated scenario
        conversation: The generated conversation
        ground_truth: The generated ground truth
        generation_timestamp: When the package was generated
        generation_parameters: Parameters used for generation
        metadata: Additional metadata about the test package
    """

    package_id: str = Field(..., description="Unique identifier for the test package")
    scenario: GeneratedScenario = Field(..., description="The generated scenario")
    conversation: GeneratedConversation = Field(..., description="The generated conversation")
    ground_truth: GeneratedGroundTruth = Field(..., description="The generated ground truth")
    generation_timestamp: datetime = Field(
        ...,
        description="When the package was generated"
    )
    generation_parameters: GenerationParameters = Field(
        ...,
        description="Parameters used for generation"
    )
    metadata: Dict[str, Any] = Field(
        default_factory=dict,
        description="Additional metadata about the test package"
    )

    @field_validator("generation_timestamp")
    @classmethod
    def validate_timestamp(cls, v: datetime) -> datetime:
        """Validate timestamp format."""
        if v is None:
            raise ValueError("Generation timestamp cannot be None")
        return v

`validate_timestamp(v)` `classmethod`

Validate timestamp format.

Source code in uaef/simulator/interfaces.py

@field_validator("generation_timestamp")
@classmethod
def validate_timestamp(cls, v: datetime) -> datetime:
    """Validate timestamp format."""
    if v is None:
        raise ValueError("Generation timestamp cannot be None")
    return v

`TurnGroundTruth`

Bases: BaseModel

Ground truth for a single conversation turn.

Contains expected output, tool calls, and arguments for validation.

Attributes:

Name	Type	Description
`turn_number`	`int`	Turn number (0-indexed)
`expected_output`	`Optional[str]`	Expected agent output for this turn
`expected_tool_calls`	`List[str]`	Expected tool calls for this turn
`expected_arguments`	`Dict[str, Dict[str, Any]]`	Expected arguments for each tool call
`validation_criteria`	`List[str]`	Additional validation criteria for this turn

Source code in uaef/simulator/interfaces.py

class TurnGroundTruth(BaseModel):
    """
    Ground truth for a single conversation turn.

    Contains expected output, tool calls, and arguments for validation.

    Attributes:
        turn_number: Turn number (0-indexed)
        expected_output: Expected agent output for this turn
        expected_tool_calls: Expected tool calls for this turn
        expected_arguments: Expected arguments for each tool call
        validation_criteria: Additional validation criteria for this turn
    """

    turn_number: int = Field(..., ge=0, description="Turn number (0-indexed)")
    expected_output: Optional[str] = Field(
        None,
        description="Expected agent output for this turn"
    )
    expected_tool_calls: List[str] = Field(
        default_factory=list,
        description="Expected tool calls for this turn"
    )
    expected_arguments: Dict[str, Dict[str, Any]] = Field(
        default_factory=dict,
        description="Expected arguments for each tool call"
    )
    validation_criteria: List[str] = Field(
        default_factory=list,
        description="Additional validation criteria for this turn"
    )

`UserBehaviorModel`

Bases: BaseModel

Model for simulating user behavior in conversations.

Defines personality traits, patience level, knowledge level, and communication style for generating realistic user messages.

Attributes:

Name	Type	Description
`personality`	`str`	Personality traits (e.g., "frustrated", "patient", "curious")
`patience_level`	`int`	How patient the user is (1-10, higher = more patient)
`knowledge_level`	`KnowledgeLevel`	User's knowledge level in the domain
`communication_style`	`CommunicationStyle`	How the user communicates
`error_tolerance`	`float`	How tolerant the user is of errors (0-1)
`clarification_frequency`	`float`	How often user asks for clarification (0-1)

Source code in uaef/simulator/interfaces.py

class UserBehaviorModel(BaseModel):
    """
    Model for simulating user behavior in conversations.

    Defines personality traits, patience level, knowledge level, and
    communication style for generating realistic user messages.

    Attributes:
        personality: Personality traits (e.g., "frustrated", "patient", "curious")
        patience_level: How patient the user is (1-10, higher = more patient)
        knowledge_level: User's knowledge level in the domain
        communication_style: How the user communicates
        error_tolerance: How tolerant the user is of errors (0-1)
        clarification_frequency: How often user asks for clarification (0-1)
    """

    personality: str = Field(
        default="neutral",
        description="Personality traits of the user"
    )
    patience_level: int = Field(
        default=5,
        ge=1,
        le=10,
        description="Patience level (1-10, higher = more patient)"
    )
    knowledge_level: KnowledgeLevel = Field(
        default=KnowledgeLevel.INTERMEDIATE,
        description="User's knowledge level in the domain"
    )
    communication_style: CommunicationStyle = Field(
        default=CommunicationStyle.DIRECT,
        description="How the user communicates"
    )
    error_tolerance: float = Field(
        default=0.5,
        ge=0.0,
        le=1.0,
        description="Tolerance for errors (0-1)"
    )
    clarification_frequency: float = Field(
        default=0.3,
        ge=0.0,
        le=1.0,
        description="Frequency of clarification requests (0-1)"
    )

`uaef.simulator.generator`

Conversation Generator - PLACEHOLDER Implementation.

This module provides placeholder implementations for conversation generation. Full implementation will be added in future work.

TODO: Implement full conversation generation logic including: - LLM-based user message generation - Agent response simulation - Turn-by-turn conversation orchestration - Realistic user behavior modeling

`ConversationGenerator`

Generates simulated conversations between users and agents.

PLACEHOLDER: This is a stub implementation. Full conversation generation logic will be implemented in future work.

The full implementation should: - Generate realistic user messages based on UserBehaviorModel - Simulate agent responses using the configured agent - Handle turn-by-turn conversation flow - Apply personality traits and communication styles - Terminate conversations based on goal achievement or patience exhaustion - Generate complete conversation traces with timestamps

Example usage (future): generator = ConversationGenerator() conversation = generator.generate_conversation( scenario=scenario_config, user_model=user_behavior, agent_config=agent_config, parameters=generation_params )

Source code in uaef/simulator/generator.py

class ConversationGenerator:
    """
    Generates simulated conversations between users and agents.

    PLACEHOLDER: This is a stub implementation. Full conversation generation
    logic will be implemented in future work.

    The full implementation should:
    - Generate realistic user messages based on UserBehaviorModel
    - Simulate agent responses using the configured agent
    - Handle turn-by-turn conversation flow
    - Apply personality traits and communication styles
    - Terminate conversations based on goal achievement or patience exhaustion
    - Generate complete conversation traces with timestamps

    Example usage (future):
        generator = ConversationGenerator()
        conversation = generator.generate_conversation(
            scenario=scenario_config,
            user_model=user_behavior,
            agent_config=agent_config,
            parameters=generation_params
        )
    """

    def __init__(self):
        """Initialize the conversation generator."""
        # TODO: Initialize LLM client for user message generation
        # TODO: Initialize agent client for agent response simulation
        # TODO: Load conversation templates and patterns
        pass

    def generate_conversation(
        self,
        scenario: ScenarioConfig,
        user_model: UserBehaviorModel,
        agent_config: AgentConfig,
        parameters: GenerationParameters,
    ) -> GeneratedConversation:
        """
        Generate a simulated conversation based on scenario and user model.

        This method should:
        1. Initialize conversation with scenario context
        2. Generate initial user message based on user_goal
        3. For each turn:
           a. Simulate agent response using agent_config
           b. Check if goal is achieved (success_criteria)
           c. Generate next user message based on user_model and agent response
           d. Decrement patience if agent response is poor
           e. Break if goal achieved or patience exhausted
        4. Return complete conversation trace

        Args:
            scenario: Scenario configuration defining the test case
            user_model: User behavior model for message generation
            agent_config: Agent configuration for response simulation
            parameters: Generation parameters (max_turns, adversarial_mode, etc.)

        Returns:
            GeneratedConversation with complete message history

        Raises:
            NotImplementedError: This is a placeholder implementation

        TODO: Implement full conversation generation logic
        TODO: Add support for adversarial mode
        TODO: Add support for ambiguity injection
        TODO: Add support for edge case generation
        TODO: Add conversation quality validation
        """
        raise NotImplementedError(
            "ConversationGenerator.generate_conversation is not yet implemented. "
            "This is a placeholder for future conversation simulation functionality. "
            "Full implementation will include LLM-based user message generation, "
            "agent response simulation, and turn-by-turn orchestration."
        )

    def generate_user_message(
        self,
        context: str,
        user_model: UserBehaviorModel,
        previous_agent_response: Optional[str] = None,
    ) -> str:
        """
        Generate a user message based on context and user model.

        This method should:
        1. Apply personality traits to message generation
        2. Apply communication style (direct, verbose, terse, etc.)
        3. Adjust message based on knowledge level
        4. Consider previous agent response for follow-up
        5. Inject ambiguity if configured
        6. Apply adversarial patterns if configured

        Args:
            context: Current conversation context
            user_model: User behavior model
            previous_agent_response: Previous agent response (optional)

        Returns:
            Generated user message

        Raises:
            NotImplementedError: This is a placeholder implementation

        TODO: Implement LLM-based user message generation
        TODO: Add personality trait application
        TODO: Add communication style templates
        TODO: Add knowledge level adjustment
        """
        raise NotImplementedError(
            "ConversationGenerator.generate_user_message is not yet implemented. "
            "This is a placeholder for future user message generation functionality."
        )

    def simulate_agent_response(
        self,
        user_message: str,
        agent_config: AgentConfig,
        conversation_history: list,
    ) -> tuple[str, list]:
        """
        Simulate an agent response to a user message.

        This method should:
        1. Call the agent with user_message and conversation_history
        2. Extract agent response text
        3. Extract tool calls made by the agent
        4. Return both response and tool calls

        Args:
            user_message: User message to respond to
            agent_config: Agent configuration
            conversation_history: Previous conversation messages

        Returns:
            Tuple of (agent_response, tool_calls)

        Raises:
            NotImplementedError: This is a placeholder implementation

        TODO: Implement agent invocation logic
        TODO: Add support for multiple agent frameworks
        TODO: Add tool call extraction
        TODO: Add error handling for agent failures
        """
        raise NotImplementedError(
            "ConversationGenerator.simulate_agent_response is not yet implemented. "
            "This is a placeholder for future agent response simulation functionality."
        )

    def check_goal_achievement(
        self,
        conversation_history: list,
        success_criteria: list[str],
    ) -> tuple[bool, str]:
        """
        Check if the conversation has achieved its goal.

        This method should:
        1. Evaluate each success criterion against conversation history
        2. Determine if all criteria are met
        3. Return achievement status and reason

        Args:
            conversation_history: Complete conversation history
            success_criteria: List of success criteria to check

        Returns:
            Tuple of (goal_achieved, reason)

        Raises:
            NotImplementedError: This is a placeholder implementation

        TODO: Implement success criteria evaluation
        TODO: Add LLM-based goal achievement detection
        TODO: Add partial goal achievement tracking
        """
        raise NotImplementedError(
            "ConversationGenerator.check_goal_achievement is not yet implemented. "
            "This is a placeholder for future goal achievement checking functionality."
        )

`check_goal_achievement(conversation_history, success_criteria)`

Check if the conversation has achieved its goal.

This method should: 1. Evaluate each success criterion against conversation history 2. Determine if all criteria are met 3. Return achievement status and reason

Parameters:

Name	Type	Description	Default
`conversation_history`	`list`	Complete conversation history	required
`success_criteria`	`list[str]`	List of success criteria to check	required

Returns:

Type	Description
`tuple[bool, str]`	Tuple of (goal_achieved, reason)

Raises:

Type	Description
`NotImplementedError`	This is a placeholder implementation

TODO: Implement success criteria evaluation TODO: Add LLM-based goal achievement detection TODO: Add partial goal achievement tracking

Source code in uaef/simulator/generator.py

def check_goal_achievement(
    self,
    conversation_history: list,
    success_criteria: list[str],
) -> tuple[bool, str]:
    """
    Check if the conversation has achieved its goal.

    This method should:
    1. Evaluate each success criterion against conversation history
    2. Determine if all criteria are met
    3. Return achievement status and reason

    Args:
        conversation_history: Complete conversation history
        success_criteria: List of success criteria to check

    Returns:
        Tuple of (goal_achieved, reason)

    Raises:
        NotImplementedError: This is a placeholder implementation

    TODO: Implement success criteria evaluation
    TODO: Add LLM-based goal achievement detection
    TODO: Add partial goal achievement tracking
    """
    raise NotImplementedError(
        "ConversationGenerator.check_goal_achievement is not yet implemented. "
        "This is a placeholder for future goal achievement checking functionality."
    )

`generate_conversation(scenario, user_model, agent_config, parameters)`

Generate a simulated conversation based on scenario and user model.

This method should: 1. Initialize conversation with scenario context 2. Generate initial user message based on user_goal 3. For each turn: a. Simulate agent response using agent_config b. Check if goal is achieved (success_criteria) c. Generate next user message based on user_model and agent response d. Decrement patience if agent response is poor e. Break if goal achieved or patience exhausted 4. Return complete conversation trace

Parameters:

Name	Type	Description	Default
`scenario`	`ScenarioConfig`	Scenario configuration defining the test case	required
`user_model`	`UserBehaviorModel`	User behavior model for message generation	required
`agent_config`	`AgentConfig`	Agent configuration for response simulation	required
`parameters`	`GenerationParameters`	Generation parameters (max_turns, adversarial_mode, etc.)	required

Returns:

Type	Description
`GeneratedConversation`	GeneratedConversation with complete message history

Raises:

Type	Description
`NotImplementedError`	This is a placeholder implementation

TODO: Implement full conversation generation logic TODO: Add support for adversarial mode TODO: Add support for ambiguity injection TODO: Add support for edge case generation TODO: Add conversation quality validation

Source code in uaef/simulator/generator.py

def generate_conversation(
    self,
    scenario: ScenarioConfig,
    user_model: UserBehaviorModel,
    agent_config: AgentConfig,
    parameters: GenerationParameters,
) -> GeneratedConversation:
    """
    Generate a simulated conversation based on scenario and user model.

    This method should:
    1. Initialize conversation with scenario context
    2. Generate initial user message based on user_goal
    3. For each turn:
       a. Simulate agent response using agent_config
       b. Check if goal is achieved (success_criteria)
       c. Generate next user message based on user_model and agent response
       d. Decrement patience if agent response is poor
       e. Break if goal achieved or patience exhausted
    4. Return complete conversation trace

    Args:
        scenario: Scenario configuration defining the test case
        user_model: User behavior model for message generation
        agent_config: Agent configuration for response simulation
        parameters: Generation parameters (max_turns, adversarial_mode, etc.)

    Returns:
        GeneratedConversation with complete message history

    Raises:
        NotImplementedError: This is a placeholder implementation

    TODO: Implement full conversation generation logic
    TODO: Add support for adversarial mode
    TODO: Add support for ambiguity injection
    TODO: Add support for edge case generation
    TODO: Add conversation quality validation
    """
    raise NotImplementedError(
        "ConversationGenerator.generate_conversation is not yet implemented. "
        "This is a placeholder for future conversation simulation functionality. "
        "Full implementation will include LLM-based user message generation, "
        "agent response simulation, and turn-by-turn orchestration."
    )

`generate_user_message(context, user_model, previous_agent_response=None)`

Generate a user message based on context and user model.

This method should: 1. Apply personality traits to message generation 2. Apply communication style (direct, verbose, terse, etc.) 3. Adjust message based on knowledge level 4. Consider previous agent response for follow-up 5. Inject ambiguity if configured 6. Apply adversarial patterns if configured

Parameters:

Name	Type	Description	Default
`context`	`str`	Current conversation context	required
`user_model`	`UserBehaviorModel`	User behavior model	required
`previous_agent_response`	`Optional[str]`	Previous agent response (optional)	`None`

Returns:

Type	Description
`str`	Generated user message

Raises:

Type	Description
`NotImplementedError`	This is a placeholder implementation

TODO: Implement LLM-based user message generation TODO: Add personality trait application TODO: Add communication style templates TODO: Add knowledge level adjustment

Source code in uaef/simulator/generator.py

def generate_user_message(
    self,
    context: str,
    user_model: UserBehaviorModel,
    previous_agent_response: Optional[str] = None,
) -> str:
    """
    Generate a user message based on context and user model.

    This method should:
    1. Apply personality traits to message generation
    2. Apply communication style (direct, verbose, terse, etc.)
    3. Adjust message based on knowledge level
    4. Consider previous agent response for follow-up
    5. Inject ambiguity if configured
    6. Apply adversarial patterns if configured

    Args:
        context: Current conversation context
        user_model: User behavior model
        previous_agent_response: Previous agent response (optional)

    Returns:
        Generated user message

    Raises:
        NotImplementedError: This is a placeholder implementation

    TODO: Implement LLM-based user message generation
    TODO: Add personality trait application
    TODO: Add communication style templates
    TODO: Add knowledge level adjustment
    """
    raise NotImplementedError(
        "ConversationGenerator.generate_user_message is not yet implemented. "
        "This is a placeholder for future user message generation functionality."
    )

`simulate_agent_response(user_message, agent_config, conversation_history)`

Simulate an agent response to a user message.

This method should: 1. Call the agent with user_message and conversation_history 2. Extract agent response text 3. Extract tool calls made by the agent 4. Return both response and tool calls

Parameters:

Name	Type	Description	Default
`user_message`	`str`	User message to respond to	required
`agent_config`	`AgentConfig`	Agent configuration	required
`conversation_history`	`list`	Previous conversation messages	required

Returns:

Type	Description
`tuple[str, list]`	Tuple of (agent_response, tool_calls)

Raises:

Type	Description
`NotImplementedError`	This is a placeholder implementation

TODO: Implement agent invocation logic TODO: Add support for multiple agent frameworks TODO: Add tool call extraction TODO: Add error handling for agent failures

Source code in uaef/simulator/generator.py

def simulate_agent_response(
    self,
    user_message: str,
    agent_config: AgentConfig,
    conversation_history: list,
) -> tuple[str, list]:
    """
    Simulate an agent response to a user message.

    This method should:
    1. Call the agent with user_message and conversation_history
    2. Extract agent response text
    3. Extract tool calls made by the agent
    4. Return both response and tool calls

    Args:
        user_message: User message to respond to
        agent_config: Agent configuration
        conversation_history: Previous conversation messages

    Returns:
        Tuple of (agent_response, tool_calls)

    Raises:
        NotImplementedError: This is a placeholder implementation

    TODO: Implement agent invocation logic
    TODO: Add support for multiple agent frameworks
    TODO: Add tool call extraction
    TODO: Add error handling for agent failures
    """
    raise NotImplementedError(
        "ConversationGenerator.simulate_agent_response is not yet implemented. "
        "This is a placeholder for future agent response simulation functionality."
    )

`uaef.simulator.orchestrator`

Scenario Orchestrator - PLACEHOLDER Implementation.

This module provides placeholder implementations for scenario orchestration. Full implementation will be added in future work.

TODO: Implement full scenario orchestration logic including: - Scenario loading and validation - Multi-scenario execution - Parallel scenario processing - Result aggregation

`ScenarioOrchestrator`

Orchestrates execution of multiple test scenarios.

PLACEHOLDER: This is a stub implementation. Full scenario orchestration logic will be implemented in future work.

The full implementation should: - Load and validate scenario configurations - Execute multiple scenarios in parallel - Manage scenario dependencies - Aggregate results across scenarios - Handle scenario failures gracefully - Provide progress tracking

Example usage (future): orchestrator = ScenarioOrchestrator() results = orchestrator.execute_scenario( scenario=scenario_config, user_model=user_behavior, agent_config=agent_config, parameters=generation_params )

Source code in uaef/simulator/orchestrator.py

class ScenarioOrchestrator:
    """
    Orchestrates execution of multiple test scenarios.

    PLACEHOLDER: This is a stub implementation. Full scenario orchestration
    logic will be implemented in future work.

    The full implementation should:
    - Load and validate scenario configurations
    - Execute multiple scenarios in parallel
    - Manage scenario dependencies
    - Aggregate results across scenarios
    - Handle scenario failures gracefully
    - Provide progress tracking

    Example usage (future):
        orchestrator = ScenarioOrchestrator()
        results = orchestrator.execute_scenario(
            scenario=scenario_config,
            user_model=user_behavior,
            agent_config=agent_config,
            parameters=generation_params
        )
    """

    def __init__(self):
        """Initialize the scenario orchestrator."""
        # TODO: Initialize conversation generator
        # TODO: Initialize result storage
        # TODO: Initialize progress tracker
        pass

    def execute_scenario(
        self,
        scenario: ScenarioConfig,
        user_model: UserBehaviorModel,
        agent_config: AgentConfig,
        parameters: GenerationParameters,
    ) -> GeneratedConversation:
        """
        Execute a single test scenario.

        This method should:
        1. Validate scenario configuration
        2. Initialize conversation generator
        3. Generate conversation using the scenario
        4. Validate conversation quality
        5. Return generated conversation

        Args:
            scenario: Scenario configuration to execute
            user_model: User behavior model for the scenario
            agent_config: Agent configuration for the scenario
            parameters: Generation parameters

        Returns:
            GeneratedConversation from scenario execution

        Raises:
            NotImplementedError: This is a placeholder implementation

        TODO: Implement scenario execution logic
        TODO: Add scenario validation
        TODO: Add conversation quality checks
        TODO: Add error handling and retries
        """
        raise NotImplementedError(
            "ScenarioOrchestrator.execute_scenario is not yet implemented. "
            "This is a placeholder for future scenario execution functionality. "
            "Full implementation will include scenario validation, conversation "
            "generation, and quality checks."
        )

    def execute_multiple_scenarios(
        self,
        scenarios: List[ScenarioConfig],
        user_models: List[UserBehaviorModel],
        agent_config: AgentConfig,
        parameters: GenerationParameters,
    ) -> List[GeneratedConversation]:
        """
        Execute multiple test scenarios in parallel.

        This method should:
        1. Validate all scenario configurations
        2. Create execution plan (parallel/sequential)
        3. Execute scenarios with progress tracking
        4. Aggregate results
        5. Handle failures gracefully

        Args:
            scenarios: List of scenario configurations
            user_models: List of user behavior models (one per scenario)
            agent_config: Agent configuration (shared across scenarios)
            parameters: Generation parameters

        Returns:
            List of GeneratedConversation objects

        Raises:
            NotImplementedError: This is a placeholder implementation

        TODO: Implement parallel scenario execution
        TODO: Add progress tracking
        TODO: Add result aggregation
        TODO: Add failure handling
        """
        raise NotImplementedError(
            "ScenarioOrchestrator.execute_multiple_scenarios is not yet implemented. "
            "This is a placeholder for future multi-scenario execution functionality."
        )

    def validate_scenario(self, scenario: ScenarioConfig) -> tuple[bool, str]:
        """
        Validate a scenario configuration.

        This method should:
        1. Check required fields are present
        2. Validate success criteria are measurable
        3. Check domain is supported
        4. Validate complexity level is appropriate

        Args:
            scenario: Scenario configuration to validate

        Returns:
            Tuple of (is_valid, error_message)

        Raises:
            NotImplementedError: This is a placeholder implementation

        TODO: Implement scenario validation logic
        TODO: Add domain-specific validation
        TODO: Add success criteria validation
        """
        raise NotImplementedError(
            "ScenarioOrchestrator.validate_scenario is not yet implemented. "
            "This is a placeholder for future scenario validation functionality."
        )

    def aggregate_results(
        self,
        conversations: List[GeneratedConversation],
    ) -> Dict[str, any]:
        """
        Aggregate results from multiple scenario executions.

        This method should:
        1. Calculate success rate across scenarios
        2. Calculate average turn count
        3. Identify common failure patterns
        4. Generate summary statistics

        Args:
            conversations: List of generated conversations

        Returns:
            Dictionary with aggregated statistics

        Raises:
            NotImplementedError: This is a placeholder implementation

        TODO: Implement result aggregation logic
        TODO: Add statistical analysis
        TODO: Add failure pattern detection
        """
        raise NotImplementedError(
            "ScenarioOrchestrator.aggregate_results is not yet implemented. "
            "This is a placeholder for future result aggregation functionality."
        )

`aggregate_results(conversations)`

Aggregate results from multiple scenario executions.

This method should: 1. Calculate success rate across scenarios 2. Calculate average turn count 3. Identify common failure patterns 4. Generate summary statistics

Parameters:

Name	Type	Description	Default
`conversations`	`List[GeneratedConversation]`	List of generated conversations	required

Returns:

Type	Description
`Dict[str, any]`	Dictionary with aggregated statistics

Raises:

Type	Description
`NotImplementedError`	This is a placeholder implementation

TODO: Implement result aggregation logic TODO: Add statistical analysis TODO: Add failure pattern detection

Source code in uaef/simulator/orchestrator.py

def aggregate_results(
    self,
    conversations: List[GeneratedConversation],
) -> Dict[str, any]:
    """
    Aggregate results from multiple scenario executions.

    This method should:
    1. Calculate success rate across scenarios
    2. Calculate average turn count
    3. Identify common failure patterns
    4. Generate summary statistics

    Args:
        conversations: List of generated conversations

    Returns:
        Dictionary with aggregated statistics

    Raises:
        NotImplementedError: This is a placeholder implementation

    TODO: Implement result aggregation logic
    TODO: Add statistical analysis
    TODO: Add failure pattern detection
    """
    raise NotImplementedError(
        "ScenarioOrchestrator.aggregate_results is not yet implemented. "
        "This is a placeholder for future result aggregation functionality."
    )

`execute_multiple_scenarios(scenarios, user_models, agent_config, parameters)`

Execute multiple test scenarios in parallel.

This method should: 1. Validate all scenario configurations 2. Create execution plan (parallel/sequential) 3. Execute scenarios with progress tracking 4. Aggregate results 5. Handle failures gracefully

Parameters:

Name	Type	Description	Default
`scenarios`	`List[ScenarioConfig]`	List of scenario configurations	required
`user_models`	`List[UserBehaviorModel]`	List of user behavior models (one per scenario)	required
`agent_config`	`AgentConfig`	Agent configuration (shared across scenarios)	required
`parameters`	`GenerationParameters`	Generation parameters	required

Returns:

Type	Description
`List[GeneratedConversation]`	List of GeneratedConversation objects

Raises:

Type	Description
`NotImplementedError`	This is a placeholder implementation

TODO: Implement parallel scenario execution TODO: Add progress tracking TODO: Add result aggregation TODO: Add failure handling

Source code in uaef/simulator/orchestrator.py

def execute_multiple_scenarios(
    self,
    scenarios: List[ScenarioConfig],
    user_models: List[UserBehaviorModel],
    agent_config: AgentConfig,
    parameters: GenerationParameters,
) -> List[GeneratedConversation]:
    """
    Execute multiple test scenarios in parallel.

    This method should:
    1. Validate all scenario configurations
    2. Create execution plan (parallel/sequential)
    3. Execute scenarios with progress tracking
    4. Aggregate results
    5. Handle failures gracefully

    Args:
        scenarios: List of scenario configurations
        user_models: List of user behavior models (one per scenario)
        agent_config: Agent configuration (shared across scenarios)
        parameters: Generation parameters

    Returns:
        List of GeneratedConversation objects

    Raises:
        NotImplementedError: This is a placeholder implementation

    TODO: Implement parallel scenario execution
    TODO: Add progress tracking
    TODO: Add result aggregation
    TODO: Add failure handling
    """
    raise NotImplementedError(
        "ScenarioOrchestrator.execute_multiple_scenarios is not yet implemented. "
        "This is a placeholder for future multi-scenario execution functionality."
    )

`execute_scenario(scenario, user_model, agent_config, parameters)`

Execute a single test scenario.

This method should: 1. Validate scenario configuration 2. Initialize conversation generator 3. Generate conversation using the scenario 4. Validate conversation quality 5. Return generated conversation

Parameters:

Name	Type	Description	Default
`scenario`	`ScenarioConfig`	Scenario configuration to execute	required
`user_model`	`UserBehaviorModel`	User behavior model for the scenario	required
`agent_config`	`AgentConfig`	Agent configuration for the scenario	required
`parameters`	`GenerationParameters`	Generation parameters	required

Returns:

Type	Description
`GeneratedConversation`	GeneratedConversation from scenario execution

Raises:

Type	Description
`NotImplementedError`	This is a placeholder implementation

TODO: Implement scenario execution logic TODO: Add scenario validation TODO: Add conversation quality checks TODO: Add error handling and retries

Source code in uaef/simulator/orchestrator.py

def execute_scenario(
    self,
    scenario: ScenarioConfig,
    user_model: UserBehaviorModel,
    agent_config: AgentConfig,
    parameters: GenerationParameters,
) -> GeneratedConversation:
    """
    Execute a single test scenario.

    This method should:
    1. Validate scenario configuration
    2. Initialize conversation generator
    3. Generate conversation using the scenario
    4. Validate conversation quality
    5. Return generated conversation

    Args:
        scenario: Scenario configuration to execute
        user_model: User behavior model for the scenario
        agent_config: Agent configuration for the scenario
        parameters: Generation parameters

    Returns:
        GeneratedConversation from scenario execution

    Raises:
        NotImplementedError: This is a placeholder implementation

    TODO: Implement scenario execution logic
    TODO: Add scenario validation
    TODO: Add conversation quality checks
    TODO: Add error handling and retries
    """
    raise NotImplementedError(
        "ScenarioOrchestrator.execute_scenario is not yet implemented. "
        "This is a placeholder for future scenario execution functionality. "
        "Full implementation will include scenario validation, conversation "
        "generation, and quality checks."
    )

`validate_scenario(scenario)`

Validate a scenario configuration.

This method should: 1. Check required fields are present 2. Validate success criteria are measurable 3. Check domain is supported 4. Validate complexity level is appropriate

Parameters:

Name	Type	Description	Default
`scenario`	`ScenarioConfig`	Scenario configuration to validate	required

Returns:

Type	Description
`tuple[bool, str]`	Tuple of (is_valid, error_message)

Raises:

Type	Description
`NotImplementedError`	This is a placeholder implementation

TODO: Implement scenario validation logic TODO: Add domain-specific validation TODO: Add success criteria validation

Source code in uaef/simulator/orchestrator.py

def validate_scenario(self, scenario: ScenarioConfig) -> tuple[bool, str]:
    """
    Validate a scenario configuration.

    This method should:
    1. Check required fields are present
    2. Validate success criteria are measurable
    3. Check domain is supported
    4. Validate complexity level is appropriate

    Args:
        scenario: Scenario configuration to validate

    Returns:
        Tuple of (is_valid, error_message)

    Raises:
        NotImplementedError: This is a placeholder implementation

    TODO: Implement scenario validation logic
    TODO: Add domain-specific validation
    TODO: Add success criteria validation
    """
    raise NotImplementedError(
        "ScenarioOrchestrator.validate_scenario is not yet implemented. "
        "This is a placeholder for future scenario validation functionality."
    )

Simulator

uaef.simulator.interfaces

AgentConfig

CommunicationStyle

ComplexityLevel

GeneratedConversation

GeneratedGroundTruth

GeneratedScenario

GenerationParameters

KnowledgeLevel

ScenarioConfig

TestPackage

validate_timestamp(v) classmethod

TurnGroundTruth

UserBehaviorModel

uaef.simulator.generator

ConversationGenerator

check_goal_achievement(conversation_history, success_criteria)

generate_conversation(scenario, user_model, agent_config, parameters)

generate_user_message(context, user_model, previous_agent_response=None)

simulate_agent_response(user_message, agent_config, conversation_history)

uaef.simulator.orchestrator

ScenarioOrchestrator

aggregate_results(conversations)

execute_multiple_scenarios(scenarios, user_models, agent_config, parameters)

execute_scenario(scenario, user_model, agent_config, parameters)

validate_scenario(scenario)

`uaef.simulator.interfaces`

`AgentConfig`

`CommunicationStyle`

`ComplexityLevel`

`GeneratedConversation`

`GeneratedGroundTruth`

`GeneratedScenario`

`GenerationParameters`

`KnowledgeLevel`

`ScenarioConfig`

`TestPackage`

`validate_timestamp(v)` `classmethod`

`TurnGroundTruth`

`UserBehaviorModel`

`uaef.simulator.generator`

`ConversationGenerator`

`check_goal_achievement(conversation_history, success_criteria)`

`generate_conversation(scenario, user_model, agent_config, parameters)`

`generate_user_message(context, user_model, previous_agent_response=None)`

`simulate_agent_response(user_message, agent_config, conversation_history)`

`uaef.simulator.orchestrator`

`ScenarioOrchestrator`

`aggregate_results(conversations)`

`execute_multiple_scenarios(scenarios, user_models, agent_config, parameters)`

`execute_scenario(scenario, user_model, agent_config, parameters)`

`validate_scenario(scenario)`