API Reference

stickler

stickler: Structured object comparison and evaluation library.

This library provides tools for comparing complex structured objects with configurable comparison strategies and detailed evaluation metrics.

stickler.structured_object_evaluator.models.structured_model.StructuredModel

Bases: BaseModel

Base class for models with structured comparison capabilities.

This class extends Pydantic's BaseModel with the ability to compare instances using configurable comparison metrics for each field.

Architecture - Delegation Pattern:

StructuredModel uses a delegation pattern where comparison logic is distributed across specialized helper classes. This refactoring reduced the class from 2584 lines to ~500 lines while maintaining all functionality.

The delegation pattern works as follows: 1. StructuredModel maintains the public API (compare, compare_with, compare_field) 2. All implementation details are delegated to specialized helper classes 3. Each helper class has a single, well-defined responsibility 4. Helpers receive the StructuredModel instance as a parameter (composition) 5. This avoids circular dependencies and keeps the architecture clean

Helper Classes and Their Responsibilities:

Model Creation: - ModelFactory: Creates dynamic StructuredModel subclasses from JSON configuration - Validates configuration structure - Converts field definitions to Pydantic fields - Creates model classes using Pydantic's create_model()

Comparison Orchestration: - ComparisonEngine: Main orchestrator for the comparison process - Coordinates between dispatcher, collectors, and calculators - Implements single-traversal optimization - Manages compare_recursive and compare_with methods

Field Comparison Routing: - ComparisonDispatcher: Routes field comparisons to appropriate handlers - Uses match-statement based dispatch for clarity - Handles null cases and type mismatches - Delegates to specialized comparators based on field type

Field-Level Comparison: - FieldComparator: Compares primitive and structured fields - Handles string, int, float comparisons - Handles nested StructuredModel comparisons - Applies threshold-based binary classification

• PrimitiveListComparator: Compares lists of primitive values
• Uses Hungarian matching for optimal pairing
• Returns hierarchical structure for API consistency

• Handles empty list cases

• StructuredListComparator: Compares lists of StructuredModels

• Uses Hungarian matching with object-level similarity
• Performs threshold-gated recursive analysis
• Calculates nested field metrics

Metrics Calculation: - ConfusionMatrixCalculator: Calculates confusion matrix metrics - Computes TP, FP, TN, FN, FD, FA counts - Handles list-level and field-level metrics - Calculates nested field metrics for structured lists

• AggregateMetricsCalculator: Rolls up child metrics to parent nodes
• Performs recursive traversal of result tree
• Sums child aggregate metrics to parent

• Provides universal field-level granularity

• DerivedMetricsCalculator: Calculates derived metrics

• Computes precision, recall, F1, accuracy
• Supports both traditional and FD-inclusive recall

• Delegates to MetricsHelper for calculations

• ConfusionMatrixBuilder: Orchestrates all metrics calculation

• Coordinates between the three calculator classes
• Ensures correct calculation order
• Builds complete confusion matrices

Non-Match Documentation: - NonMatchCollector: Documents non-matching fields - Collects object-level non-matches for lists - Collects field-level non-matches (legacy format) - Handles nested StructuredModel recursion

Existing Helpers (Pre-Refactoring): - HungarianHelper: Hungarian algorithm for list matching - MetricsHelper: Derived metrics calculation formulas - ConfigurationHelper: Field configuration management - ComparisonHelper: Comparison utility methods - EvaluatorFormatHelper: Output formatting for evaluators - NonMatchesHelper: Non-match collection utilities - FieldHelper: Field type and null checking utilities

Benefits of Delegation Pattern:

1. Maintainability: Each class has a single responsibility
2. Testability: Components can be tested in isolation
3. Extensibility: Easy to add new field types or metrics
4. Readability: Clear separation of concerns
5. Performance: No overhead - delegation is just function calls

Migration Notes:

• All public APIs remain unchanged (complete backward compatibility)
• All tests pass without modification (80+ test files)
• Performance characteristics maintained (single-traversal optimization)
• No breaking changes for existing users

Features:

• Field-level comparison configuration via ComparableField
• Nested model comparison with recursive evaluation
• Integration with ANLS* comparators
• JSON schema generation with comparison metadata
• Unordered list comparison using Hungarian matching
• Confusion matrix metrics (TP, FP, FN, TN, FA, FD)
• Aggregate metrics rollup from nested fields
• Retention of extra fields not defined in the model
• Dynamic model creation from JSON configuration
• Threshold-gated recursive analysis for performance

Example Usage:

from stickler.structured_object_evaluator.models import StructuredModel from stickler.structured_object_evaluator.models import ComparableField from stickler.comparators import LevenshteinComparator

class Product(StructuredModel): ... name: str = ComparableField( ... comparator=LevenshteinComparator(), ... threshold=0.8, ... weight=2.0 ... ) ... price: float = ComparableField( ... comparator=NumericComparator(), ... threshold=0.9 ... )

gt = Product(name="Widget", price=29.99) pred = Product(name="Widgit", price=29.99) # Typo in name

Simple comparison (returns overall similarity score)

score = gt.compare(pred) print(f"Similarity: {score:.2f}")

Detailed comparison with confusion matrix

result = gt.compare_with(pred, include_confusion_matrix=True) print(f"TP: {result['overall']['tp']}, FD: {result['overall']['fd']}") print(f"F1: {result['aggregate']['derived']['cm_f1']:.2f}")

Source code in stickler/structured_object_evaluator/models/structured_model.py

class StructuredModel(BaseModel):
    """Base class for models with structured comparison capabilities.

    This class extends Pydantic's BaseModel with the ability to compare
    instances using configurable comparison metrics for each field.

    Architecture - Delegation Pattern:
    ----------------------------------
    StructuredModel uses a delegation pattern where comparison logic is
    distributed across specialized helper classes. This refactoring reduced
    the class from 2584 lines to ~500 lines while maintaining all functionality.

    The delegation pattern works as follows:
    1. StructuredModel maintains the public API (compare, compare_with, compare_field)
    2. All implementation details are delegated to specialized helper classes
    3. Each helper class has a single, well-defined responsibility
    4. Helpers receive the StructuredModel instance as a parameter (composition)
    5. This avoids circular dependencies and keeps the architecture clean

    Helper Classes and Their Responsibilities:
    ------------------------------------------

    **Model Creation:**
    - ModelFactory: Creates dynamic StructuredModel subclasses from JSON configuration
      - Validates configuration structure
      - Converts field definitions to Pydantic fields
      - Creates model classes using Pydantic's create_model()

    **Comparison Orchestration:**
    - ComparisonEngine: Main orchestrator for the comparison process
      - Coordinates between dispatcher, collectors, and calculators
      - Implements single-traversal optimization
      - Manages compare_recursive and compare_with methods

    **Field Comparison Routing:**
    - ComparisonDispatcher: Routes field comparisons to appropriate handlers
      - Uses match-statement based dispatch for clarity
      - Handles null cases and type mismatches
      - Delegates to specialized comparators based on field type

    **Field-Level Comparison:**
    - FieldComparator: Compares primitive and structured fields
      - Handles string, int, float comparisons
      - Handles nested StructuredModel comparisons
      - Applies threshold-based binary classification

    - PrimitiveListComparator: Compares lists of primitive values
      - Uses Hungarian matching for optimal pairing
      - Returns hierarchical structure for API consistency
      - Handles empty list cases

    - StructuredListComparator: Compares lists of StructuredModels
      - Uses Hungarian matching with object-level similarity
      - Performs threshold-gated recursive analysis
      - Calculates nested field metrics

    **Metrics Calculation:**
    - ConfusionMatrixCalculator: Calculates confusion matrix metrics
      - Computes TP, FP, TN, FN, FD, FA counts
      - Handles list-level and field-level metrics
      - Calculates nested field metrics for structured lists

    - AggregateMetricsCalculator: Rolls up child metrics to parent nodes
      - Performs recursive traversal of result tree
      - Sums child aggregate metrics to parent
      - Provides universal field-level granularity

    - DerivedMetricsCalculator: Calculates derived metrics
      - Computes precision, recall, F1, accuracy
      - Supports both traditional and FD-inclusive recall
      - Delegates to MetricsHelper for calculations

    - ConfusionMatrixBuilder: Orchestrates all metrics calculation
      - Coordinates between the three calculator classes
      - Ensures correct calculation order
      - Builds complete confusion matrices

    **Non-Match Documentation:**
    - NonMatchCollector: Documents non-matching fields
      - Collects object-level non-matches for lists
      - Collects field-level non-matches (legacy format)
      - Handles nested StructuredModel recursion

    **Existing Helpers (Pre-Refactoring):**
    - HungarianHelper: Hungarian algorithm for list matching
    - MetricsHelper: Derived metrics calculation formulas
    - ConfigurationHelper: Field configuration management
    - ComparisonHelper: Comparison utility methods
    - EvaluatorFormatHelper: Output formatting for evaluators
    - NonMatchesHelper: Non-match collection utilities
    - FieldHelper: Field type and null checking utilities

    Benefits of Delegation Pattern:
    --------------------------------
    1. **Maintainability**: Each class has a single responsibility
    2. **Testability**: Components can be tested in isolation
    3. **Extensibility**: Easy to add new field types or metrics
    4. **Readability**: Clear separation of concerns
    5. **Performance**: No overhead - delegation is just function calls

    Migration Notes:
    ----------------
    - All public APIs remain unchanged (complete backward compatibility)
    - All tests pass without modification (80+ test files)
    - Performance characteristics maintained (single-traversal optimization)
    - No breaking changes for existing users

    Features:
    ---------
    - Field-level comparison configuration via ComparableField
    - Nested model comparison with recursive evaluation
    - Integration with ANLS* comparators
    - JSON schema generation with comparison metadata
    - Unordered list comparison using Hungarian matching
    - Confusion matrix metrics (TP, FP, FN, TN, FA, FD)
    - Aggregate metrics rollup from nested fields
    - Retention of extra fields not defined in the model
    - Dynamic model creation from JSON configuration
    - Threshold-gated recursive analysis for performance

    Example Usage:
    --------------
    >>> from stickler.structured_object_evaluator.models import StructuredModel
    >>> from stickler.structured_object_evaluator.models import ComparableField
    >>> from stickler.comparators import LevenshteinComparator
    >>> 
    >>> class Product(StructuredModel):
    ...     name: str = ComparableField(
    ...         comparator=LevenshteinComparator(),
    ...         threshold=0.8,
    ...         weight=2.0
    ...     )
    ...     price: float = ComparableField(
    ...         comparator=NumericComparator(),
    ...         threshold=0.9
    ...     )
    >>> 
    >>> gt = Product(name="Widget", price=29.99)
    >>> pred = Product(name="Widgit", price=29.99)  # Typo in name
    >>> 
    >>> # Simple comparison (returns overall similarity score)
    >>> score = gt.compare(pred)
    >>> print(f"Similarity: {score:.2f}")
    >>> 
    >>> # Detailed comparison with confusion matrix
    >>> result = gt.compare_with(pred, include_confusion_matrix=True)
    >>> print(f"TP: {result['overall']['tp']}, FD: {result['overall']['fd']}")
    >>> print(f"F1: {result['aggregate']['derived']['cm_f1']:.2f}")
    """

    # Default match threshold - can be overridden in subclasses
    match_threshold: ClassVar[float] = 0.7

    extra_fields: Dict[str, Any] = Field(default_factory=dict, exclude=True)

    model_config = {
        "arbitrary_types_allowed": True,
        "extra": "allow",  # Allow extra fields to be stored in extra_fields
    }

    def __init_subclass__(cls, **kwargs):
        """Validate field configurations when a StructuredModel subclass is defined."""
        super().__init_subclass__(**kwargs)

        # Validate field configurations using class annotations since model_fields isn't populated yet
        if hasattr(cls, "__annotations__"):
            for field_name, field_type in cls.__annotations__.items():
                if field_name == "extra_fields":
                    continue

                # Get the field default value if it exists
                field_default = getattr(cls, field_name, None)

                # Since ComparableField is now always a function that returns a Field,
                # we need to check if field_default has comparison metadata
                if hasattr(field_default, "json_schema_extra") and callable(
                    field_default.json_schema_extra
                ):
                    # Check for comparison metadata
                    temp_schema = {}
                    field_default.json_schema_extra(temp_schema)
                    if "x-comparison" in temp_schema:
                        # This field was created with ComparableField function - validate constraints
                        if cls._is_list_of_structured_model_type(field_type):
                            comparison_config = temp_schema["x-comparison"]

                            # Threshold validation - only flag if explicitly set to non-default value
                            threshold = comparison_config.get("threshold", 0.5)
                            if threshold != 0.5:  # Default threshold value
                                raise ValueError(
                                    f"Field '{field_name}' is a List[StructuredModel] and cannot have a "
                                    f"'threshold' parameter in ComparableField. Hungarian matching uses each "
                                    f"StructuredModel's 'match_threshold' class attribute instead. "
                                    f"Set 'match_threshold = {threshold}' on the list element class."
                                )

                            # Comparator validation - only flag if explicitly set to non-default type
                            comparator_type = comparison_config.get(
                                "comparator_type", "LevenshteinComparator"
                            )
                            if (
                                comparator_type != "LevenshteinComparator"
                            ):  # Default comparator type
                                raise ValueError(
                                    f"Field '{field_name}' is a List[StructuredModel] and cannot have a "
                                    f"'comparator' parameter in ComparableField. Object comparison uses each "
                                    f"StructuredModel's individual field comparators instead."
                                )

    @classmethod
    def _is_list_of_structured_model_type(cls, field_type) -> bool:
        """Check if a field type annotation represents List[StructuredModel].

        Args:
            field_type: The field type annotation

        Returns:
            True if the field is a List[StructuredModel] type
        """
        # Handle direct imports and typing constructs
        origin = get_origin(field_type)
        if origin is list or origin is List:
            args = get_args(field_type)
            if args:
                element_type = args[0]
                # Check if element type is a StructuredModel subclass
                try:
                    return inspect.isclass(element_type) and issubclass(
                        element_type, StructuredModel
                    )
                except (TypeError, AttributeError):
                    return False

        # Handle Union types (like Optional[List[StructuredModel]])
        elif origin is Union:
            args = get_args(field_type)
            for arg in args:
                if cls._is_list_of_structured_model_type(arg):
                    return True

        return False

    @classmethod
    def from_json(cls, json_data: Dict[str, Any]) -> "StructuredModel":
        """Create a StructuredModel instance from JSON data.

        This method handles missing fields gracefully and stores extra fields
        in the extra_fields attribute.

        Args:
            json_data: Dictionary containing the JSON data

        Returns:
            StructuredModel instance created from the JSON data
        """
        return ConfigurationHelper.from_json(cls, json_data)

    @classmethod
    def model_from_json(cls, config: Dict[str, Any]) -> Type["StructuredModel"]:
        """Create a StructuredModel subclass from JSON configuration using Pydantic's create_model().

        This method leverages Pydantic's native dynamic model creation capabilities to ensure
        full compatibility with all Pydantic features while adding structured comparison
        functionality through inherited StructuredModel methods.

        The generated model inherits all StructuredModel capabilities:
        - compare_with() method for detailed comparisons
        - Field-level comparison configuration
        - Hungarian algorithm for list matching
        - Confusion matrix generation
        - JSON schema with comparison metadata

        Args:
            config: JSON configuration with fields, comparators, and model settings.
                   Required keys:
                   - fields: Dict mapping field names to field configurations
                   Optional keys:
                   - model_name: Name for the generated class (default: "DynamicModel")
                   - match_threshold: Overall matching threshold (default: 0.7)

                   Field configuration format:
                   {
                       "type": "str|int|float|bool|List[str]|etc.",  # Required
                       "comparator": "LevenshteinComparator|ExactComparator|etc.",  # Optional
                       "threshold": 0.8,  # Optional, default 0.5
                       "weight": 2.0,     # Optional, default 1.0
                       "required": true,  # Optional, default false
                       "default": "value", # Optional
                       "description": "Field description",  # Optional
                       "alias": "field_alias",  # Optional
                       "examples": ["example1", "example2"]  # Optional
                   }

        Returns:
            A fully functional StructuredModel subclass created with create_model()

        Raises:
            ValueError: If configuration is invalid or contains unsupported types/comparators
            KeyError: If required configuration keys are missing

        Examples:
            >>> config = {
            ...     "model_name": "Product",
            ...     "match_threshold": 0.8,
            ...     "fields": {
            ...         "name": {
            ...             "type": "str",
            ...             "comparator": "LevenshteinComparator",
            ...             "threshold": 0.8,
            ...             "weight": 2.0,
            ...             "required": True
            ...         },
            ...         "price": {
            ...             "type": "float",
            ...             "comparator": "NumericComparator",
            ...             "default": 0.0
            ...         }
            ...     }
            ... }
            >>> ProductClass = StructuredModel.model_from_json(config)
            >>> isinstance(ProductClass.model_fields, dict)  # Full Pydantic compatibility
            True
            >>> product = ProductClass(name="Widget", price=29.99)
            >>> product.name
            'Widget'
            >>> result = product.compare_with(ProductClass(name="Widget", price=29.99))
            >>> result["overall_score"]
            1.0
        """
        # Delegate to ModelFactory for dynamic model creation
        from .model_factory import ModelFactory

        return ModelFactory.create_model_from_json(config, base_class=cls)

    @classmethod
    def from_json_schema(cls, schema: Dict[str, Any]) -> Type["StructuredModel"]:
        """Create a StructuredModel subclass from a JSON Schema document.

        This method accepts standard JSON Schema documents and creates fully functional
        StructuredModel classes with comparison capabilities. Supports JSON Schema draft-07+.

        Comparison behavior can be customized using x-aws-stickler-* extension fields:

        Field-Level Extensions:
        -----------------------
        - x-aws-stickler-comparator: Comparator algorithm name (built-in or registered custom)
        - x-aws-stickler-threshold: Similarity threshold for match/no-match (0.0-1.0, default: 0.5)
        - x-aws-stickler-weight: Field importance in overall scoring (>0.0, default: 1.0)
        - x-aws-stickler-clip-under-threshold: Clip scores below threshold to 0.0 (bool, default: false)
        - x-aws-stickler-aggregate: Include field metrics in parent aggregation (bool, default: false)

        Model-Level Extensions:
        -----------------------
        - x-aws-stickler-model-name: Generated class name (default: "DynamicModel")
        - x-aws-stickler-match-threshold: Overall match threshold (default: 0.7)

        Supported Features:
        -------------------
        - Primitive types: string, number, integer, boolean
        - Nested objects and arrays (primitive/object items)
        - Required fields, defaults, descriptions
        - Schema references ($ref with #/definitions/ and #/$defs/)

        Default Type Mappings:
        ----------------------
        - string → LevenshteinComparator (threshold: 0.5)
        - number/integer → NumericComparator (threshold: 0.5)
        - boolean → ExactComparator (threshold: 1.0)
        - arrays → Hungarian matching with element-appropriate comparators
        - objects → Recursive field-by-field comparison

        Args:
            schema: JSON Schema document as a dictionary

        Returns:
            StructuredModel subclass created from the schema

        Raises:
            ValueError: If schema is invalid or contains unsupported features
            jsonschema.exceptions.SchemaError: If schema doesn't conform to JSON Schema spec

        Examples:
            Basic usage with standard JSON Schema:
            >>> schema = {
            ...     "type": "object",
            ...     "properties": {
            ...         "name": {"type": "string"},
            ...         "age": {"type": "integer"},
            ...         "email": {"type": "string"}
            ...     },
            ...     "required": ["name", "email"]
            ... }
            >>> PersonModel = StructuredModel.from_json_schema(schema)
            >>> person1 = PersonModel(name="Alice", age=30, email="alice@example.com")
            >>> person2 = PersonModel(name="Alicia", age=30, email="alice@example.com")
            >>> result = person1.compare_with(person2)
            >>> # name field uses LevenshteinComparator, age uses NumericComparator

            Advanced usage with x-aws-stickler-* extensions:
            >>> schema = {
            ...     "type": "object",
            ...     "x-aws-stickler-model-name": "Product",
            ...     "x-aws-stickler-match-threshold": 0.8,
            ...     "properties": {
            ...         "name": {
            ...             "type": "string",
            ...             "x-aws-stickler-comparator": "LevenshteinComparator",
            ...             "x-aws-stickler-threshold": 0.9,
            ...             "x-aws-stickler-weight": 2.0,
            ...             "x-aws-stickler-aggregate": true
            ...         },
            ...         "price": {
            ...             "type": "number",
            ...             "x-aws-stickler-comparator": "NumericComparator",
            ...             "x-aws-stickler-threshold": 0.95,
            ...             "x-aws-stickler-clip-under-threshold": true
            ...         }
            ...     },
            ...     "required": ["name"]
            ... }
            >>> ProductModel = StructuredModel.from_json_schema(schema)
            >>> result = product1.compare_with(product2)
            >>> # name field has weight=2.0, price field clips scores below 0.95
            """

        return cls._from_json_schema_internal(schema, field_path="")

    @classmethod
    def _from_json_schema_internal(
        cls, schema: Dict[str, Any], field_path: str
    ) -> Type["StructuredModel"]:
        """Internal method for creating StructuredModel from JSON Schema with field path tracking.

        This is used internally for recursive calls to track field paths for error messages.
        External callers should use from_json_schema() instead.

        Args:
            schema: JSON Schema document as a dictionary
            field_path: Current field path for error messages (e.g., "address.street")

        Returns:
            StructuredModel subclass created from the schema
        """
        # Import dependencies
        from ..utils.json_schema_validator import validate_json_schema
        from .json_schema_field_converter import JsonSchemaFieldConverter
        from .model_factory import ModelFactory

        # Subtask 4.2: Validate JSON Schema
        try:
            validate_json_schema(schema)
        except Exception as e:
            raise ValueError(
                f"Invalid JSON Schema: {e}. "
                f"Please ensure the schema conforms to JSON Schema draft-07 specification."
            )

        # Subtask 4.3: Extract model-level configuration
        model_name = schema.get("x-aws-stickler-model-name", "DynamicModel")
        match_threshold = schema.get("x-aws-stickler-match-threshold", 0.7)

        # Validate model name
        if not isinstance(model_name, str) or not model_name.isidentifier():
            raise ValueError(
                f"x-aws-stickler-model-name must be a valid Python identifier, "
                f"got: {model_name}"
            )

        # Validate match threshold
        if not isinstance(match_threshold, (int, float)):
            raise ValueError(
                f"x-aws-stickler-match-threshold must be a number, "
                f"got: {type(match_threshold).__name__}"
            )

        if not (0.0 <= match_threshold <= 1.0):
            raise ValueError(
                f"x-aws-stickler-match-threshold must be between 0.0 and 1.0, "
                f"got: {match_threshold}"
            )

        # Subtask 4.4: Convert fields and create model
        # Ensure schema has properties
        if "properties" not in schema:
            raise ValueError(
                "JSON Schema must contain 'properties' key for object type"
            )

        properties = schema.get("properties", {})
        required = schema.get("required", [])

        # Create converter and convert properties to field definitions
        converter = JsonSchemaFieldConverter(schema, field_path=field_path)
        field_definitions = converter.convert_properties_to_fields(properties, required)

        # Create the model using ModelFactory
        return ModelFactory.create_model_from_fields(
            model_name=model_name,
            field_definitions=field_definitions,
            match_threshold=match_threshold,
            base_class=cls
        )

    @classmethod
    def _is_structured_field_type(cls, field_info) -> bool:
        """Check if a field represents a structured type that needs special handling.

        Args:
            field_info: Pydantic field info object

        Returns:
            True if the field is a List[StructuredModel] or StructuredModel type
        """
        return ConfigurationHelper.is_structured_field_type(field_info)

    @classmethod
    def _get_comparison_info(cls, field_name: str) -> ComparableField:
        """Extract comparison info from a field.

        Args:
            field_name: Name of the field to get comparison info for

        Returns:
            ComparableField object with comparison configuration
        """
        return ConfigurationHelper.get_comparison_info(cls, field_name)



    @classmethod
    def _is_aggregate_field(cls, field_name: str) -> bool:
        """Check if field is marked for confusion matrix aggregation.

        Args:
            field_name: Name of the field to check

        Returns:
            True if the field is marked for aggregation, False otherwise
        """
        return ConfigurationHelper.is_aggregate_field(cls, field_name)

    def _is_truly_null(self, val: Any) -> bool:
        """Check if a value is truly null (None).

        DEPRECATED: Delegates to NullHelper for consistency.
        Kept for backward compatibility with any external callers.
        """
        from .null_helper import NullHelper
        return NullHelper.is_truly_null(val)

    def _should_use_hierarchical_structure(self, val: Any, field_name: str) -> bool:
        """Check if a list value should maintain hierarchical structure.

        For lists, we need to check if they should maintain hierarchical structure
        based on their field type configuration.

        Args:
            val: Value to check (typically a list)
            field_name: Name of the field being evaluated

        Returns:
            True if the value should use hierarchical structure, False otherwise
        """
        if isinstance(val, list):
            # Check if this field is configured as List[StructuredModel]
            field_info = self.__class__.model_fields.get(field_name)
            if field_info and self._is_structured_field_type(field_info):
                return True
        return False

    def _is_effectively_null_for_lists(self, val: Any) -> bool:
        """Check if a list value is effectively null (None or empty list).

        DEPRECATED: Delegates to NullHelper for consistency.
        Kept for backward compatibility with any external callers.
        """
        from .null_helper import NullHelper
        return NullHelper.is_effectively_null_for_lists(val)

    def _is_effectively_null_for_primitives(self, val: Any) -> bool:
        """Check if a primitive value is effectively null.

        DEPRECATED: Delegates to NullHelper for consistency.
        Kept for backward compatibility with any external callers.
        """
        from .null_helper import NullHelper
        return NullHelper.is_effectively_null_for_primitives(val)

    def _is_list_field(self, field_name: str) -> bool:
        """Check if a field is ANY list type.

        Args:
            field_name: Name of the field to check

        Returns:
            True if the field is a list type (List[str], List[StructuredModel], etc.)
        """
        field_info = self.__class__.model_fields.get(field_name)
        if not field_info:
            return False

        field_type = field_info.annotation
        # Handle Optional types and direct List types
        if hasattr(field_type, "__origin__"):
            origin = field_type.__origin__
            if origin is list or origin is List:
                return True
            elif origin is Union:  # Optional[List[...]] case
                args = field_type.__args__
                for arg in args:
                    if hasattr(arg, "__origin__") and (
                        arg.__origin__ is list or arg.__origin__ is List
                    ):
                        return True
        return False

    def _handle_list_field_dispatch(
        self, gt_val: Any, pred_val: Any, weight: float
    ) -> dict:
        """Handle list field comparison using match statements.

        DEPRECATED: This method now delegates to ComparisonDispatcher.
        Kept for backward compatibility with any external callers.

        Args:
            gt_val: Ground truth list value
            pred_val: Predicted list value
            weight: Field weight for scoring

        Returns:
            Comparison result dictionary
        """
        from .comparison_dispatcher import ComparisonDispatcher
        dispatcher = ComparisonDispatcher(self)
        return dispatcher.handle_list_field_dispatch(gt_val, pred_val, weight)

    def _create_true_negative_result(self, weight: float) -> dict:
        """Create a true negative result.

        DEPRECATED: Delegates to ResultHelper for consistency.
        Kept for backward compatibility with any external callers.
        """
        from .result_helper import ResultHelper
        return ResultHelper.create_true_negative_result(weight)

    def _create_false_alarm_result(self, weight: float) -> dict:
        """Create a false alarm result.

        DEPRECATED: Delegates to ResultHelper for consistency.
        Kept for backward compatibility with any external callers.
        """
        from .result_helper import ResultHelper
        return ResultHelper.create_false_alarm_result(weight)

    def _create_false_negative_result(self, weight: float) -> dict:
        """Create a false negative result.

        DEPRECATED: Delegates to ResultHelper for consistency.
        Kept for backward compatibility with any external callers.
        """
        from .result_helper import ResultHelper
        return ResultHelper.create_false_negative_result(weight)

    def _handle_struct_list_empty_cases(
        self,
        gt_list: List["StructuredModel"],
        pred_list: List["StructuredModel"],
        weight: float,
    ) -> dict:
        """Handle empty list cases with beautiful match statements.

        DEPRECATED: Delegates to ResultHelper for consistency.
        Kept for backward compatibility with any external callers.

        Args:
            gt_list: Ground truth list (may be None)
            pred_list: Predicted list (may be None)
            weight: Field weight for scoring

        Returns:
            Result dictionary if early exit needed, None if should continue processing
        """
        from .result_helper import ResultHelper

        # Normalize None to empty lists for consistent handling
        gt_len = len(gt_list or [])
        pred_len = len(pred_list or [])

        return ResultHelper.create_empty_list_result(gt_len, pred_len, weight)

    def _calculate_object_level_metrics(
        self,
        gt_list: List["StructuredModel"],
        pred_list: List["StructuredModel"],
        match_threshold: float,
    ) -> tuple:
        """Calculate object-level metrics using Hungarian matching.

        Args:
            gt_list: Ground truth list
            pred_list: Predicted list
            match_threshold: Threshold for considering objects as matches

        Returns:
            Tuple of (object_metrics_dict, matched_pairs, matched_gt_indices, matched_pred_indices)
        """
        # Use Hungarian matching for OBJECT-LEVEL counts - OPTIMIZED: Single call gets all info
        hungarian_helper = HungarianHelper()
        hungarian_info = hungarian_helper.get_complete_matching_info(gt_list, pred_list)
        matched_pairs = hungarian_info["matched_pairs"]

        # Count OBJECTS, not individual fields
        tp_objects = 0  # Objects with similarity >= match_threshold
        fd_objects = 0  # Objects with similarity < match_threshold
        for gt_idx, pred_idx, similarity in matched_pairs:
            if similarity >= match_threshold:
                tp_objects += 1
            else:
                fd_objects += 1

        # Count unmatched objects
        matched_gt_indices = {idx for idx, _, _ in matched_pairs}
        matched_pred_indices = {idx for _, idx, _ in matched_pairs}
        fn_objects = len(gt_list) - len(matched_gt_indices)  # Unmatched GT objects
        fa_objects = len(pred_list) - len(
            matched_pred_indices
        )  # Unmatched pred objects

        # Build list-level metrics counting OBJECTS (not fields)
        object_level_metrics = {
            "tp": tp_objects,
            "fa": fa_objects,
            "fd": fd_objects,
            "fp": fa_objects + fd_objects,  # Total false positives
            "tn": 0,  # No true negatives at object level for non-empty lists
            "fn": fn_objects,
        }

        return (
            object_level_metrics,
            matched_pairs,
            matched_gt_indices,
            matched_pred_indices,
        )

    def _calculate_struct_list_similarity(
        self,
        gt_list: List["StructuredModel"],
        pred_list: List["StructuredModel"],
        info: "ComparableField",
    ) -> float:
        """Calculate raw similarity score for structured list.

        Args:
            gt_list: Ground truth list
            pred_list: Predicted list
            info: Field comparison info

        Returns:
            Raw similarity score between 0.0 and 1.0
        """
        if len(pred_list) > 0:
            match_result = self._compare_unordered_lists(
                gt_list, pred_list, info.comparator, info.threshold
            )
            return match_result.get("overall_score", 0.0)
        else:
            return 0.0

    def _compare_unordered_lists(
        self,
        list1: List[Any],
        list2: List[Any],
        comparator: BaseComparator,
        threshold: float,
    ) -> Dict[str, Any]:
        """Compare two lists as unordered collections using Hungarian matching.

        Args:
            list1: First list
            list2: Second list
            comparator: Comparator to use for item comparison
            threshold: Minimum score to consider a match

        Returns:
            Dictionary with confusion matrix metrics including:
            - tp: True positives (matches >= threshold)
            - fd: False discoveries (matches < threshold)
            - fa: False alarms (unmatched prediction items)
            - fn: False negatives (unmatched ground truth items)
            - fp: Total false positives (fd + fa)
            - overall_score: Similarity score for backward compatibility
        """
        return ComparisonHelper.compare_unordered_lists(
            list1, list2, comparator, threshold
        )

    def compare_field(self, field_name: str, other_value: Any) -> float:
        """Compare a single field with a value using the configured comparator.

        Args:
            field_name: Name of the field to compare
            other_value: Value to compare with

        Returns:
            Similarity score between 0.0 and 1.0
        """
        # Get our field value
        my_value = getattr(self, field_name)

        # If both values are StructuredModel instances, use recursive compare_with
        if isinstance(my_value, StructuredModel) and isinstance(
            other_value, StructuredModel
        ):
            # Use compare_with for rich comparison
            comparison_result = my_value.compare_with(
                other_value,
                include_confusion_matrix=False,
                document_non_matches=False,
                evaluator_format=False,
                recall_with_fd=False,
            )
            # Apply field-level threshold if configured
            info = self._get_comparison_info(field_name)
            raw_score = comparison_result["overall_score"]
            return (
                raw_score
                if raw_score >= info.threshold or not info.clip_under_threshold
                else 0.0
            )

        # CRITICAL FIX: For lists, don't clip under threshold for partial matches
        if isinstance(my_value, list) and isinstance(other_value, list):
            # Get field info
            info = self._get_comparison_info(field_name)

            # Use the raw comparison result without threshold clipping for lists
            result = ComparisonHelper.compare_unordered_lists(
                my_value, other_value, info.comparator, info.threshold
            )

            # Return the overall score directly (don't clip based on threshold for lists)
            return result["overall_score"]

        # For other fields, use existing logic
        return ComparisonHelper.compare_field_with_threshold(
            self, field_name, other_value
        )

    def compare_field_raw(self, field_name: str, other_value: Any) -> float:
        """Compare a single field with a value WITHOUT applying thresholds.

        This version is used by the compare method to get raw similarity scores.

        Args:
            field_name: Name of the field to compare
            other_value: Value to compare with

        Returns:
            Raw similarity score between 0.0 and 1.0 without threshold filtering
        """
        # Get our field value
        my_value = getattr(self, field_name)

        # If both values are StructuredModel instances, use recursive compare_with
        if isinstance(my_value, StructuredModel) and isinstance(
            other_value, StructuredModel
        ):
            # Use compare_with for rich comparison, but extract the raw score
            comparison_result = my_value.compare_with(
                other_value,
                include_confusion_matrix=False,
                document_non_matches=False,
                evaluator_format=False,
                recall_with_fd=False,
            )
            return comparison_result["overall_score"]

        # For non-StructuredModel fields, use existing logic
        return ComparisonHelper.compare_field_raw(self, field_name, other_value)

    def compare_recursive(self, other: "StructuredModel") -> dict:
        """The ONE clean recursive function that handles everything.

        Enhanced to capture BOTH confusion matrix metrics AND similarity scores
        in a single traversal to eliminate double traversal inefficiency.

        PHASE 2: Delegates to ComparisonEngine while maintaining identical behavior.

        Args:
            other: Another instance of the same model to compare with

        Returns:
            Dictionary with clean hierarchical structure:
            - overall: TP, FP, TN, FN, FD, FA counts + similarity_score + all_fields_matched
            - fields: Recursive structure for each field with scores
            - non_matches: List of non-matching items
        """
        from .comparison_engine import ComparisonEngine
        engine = ComparisonEngine(self)
        return engine.compare_recursive(other)

    def _dispatch_field_comparison(
        self, field_name: str, gt_val: Any, pred_val: Any
    ) -> dict:
        """Enhanced case-based dispatch using match statements for clean logic flow.

        DEPRECATED: This method now delegates to ComparisonDispatcher.
        Kept for backward compatibility with any external callers.
        """
        from .comparison_dispatcher import ComparisonDispatcher
        dispatcher = ComparisonDispatcher(self)
        return dispatcher.dispatch_field_comparison(field_name, gt_val, pred_val)







    def _calculate_aggregate_metrics(self, result: dict) -> dict:
        """Calculate aggregate metrics for all nodes in the result tree.

        This method delegates to AggregateMetricsCalculator for the actual implementation.

        CRITICAL FIX: Enhanced deep nesting traversal to handle arbitrary nesting depth.
        The aggregate field contains the sum of all primitive field confusion matrices
        below that node in the tree. This provides universal field-level granularity.

        Args:
            result: Result from compare_recursive with hierarchical structure

        Returns:
            Modified result with 'aggregate' fields added at each level
        """
        from .aggregate_metrics_calculator import AggregateMetricsCalculator
        calculator = AggregateMetricsCalculator()
        return calculator.calculate_aggregate_metrics(result)

    def _add_derived_metrics_to_result(self, result: dict, recall_with_fd: bool = False) -> dict:
        """Walk through result and add 'derived' fields with F1, precision, recall, accuracy.

        This method delegates to DerivedMetricsCalculator for the actual implementation.

        Args:
            result: Result from compare_recursive with basic TP, FP, FN, etc. metrics
            recall_with_fd: If True, include FD in recall denominator (TP/(TP+FN+FD))
                           If False, use traditional recall (TP/(TP+FN))

        Returns:
            Modified result with 'derived' fields added at each level
        """
        from .derived_metrics_calculator import DerivedMetricsCalculator
        calculator = DerivedMetricsCalculator()
        return calculator.add_derived_metrics_to_result(result, recall_with_fd)

    def _has_basic_metrics(self, metrics_dict: dict) -> bool:
        """Check if a dictionary has basic confusion matrix metrics.

        Args:
            metrics_dict: Dictionary to check

        Returns:
            True if it has the basic metrics (tp, fp, fn, etc.)
        """
        basic_metrics = ["tp", "fp", "fn", "tn", "fa", "fd"]
        return all(metric in metrics_dict for metric in basic_metrics)

    def _classify_field_for_confusion_matrix(
        self, field_name: str, other_value: Any, threshold: float = None
    ) -> Dict[str, Any]:
        """Classify a field comparison according to the confusion matrix rules.

        This method delegates to ConfusionMatrixCalculator for the actual implementation.

        Args:
            field_name: Name of the field being compared
            other_value: Value to compare with
            threshold: Threshold for matching (uses field's threshold if None)

        Returns:
            Dictionary with TP, FP, TN, FN, FD counts and derived metrics
        """
        from .confusion_matrix_calculator import ConfusionMatrixCalculator
        calculator = ConfusionMatrixCalculator(self)
        return calculator.classify_field_for_confusion_matrix(field_name, other_value, threshold)

    def _calculate_list_confusion_matrix(
        self, field_name: str, other_list: List[Any]
    ) -> Dict[str, Any]:
        """Calculate confusion matrix for a list field, including nested field metrics.

        This method delegates to ConfusionMatrixCalculator for the actual implementation.

        Args:
            field_name: Name of the list field being compared
            other_list: Predicted list to compare with

        Returns:
            Dictionary with:
            - Top-level TP, FP, TN, FN, FD, FA counts and derived metrics for the list field
            - nested_fields: Dict with metrics for individual fields within list items (e.g., "transactions.date")
            - non_matches: List of individual object-level non-matches for detailed analysis
        """
        from .confusion_matrix_calculator import ConfusionMatrixCalculator
        calculator = ConfusionMatrixCalculator(self)
        return calculator.calculate_list_confusion_matrix(field_name, other_list)

    def _calculate_nested_field_metrics(
        self,
        list_field_name: str,
        gt_list: List["StructuredModel"],
        pred_list: List["StructuredModel"],
        threshold: float,
    ) -> Dict[str, Dict[str, Any]]:
        """Calculate confusion matrix metrics for individual fields within list items.

        This method delegates to ConfusionMatrixCalculator for the actual implementation.

        THRESHOLD-GATED RECURSION: Only perform recursive field analysis for object pairs
        with similarity >= StructuredModel.match_threshold. Poor matches and unmatched
        items are treated as atomic units.

        Args:
            list_field_name: Name of the parent list field (e.g., "transactions")
            gt_list: Ground truth list of StructuredModel objects
            pred_list: Predicted list of StructuredModel objects
            threshold: Matching threshold (not used for threshold-gating)

        Returns:
            Dictionary mapping nested field paths to their confusion matrix metrics
            E.g., {"transactions.date": {...}, "transactions.description": {...}}
        """
        from .confusion_matrix_calculator import ConfusionMatrixCalculator
        calculator = ConfusionMatrixCalculator(self)
        return calculator.calculate_nested_field_metrics(list_field_name, gt_list, pred_list, threshold)

    def _calculate_single_nested_field_metrics(
        self,
        parent_field_name: str,
        gt_nested: "StructuredModel",
        pred_nested: "StructuredModel",
        parent_is_aggregate: bool = False,
    ) -> Dict[str, Dict[str, Any]]:
        """Calculate confusion matrix metrics for fields within a single nested StructuredModel.

        This method delegates to ConfusionMatrixCalculator for the actual implementation.

        Args:
            parent_field_name: Name of the parent field (e.g., "address")
            gt_nested: Ground truth nested StructuredModel
            pred_nested: Predicted nested StructuredModel
            parent_is_aggregate: Whether the parent field should aggregate child metrics

        Returns:
            Dictionary mapping nested field paths to their confusion matrix metrics
            E.g., {"address.street": {...}, "address.city": {...}}
        """
        from .confusion_matrix_calculator import ConfusionMatrixCalculator
        calculator = ConfusionMatrixCalculator(self)
        return calculator.calculate_single_nested_field_metrics(
            parent_field_name, gt_nested, pred_nested, parent_is_aggregate
        )

    def _collect_enhanced_non_matches(
        self, recursive_result: dict, other: "StructuredModel"
    ) -> List[Dict[str, Any]]:
        """Collect enhanced non-matches with object-level granularity.

        This method delegates to NonMatchCollector for the actual implementation.

        Args:
            recursive_result: Result from compare_recursive containing field comparison details
            other: The predicted StructuredModel instance

        Returns:
            List of non-match dictionaries with enhanced object-level information
        """
        from .non_match_collector import NonMatchCollector
        collector = NonMatchCollector(self)
        return collector.collect_enhanced_non_matches(recursive_result, other)

    def _collect_non_matches(
        self, other: "StructuredModel", base_path: str = ""
    ) -> List[NonMatchField]:
        """Collect non-matches for detailed analysis.

        This method delegates to NonMatchCollector for the actual implementation.

        Args:
            other: Other model to compare with
            base_path: Base path for field naming (e.g., "address")

        Returns:
            List of NonMatchField objects documenting non-matches
        """
        from .non_match_collector import NonMatchCollector
        collector = NonMatchCollector(self)
        return collector.collect_non_matches(other, base_path)

    def compare(self, other: "StructuredModel") -> float:
        """Compare this model with another and return a scalar similarity score.

        Returns the overall weighted average score regardless of sufficient/necessary field matching.
        This provides a more nuanced score for use in comparators.

        Args:
            other: Another instance of the same model to compare with

        Returns:
            Similarity score between 0.0 and 1.0
        """
        # We'll calculate the overall weighted score directly instead of using compare_with
        # This ensures that sufficient/necessary field rules don't cause a zero score
        # when at least some fields match

        total_score = 0.0
        total_weight = 0.0

        for field_name in self.__class__.model_fields:
            # Skip the extra_fields attribute in comparison
            if field_name == "extra_fields":
                continue
            if hasattr(other, field_name):
                # Get field configuration
                info = self.__class__._get_comparison_info(field_name)
                # Use weight from ComparableField object
                weight = info.weight

                # Compare field values WITHOUT applying thresholds
                field_score = self.compare_field_raw(
                    field_name, getattr(other, field_name)
                )

                # Update total score
                total_score += field_score * weight
                total_weight += weight

        # Calculate overall score
        if total_weight > 0:
            return total_score / total_weight
        else:
            return 0.0

    def compare_with(
        self,
        other: "StructuredModel",
        include_confusion_matrix: bool = False,
        document_non_matches: bool = False,
        evaluator_format: bool = False,
        recall_with_fd: bool = False,
        add_derived_metrics: bool = True,
    ) -> Dict[str, Any]:
        """Compare this model with another instance using SINGLE TRAVERSAL optimization.

        PHASE 2: Delegates to ComparisonEngine while maintaining identical behavior.

        Args:
            other: Another instance of the same model to compare with
            include_confusion_matrix: Whether to include confusion matrix calculations
            document_non_matches: Whether to document non-matches for analysis
            evaluator_format: Whether to format results for the evaluator
            recall_with_fd: If True, include FD in recall denominator (TP/(TP+FN+FD))
                            If False, use traditional recall (TP/(TP+FN))
            add_derived_metrics: Whether to add derived metrics to confusion matrix

        Returns:
            Dictionary with comparison results including:
            - field_scores: Scores for each field
            - overall_score: Weighted average score
            - all_fields_matched: Whether all fields matched
            - confusion_matrix: (optional) Confusion matrix data if requested
            - non_matches: (optional) Non-match documentation if requested
        """
        from .comparison_engine import ComparisonEngine
        engine = ComparisonEngine(self)
        return engine.compare_with(
            other,
            include_confusion_matrix=include_confusion_matrix,
            document_non_matches=document_non_matches,
            evaluator_format=evaluator_format,
            recall_with_fd=recall_with_fd,
            add_derived_metrics=add_derived_metrics,
        )

    def _convert_score_to_binary_metrics(
        self, score: float, threshold: float = 0.5
    ) -> Dict[str, float]:
        """Convert similarity score to binary classification metrics using MetricsHelper.

        Args:
            score: Similarity score [0-1]
            threshold: Threshold for considering a match

        Returns:
            Dictionary with TP, FP, FN, TN counts converted to metrics
        """
        metrics_helper = MetricsHelper()
        return metrics_helper.convert_score_to_binary_metrics(score, threshold)

    def _format_for_evaluator(
        self,
        result: Dict[str, Any],
        other: "StructuredModel",
        recall_with_fd: bool = False,
    ) -> Dict[str, Any]:
        """Format comparison results for evaluator compatibility.

        Args:
            result: Standard comparison result from compare_with
            other: The other model being compared
            recall_with_fd: Whether to include FD in recall denominator

        Returns:
            Dictionary in evaluator format with overall, fields, confusion_matrix
        """
        return EvaluatorFormatHelper.format_for_evaluator(
            self, result, other, recall_with_fd
        )

    def _calculate_list_item_metrics(
        self,
        field_name: str,
        gt_list: List[Any],
        pred_list: List[Any],
        recall_with_fd: bool = False,
    ) -> List[Dict[str, Any]]:
        """Calculate metrics for individual items in a list field.

        Args:
            field_name: Name of the list field
            gt_list: Ground truth list
            pred_list: Prediction list
            recall_with_fd: Whether to include FD in recall denominator

        Returns:
            List of metrics dictionaries for each matched item pair
        """
        return EvaluatorFormatHelper.calculate_list_item_metrics(
            field_name, gt_list, pred_list, recall_with_fd
        )

    @classmethod
    def model_json_schema(cls, **kwargs):
        """Override to add model-level comparison metadata.

        Extends the standard Pydantic JSON schema with comparison metadata
        at the field level.

        Args:
            **kwargs: Arguments to pass to the parent method

        Returns:
            JSON schema with added comparison metadata
        """
        schema = super().model_json_schema(**kwargs)

        # Add comparison metadata to each field in the schema
        for field_name, field_info in cls.model_fields.items():
            if field_name == "extra_fields":
                continue

            # Get the schema property for this field
            if field_name not in schema.get("properties", {}):
                continue

            field_props = schema["properties"][field_name]

            # Since ComparableField is now always a function, check for json_schema_extra
            if hasattr(field_info, "json_schema_extra") and callable(
                field_info.json_schema_extra
            ):
                # Fallback: Check for json_schema_extra function
                temp_schema = {}
                field_info.json_schema_extra(temp_schema)

                if "x-comparison" in temp_schema:
                    # Copy the comparison metadata from the temp schema to the real schema
                    field_props["x-comparison"] = temp_schema["x-comparison"]

        return schema

__init_subclass__(**kwargs)

Validate field configurations when a StructuredModel subclass is defined.

Source code in stickler/structured_object_evaluator/models/structured_model.py

def __init_subclass__(cls, **kwargs):
    """Validate field configurations when a StructuredModel subclass is defined."""
    super().__init_subclass__(**kwargs)

    # Validate field configurations using class annotations since model_fields isn't populated yet
    if hasattr(cls, "__annotations__"):
        for field_name, field_type in cls.__annotations__.items():
            if field_name == "extra_fields":
                continue

            # Get the field default value if it exists
            field_default = getattr(cls, field_name, None)

            # Since ComparableField is now always a function that returns a Field,
            # we need to check if field_default has comparison metadata
            if hasattr(field_default, "json_schema_extra") and callable(
                field_default.json_schema_extra
            ):
                # Check for comparison metadata
                temp_schema = {}
                field_default.json_schema_extra(temp_schema)
                if "x-comparison" in temp_schema:
                    # This field was created with ComparableField function - validate constraints
                    if cls._is_list_of_structured_model_type(field_type):
                        comparison_config = temp_schema["x-comparison"]

                        # Threshold validation - only flag if explicitly set to non-default value
                        threshold = comparison_config.get("threshold", 0.5)
                        if threshold != 0.5:  # Default threshold value
                            raise ValueError(
                                f"Field '{field_name}' is a List[StructuredModel] and cannot have a "
                                f"'threshold' parameter in ComparableField. Hungarian matching uses each "
                                f"StructuredModel's 'match_threshold' class attribute instead. "
                                f"Set 'match_threshold = {threshold}' on the list element class."
                            )

                        # Comparator validation - only flag if explicitly set to non-default type
                        comparator_type = comparison_config.get(
                            "comparator_type", "LevenshteinComparator"
                        )
                        if (
                            comparator_type != "LevenshteinComparator"
                        ):  # Default comparator type
                            raise ValueError(
                                f"Field '{field_name}' is a List[StructuredModel] and cannot have a "
                                f"'comparator' parameter in ComparableField. Object comparison uses each "
                                f"StructuredModel's individual field comparators instead."
                            )

compare(other)

Compare this model with another and return a scalar similarity score.

Returns the overall weighted average score regardless of sufficient/necessary field matching. This provides a more nuanced score for use in comparators.

Parameters:

Name	Type	Description	Default
other	StructuredModel	Another instance of the same model to compare with	required

Returns:

Type	Description
float	Similarity score between 0.0 and 1.0

Source code in stickler/structured_object_evaluator/models/structured_model.py

def compare(self, other: "StructuredModel") -> float:
    """Compare this model with another and return a scalar similarity score.

    Returns the overall weighted average score regardless of sufficient/necessary field matching.
    This provides a more nuanced score for use in comparators.

    Args:
        other: Another instance of the same model to compare with

    Returns:
        Similarity score between 0.0 and 1.0
    """
    # We'll calculate the overall weighted score directly instead of using compare_with
    # This ensures that sufficient/necessary field rules don't cause a zero score
    # when at least some fields match

    total_score = 0.0
    total_weight = 0.0

    for field_name in self.__class__.model_fields:
        # Skip the extra_fields attribute in comparison
        if field_name == "extra_fields":
            continue
        if hasattr(other, field_name):
            # Get field configuration
            info = self.__class__._get_comparison_info(field_name)
            # Use weight from ComparableField object
            weight = info.weight

            # Compare field values WITHOUT applying thresholds
            field_score = self.compare_field_raw(
                field_name, getattr(other, field_name)
            )

            # Update total score
            total_score += field_score * weight
            total_weight += weight

    # Calculate overall score
    if total_weight > 0:
        return total_score / total_weight
    else:
        return 0.0

compare_field(field_name, other_value)

Compare a single field with a value using the configured comparator.

Parameters:

Name	Type	Description	Default
field_name	str	Name of the field to compare	required
other_value	Any	Value to compare with	required

Returns:

Type	Description
float	Similarity score between 0.0 and 1.0

Source code in stickler/structured_object_evaluator/models/structured_model.py

def compare_field(self, field_name: str, other_value: Any) -> float:
    """Compare a single field with a value using the configured comparator.

    Args:
        field_name: Name of the field to compare
        other_value: Value to compare with

    Returns:
        Similarity score between 0.0 and 1.0
    """
    # Get our field value
    my_value = getattr(self, field_name)

    # If both values are StructuredModel instances, use recursive compare_with
    if isinstance(my_value, StructuredModel) and isinstance(
        other_value, StructuredModel
    ):
        # Use compare_with for rich comparison
        comparison_result = my_value.compare_with(
            other_value,
            include_confusion_matrix=False,
            document_non_matches=False,
            evaluator_format=False,
            recall_with_fd=False,
        )
        # Apply field-level threshold if configured
        info = self._get_comparison_info(field_name)
        raw_score = comparison_result["overall_score"]
        return (
            raw_score
            if raw_score >= info.threshold or not info.clip_under_threshold
            else 0.0
        )

    # CRITICAL FIX: For lists, don't clip under threshold for partial matches
    if isinstance(my_value, list) and isinstance(other_value, list):
        # Get field info
        info = self._get_comparison_info(field_name)

        # Use the raw comparison result without threshold clipping for lists
        result = ComparisonHelper.compare_unordered_lists(
            my_value, other_value, info.comparator, info.threshold
        )

        # Return the overall score directly (don't clip based on threshold for lists)
        return result["overall_score"]

    # For other fields, use existing logic
    return ComparisonHelper.compare_field_with_threshold(
        self, field_name, other_value
    )

compare_field_raw(field_name, other_value)

Compare a single field with a value WITHOUT applying thresholds.

This version is used by the compare method to get raw similarity scores.

Parameters:

Name	Type	Description	Default
field_name	str	Name of the field to compare	required
other_value	Any	Value to compare with	required

Returns:

Type	Description
float	Raw similarity score between 0.0 and 1.0 without threshold filtering

Source code in stickler/structured_object_evaluator/models/structured_model.py

def compare_field_raw(self, field_name: str, other_value: Any) -> float:
    """Compare a single field with a value WITHOUT applying thresholds.

    This version is used by the compare method to get raw similarity scores.

    Args:
        field_name: Name of the field to compare
        other_value: Value to compare with

    Returns:
        Raw similarity score between 0.0 and 1.0 without threshold filtering
    """
    # Get our field value
    my_value = getattr(self, field_name)

    # If both values are StructuredModel instances, use recursive compare_with
    if isinstance(my_value, StructuredModel) and isinstance(
        other_value, StructuredModel
    ):
        # Use compare_with for rich comparison, but extract the raw score
        comparison_result = my_value.compare_with(
            other_value,
            include_confusion_matrix=False,
            document_non_matches=False,
            evaluator_format=False,
            recall_with_fd=False,
        )
        return comparison_result["overall_score"]

    # For non-StructuredModel fields, use existing logic
    return ComparisonHelper.compare_field_raw(self, field_name, other_value)

compare_recursive(other)

The ONE clean recursive function that handles everything.

Enhanced to capture BOTH confusion matrix metrics AND similarity scores in a single traversal to eliminate double traversal inefficiency.

PHASE 2: Delegates to ComparisonEngine while maintaining identical behavior.

Parameters:

Name	Type	Description	Default
other	StructuredModel	Another instance of the same model to compare with	required

Returns:

Type	Description
dict	Dictionary with clean hierarchical structure:
dict	overall: TP, FP, TN, FN, FD, FA counts + similarity_score + all_fields_matched
dict	fields: Recursive structure for each field with scores
dict	non_matches: List of non-matching items

Source code in stickler/structured_object_evaluator/models/structured_model.py

def compare_recursive(self, other: "StructuredModel") -> dict:
    """The ONE clean recursive function that handles everything.

    Enhanced to capture BOTH confusion matrix metrics AND similarity scores
    in a single traversal to eliminate double traversal inefficiency.

    PHASE 2: Delegates to ComparisonEngine while maintaining identical behavior.

    Args:
        other: Another instance of the same model to compare with

    Returns:
        Dictionary with clean hierarchical structure:
        - overall: TP, FP, TN, FN, FD, FA counts + similarity_score + all_fields_matched
        - fields: Recursive structure for each field with scores
        - non_matches: List of non-matching items
    """
    from .comparison_engine import ComparisonEngine
    engine = ComparisonEngine(self)
    return engine.compare_recursive(other)

compare_with(other, include_confusion_matrix=False, document_non_matches=False, evaluator_format=False, recall_with_fd=False, add_derived_metrics=True)

Compare this model with another instance using SINGLE TRAVERSAL optimization.

PHASE 2: Delegates to ComparisonEngine while maintaining identical behavior.

Parameters:

Name	Type	Description	Default
other	StructuredModel	Another instance of the same model to compare with	required
include_confusion_matrix	bool	Whether to include confusion matrix calculations	False
document_non_matches	bool	Whether to document non-matches for analysis	False
evaluator_format	bool	Whether to format results for the evaluator	False
recall_with_fd	bool	If True, include FD in recall denominator (TP/(TP+FN+FD)) If False, use traditional recall (TP/(TP+FN))	False
add_derived_metrics	bool	Whether to add derived metrics to confusion matrix	True

Returns:

Type	Description
Dict[str, Any]	Dictionary with comparison results including:
Dict[str, Any]	field_scores: Scores for each field
Dict[str, Any]	overall_score: Weighted average score
Dict[str, Any]	all_fields_matched: Whether all fields matched
Dict[str, Any]	confusion_matrix: (optional) Confusion matrix data if requested
Dict[str, Any]	non_matches: (optional) Non-match documentation if requested

Source code in stickler/structured_object_evaluator/models/structured_model.py

def compare_with(
    self,
    other: "StructuredModel",
    include_confusion_matrix: bool = False,
    document_non_matches: bool = False,
    evaluator_format: bool = False,
    recall_with_fd: bool = False,
    add_derived_metrics: bool = True,
) -> Dict[str, Any]:
    """Compare this model with another instance using SINGLE TRAVERSAL optimization.

    PHASE 2: Delegates to ComparisonEngine while maintaining identical behavior.

    Args:
        other: Another instance of the same model to compare with
        include_confusion_matrix: Whether to include confusion matrix calculations
        document_non_matches: Whether to document non-matches for analysis
        evaluator_format: Whether to format results for the evaluator
        recall_with_fd: If True, include FD in recall denominator (TP/(TP+FN+FD))
                        If False, use traditional recall (TP/(TP+FN))
        add_derived_metrics: Whether to add derived metrics to confusion matrix

    Returns:
        Dictionary with comparison results including:
        - field_scores: Scores for each field
        - overall_score: Weighted average score
        - all_fields_matched: Whether all fields matched
        - confusion_matrix: (optional) Confusion matrix data if requested
        - non_matches: (optional) Non-match documentation if requested
    """
    from .comparison_engine import ComparisonEngine
    engine = ComparisonEngine(self)
    return engine.compare_with(
        other,
        include_confusion_matrix=include_confusion_matrix,
        document_non_matches=document_non_matches,
        evaluator_format=evaluator_format,
        recall_with_fd=recall_with_fd,
        add_derived_metrics=add_derived_metrics,
    )

from_json(json_data) classmethod

Create a StructuredModel instance from JSON data.

This method handles missing fields gracefully and stores extra fields in the extra_fields attribute.

Parameters:

Name	Type	Description	Default
json_data	Dict[str, Any]	Dictionary containing the JSON data	required

Returns:

Type	Description
StructuredModel	StructuredModel instance created from the JSON data

Source code in stickler/structured_object_evaluator/models/structured_model.py

@classmethod
def from_json(cls, json_data: Dict[str, Any]) -> "StructuredModel":
    """Create a StructuredModel instance from JSON data.

    This method handles missing fields gracefully and stores extra fields
    in the extra_fields attribute.

    Args:
        json_data: Dictionary containing the JSON data

    Returns:
        StructuredModel instance created from the JSON data
    """
    return ConfigurationHelper.from_json(cls, json_data)

from_json_schema(schema) classmethod

Create a StructuredModel subclass from a JSON Schema document.

This method accepts standard JSON Schema documents and creates fully functional StructuredModel classes with comparison capabilities. Supports JSON Schema draft-07+.

Comparison behavior can be customized using x-aws-stickler-* extension fields:

Field-Level Extensions:

• x-aws-stickler-comparator: Comparator algorithm name (built-in or registered custom)
• x-aws-stickler-threshold: Similarity threshold for match/no-match (0.0-1.0, default: 0.5)
• x-aws-stickler-weight: Field importance in overall scoring (>0.0, default: 1.0)
• x-aws-stickler-clip-under-threshold: Clip scores below threshold to 0.0 (bool, default: false)
• x-aws-stickler-aggregate: Include field metrics in parent aggregation (bool, default: false)

Model-Level Extensions:

• x-aws-stickler-model-name: Generated class name (default: "DynamicModel")
• x-aws-stickler-match-threshold: Overall match threshold (default: 0.7)

Supported Features:

• Primitive types: string, number, integer, boolean
• Nested objects and arrays (primitive/object items)
• Required fields, defaults, descriptions
• Schema references ($ref with #/definitions/ and #/$defs/)

Default Type Mappings:

• string → LevenshteinComparator (threshold: 0.5)
• number/integer → NumericComparator (threshold: 0.5)
• boolean → ExactComparator (threshold: 1.0)
• arrays → Hungarian matching with element-appropriate comparators
• objects → Recursive field-by-field comparison

Parameters:

Name	Type	Description	Default
schema	Dict[str, Any]	JSON Schema document as a dictionary	required

Returns:

Type	Description
Type[StructuredModel]	StructuredModel subclass created from the schema

Raises:

Type	Description
ValueError	If schema is invalid or contains unsupported features
SchemaError	If schema doesn't conform to JSON Schema spec

Examples:

Basic usage with standard JSON Schema:

>>> schema = {
...     "type": "object",
...     "properties": {
...         "name": {"type": "string"},
...         "age": {"type": "integer"},
...         "email": {"type": "string"}
...     },
...     "required": ["name", "email"]
... }
>>> PersonModel = StructuredModel.from_json_schema(schema)
>>> person1 = PersonModel(name="Alice", age=30, email="alice@example.com")
>>> person2 = PersonModel(name="Alicia", age=30, email="alice@example.com")
>>> result = person1.compare_with(person2)
>>> # name field uses LevenshteinComparator, age uses NumericComparator

Advanced usage with x-aws-stickler-* extensions:

>>> schema = {
...     "type": "object",
...     "x-aws-stickler-model-name": "Product",
...     "x-aws-stickler-match-threshold": 0.8,
...     "properties": {
...         "name": {
...             "type": "string",
...             "x-aws-stickler-comparator": "LevenshteinComparator",
...             "x-aws-stickler-threshold": 0.9,
...             "x-aws-stickler-weight": 2.0,
...             "x-aws-stickler-aggregate": true
...         },
...         "price": {
...             "type": "number",
...             "x-aws-stickler-comparator": "NumericComparator",
...             "x-aws-stickler-threshold": 0.95,
...             "x-aws-stickler-clip-under-threshold": true
...         }
...     },
...     "required": ["name"]
... }
>>> ProductModel = StructuredModel.from_json_schema(schema)
>>> result = product1.compare_with(product2)
>>> # name field has weight=2.0, price field clips scores below 0.95

Source code in stickler/structured_object_evaluator/models/structured_model.py

@classmethod
def from_json_schema(cls, schema: Dict[str, Any]) -> Type["StructuredModel"]:
    """Create a StructuredModel subclass from a JSON Schema document.

    This method accepts standard JSON Schema documents and creates fully functional
    StructuredModel classes with comparison capabilities. Supports JSON Schema draft-07+.

    Comparison behavior can be customized using x-aws-stickler-* extension fields:

    Field-Level Extensions:
    -----------------------
    - x-aws-stickler-comparator: Comparator algorithm name (built-in or registered custom)
    - x-aws-stickler-threshold: Similarity threshold for match/no-match (0.0-1.0, default: 0.5)
    - x-aws-stickler-weight: Field importance in overall scoring (>0.0, default: 1.0)
    - x-aws-stickler-clip-under-threshold: Clip scores below threshold to 0.0 (bool, default: false)
    - x-aws-stickler-aggregate: Include field metrics in parent aggregation (bool, default: false)

    Model-Level Extensions:
    -----------------------
    - x-aws-stickler-model-name: Generated class name (default: "DynamicModel")
    - x-aws-stickler-match-threshold: Overall match threshold (default: 0.7)

    Supported Features:
    -------------------
    - Primitive types: string, number, integer, boolean
    - Nested objects and arrays (primitive/object items)
    - Required fields, defaults, descriptions
    - Schema references ($ref with #/definitions/ and #/$defs/)

    Default Type Mappings:
    ----------------------
    - string → LevenshteinComparator (threshold: 0.5)
    - number/integer → NumericComparator (threshold: 0.5)
    - boolean → ExactComparator (threshold: 1.0)
    - arrays → Hungarian matching with element-appropriate comparators
    - objects → Recursive field-by-field comparison

    Args:
        schema: JSON Schema document as a dictionary

    Returns:
        StructuredModel subclass created from the schema

    Raises:
        ValueError: If schema is invalid or contains unsupported features
        jsonschema.exceptions.SchemaError: If schema doesn't conform to JSON Schema spec

    Examples:
        Basic usage with standard JSON Schema:
        >>> schema = {
        ...     "type": "object",
        ...     "properties": {
        ...         "name": {"type": "string"},
        ...         "age": {"type": "integer"},
        ...         "email": {"type": "string"}
        ...     },
        ...     "required": ["name", "email"]
        ... }
        >>> PersonModel = StructuredModel.from_json_schema(schema)
        >>> person1 = PersonModel(name="Alice", age=30, email="alice@example.com")
        >>> person2 = PersonModel(name="Alicia", age=30, email="alice@example.com")
        >>> result = person1.compare_with(person2)
        >>> # name field uses LevenshteinComparator, age uses NumericComparator

        Advanced usage with x-aws-stickler-* extensions:
        >>> schema = {
        ...     "type": "object",
        ...     "x-aws-stickler-model-name": "Product",
        ...     "x-aws-stickler-match-threshold": 0.8,
        ...     "properties": {
        ...         "name": {
        ...             "type": "string",
        ...             "x-aws-stickler-comparator": "LevenshteinComparator",
        ...             "x-aws-stickler-threshold": 0.9,
        ...             "x-aws-stickler-weight": 2.0,
        ...             "x-aws-stickler-aggregate": true
        ...         },
        ...         "price": {
        ...             "type": "number",
        ...             "x-aws-stickler-comparator": "NumericComparator",
        ...             "x-aws-stickler-threshold": 0.95,
        ...             "x-aws-stickler-clip-under-threshold": true
        ...         }
        ...     },
        ...     "required": ["name"]
        ... }
        >>> ProductModel = StructuredModel.from_json_schema(schema)
        >>> result = product1.compare_with(product2)
        >>> # name field has weight=2.0, price field clips scores below 0.95
        """

    return cls._from_json_schema_internal(schema, field_path="")

model_from_json(config) classmethod

Create a StructuredModel subclass from JSON configuration using Pydantic's create_model().

This method leverages Pydantic's native dynamic model creation capabilities to ensure full compatibility with all Pydantic features while adding structured comparison functionality through inherited StructuredModel methods.

The generated model inherits all StructuredModel capabilities: - compare_with() method for detailed comparisons - Field-level comparison configuration - Hungarian algorithm for list matching - Confusion matrix generation - JSON schema with comparison metadata

Parameters:

Name	Type	Description	Default
config	Dict[str, Any]	JSON configuration with fields, comparators, and model settings. Required keys: - fields: Dict mapping field names to field configurations Optional keys: - model_name: Name for the generated class (default: "DynamicModel") - match_threshold: Overall matching threshold (default: 0.7) Field configuration format: { "type": "str|int|float|bool|List[str]|etc.", # Required "comparator": "LevenshteinComparator|ExactComparator|etc.", # Optional "threshold": 0.8, # Optional, default 0.5 "weight": 2.0, # Optional, default 1.0 "required": true, # Optional, default false "default": "value", # Optional "description": "Field description", # Optional "alias": "field_alias", # Optional "examples": ["example1", "example2"] # Optional }	required

Returns:

Type	Description
Type[StructuredModel]	A fully functional StructuredModel subclass created with create_model()

Raises:

Type	Description
ValueError	If configuration is invalid or contains unsupported types/comparators
KeyError	If required configuration keys are missing

Examples:

>>> config = {
...     "model_name": "Product",
...     "match_threshold": 0.8,
...     "fields": {
...         "name": {
...             "type": "str",
...             "comparator": "LevenshteinComparator",
...             "threshold": 0.8,
...             "weight": 2.0,
...             "required": True
...         },
...         "price": {
...             "type": "float",
...             "comparator": "NumericComparator",
...             "default": 0.0
...         }
...     }
... }
>>> ProductClass = StructuredModel.model_from_json(config)
>>> isinstance(ProductClass.model_fields, dict)  # Full Pydantic compatibility
True
>>> product = ProductClass(name="Widget", price=29.99)
>>> product.name
'Widget'
>>> result = product.compare_with(ProductClass(name="Widget", price=29.99))
>>> result["overall_score"]
1.0

Source code in stickler/structured_object_evaluator/models/structured_model.py

@classmethod
def model_from_json(cls, config: Dict[str, Any]) -> Type["StructuredModel"]:
    """Create a StructuredModel subclass from JSON configuration using Pydantic's create_model().

    This method leverages Pydantic's native dynamic model creation capabilities to ensure
    full compatibility with all Pydantic features while adding structured comparison
    functionality through inherited StructuredModel methods.

    The generated model inherits all StructuredModel capabilities:
    - compare_with() method for detailed comparisons
    - Field-level comparison configuration
    - Hungarian algorithm for list matching
    - Confusion matrix generation
    - JSON schema with comparison metadata

    Args:
        config: JSON configuration with fields, comparators, and model settings.
               Required keys:
               - fields: Dict mapping field names to field configurations
               Optional keys:
               - model_name: Name for the generated class (default: "DynamicModel")
               - match_threshold: Overall matching threshold (default: 0.7)

               Field configuration format:
               {
                   "type": "str|int|float|bool|List[str]|etc.",  # Required
                   "comparator": "LevenshteinComparator|ExactComparator|etc.",  # Optional
                   "threshold": 0.8,  # Optional, default 0.5
                   "weight": 2.0,     # Optional, default 1.0
                   "required": true,  # Optional, default false
                   "default": "value", # Optional
                   "description": "Field description",  # Optional
                   "alias": "field_alias",  # Optional
                   "examples": ["example1", "example2"]  # Optional
               }

    Returns:
        A fully functional StructuredModel subclass created with create_model()

    Raises:
        ValueError: If configuration is invalid or contains unsupported types/comparators
        KeyError: If required configuration keys are missing

    Examples:
        >>> config = {
        ...     "model_name": "Product",
        ...     "match_threshold": 0.8,
        ...     "fields": {
        ...         "name": {
        ...             "type": "str",
        ...             "comparator": "LevenshteinComparator",
        ...             "threshold": 0.8,
        ...             "weight": 2.0,
        ...             "required": True
        ...         },
        ...         "price": {
        ...             "type": "float",
        ...             "comparator": "NumericComparator",
        ...             "default": 0.0
        ...         }
        ...     }
        ... }
        >>> ProductClass = StructuredModel.model_from_json(config)
        >>> isinstance(ProductClass.model_fields, dict)  # Full Pydantic compatibility
        True
        >>> product = ProductClass(name="Widget", price=29.99)
        >>> product.name
        'Widget'
        >>> result = product.compare_with(ProductClass(name="Widget", price=29.99))
        >>> result["overall_score"]
        1.0
    """
    # Delegate to ModelFactory for dynamic model creation
    from .model_factory import ModelFactory

    return ModelFactory.create_model_from_json(config, base_class=cls)

model_json_schema(**kwargs) classmethod

Override to add model-level comparison metadata.

Extends the standard Pydantic JSON schema with comparison metadata at the field level.

Parameters:

Name	Type	Description	Default
**kwargs		Arguments to pass to the parent method	{}

Returns:

Type	Description
	JSON schema with added comparison metadata

Source code in stickler/structured_object_evaluator/models/structured_model.py

@classmethod
def model_json_schema(cls, **kwargs):
    """Override to add model-level comparison metadata.

    Extends the standard Pydantic JSON schema with comparison metadata
    at the field level.

    Args:
        **kwargs: Arguments to pass to the parent method

    Returns:
        JSON schema with added comparison metadata
    """
    schema = super().model_json_schema(**kwargs)

    # Add comparison metadata to each field in the schema
    for field_name, field_info in cls.model_fields.items():
        if field_name == "extra_fields":
            continue

        # Get the schema property for this field
        if field_name not in schema.get("properties", {}):
            continue

        field_props = schema["properties"][field_name]

        # Since ComparableField is now always a function, check for json_schema_extra
        if hasattr(field_info, "json_schema_extra") and callable(
            field_info.json_schema_extra
        ):
            # Fallback: Check for json_schema_extra function
            temp_schema = {}
            field_info.json_schema_extra(temp_schema)

            if "x-comparison" in temp_schema:
                # Copy the comparison metadata from the temp schema to the real schema
                field_props["x-comparison"] = temp_schema["x-comparison"]

    return schema

stickler.structured_object_evaluator.models.comparable_field.ComparableField(comparator=None, threshold=0.5, weight=1.0, default=None, aggregate=False, clip_under_threshold=True, alias=None, description=None, examples=None, **field_kwargs)

Create a Pydantic Field with comparison metadata.

This function creates a proper Pydantic Field with embedded comparison configuration, enabling both comparison functionality and native Pydantic features like aliases.

Parameters:

Name	Type	Description	Default
comparator	Optional[BaseComparator]	Comparator to use for field comparison (default: LevenshteinComparator)	None
threshold	float	Minimum similarity score to consider a match (default: 0.5)	0.5
weight	float	Weight of this field in overall score calculation (default: 1.0)	1.0
default	Any	Default value for the field (default: None)	None
aggregate	bool	DEPRECATED - This parameter is deprecated and will be removed in a future version. Use the new universal 'aggregate' field in compare_with() output instead.	False
clip_under_threshold	bool	Whether to zero out scores below threshold (default: True)	True
alias	Optional[str]	Pydantic field alias for serialization (default: None)	None
description	Optional[str]	Field description for documentation (default: None)	None
examples	Optional[list]	Example values for the field (default: None)	None
**field_kwargs		Additional Pydantic Field arguments	{}

Returns:

Type	Description
	Pydantic Field with embedded comparison metadata

Example

class MyModel(StructuredModel): # Basic usage (no alias): name: str = ComparableField(threshold=0.8)

# With alias (new feature):
email: str = ComparableField(
    threshold=0.9,
    alias="email_address",
    description="User's email",
    examples=["user@example.com"]
)

Source code in stickler/structured_object_evaluator/models/comparable_field.py

def ComparableField(
    comparator: Optional[BaseComparator] = None,
    threshold: float = 0.5,
    weight: float = 1.0,
    default: Any = None,
    aggregate: bool = False,
    clip_under_threshold: bool = True,
    # Pydantic Field parameters (all optional, just like Field)
    alias: Optional[str] = None,
    description: Optional[str] = None,
    examples: Optional[list] = None,
    **field_kwargs,
):
    """Create a Pydantic Field with comparison metadata.

    This function creates a proper Pydantic Field with embedded comparison configuration,
    enabling both comparison functionality and native Pydantic features like aliases.

    Args:
        comparator: Comparator to use for field comparison (default: LevenshteinComparator)
        threshold: Minimum similarity score to consider a match (default: 0.5)
        weight: Weight of this field in overall score calculation (default: 1.0)
        default: Default value for the field (default: None)
        aggregate: DEPRECATED - This parameter is deprecated and will be removed in a future version.
                  Use the new universal 'aggregate' field in compare_with() output instead.
        clip_under_threshold: Whether to zero out scores below threshold (default: True)
        alias: Pydantic field alias for serialization (default: None)
        description: Field description for documentation (default: None)
        examples: Example values for the field (default: None)
        **field_kwargs: Additional Pydantic Field arguments

    Returns:
        Pydantic Field with embedded comparison metadata

    Example:
        class MyModel(StructuredModel):
            # Basic usage (no alias):
            name: str = ComparableField(threshold=0.8)

            # With alias (new feature):
            email: str = ComparableField(
                threshold=0.9,
                alias="email_address",
                description="User's email",
                examples=["user@example.com"]
            )
    """
    # Issue deprecation warning if aggregate=True is used
    if aggregate:
        warnings.warn(
            "The 'aggregate' parameter in ComparableField is deprecated and will be removed "
            "in a future version. All nodes now automatically include an 'aggregate' field "
            "in the compare_with() output that sums primitive field metrics below that node.",
            DeprecationWarning,
            stacklevel=2,
        )

    # Create the actual comparator instance
    actual_comparator = comparator or LevenshteinComparator()

    # Create serializable metadata for JSON schema compatibility
    serializable_metadata = {
        "comparator_type": actual_comparator.__class__.__name__,
        "comparator_name": getattr(actual_comparator, "name", "unknown"),
        "comparator_config": getattr(actual_comparator, "config", {}),
        "threshold": threshold,
        "weight": weight,
        "clip_under_threshold": clip_under_threshold,
        "aggregate": aggregate,
    }

    # Create json_schema_extra function that stores runtime data
    def json_schema_extra_func(schema: Dict[str, Any]) -> None:
        schema["x-comparison"] = serializable_metadata

    # HYBRID APPROACH: Store runtime instances as function attributes
    # This works around FieldInfo's __slots__ restriction
    json_schema_extra_func._comparator_instance = actual_comparator
    json_schema_extra_func._threshold = threshold
    json_schema_extra_func._weight = weight
    json_schema_extra_func._clip_under_threshold = clip_under_threshold
    json_schema_extra_func._aggregate = aggregate
    json_schema_extra_func._comparison_metadata = serializable_metadata

    # Merge with existing json_schema_extra if provided
    existing_json_schema_extra = field_kwargs.get("json_schema_extra", {})
    if callable(existing_json_schema_extra):

        def enhanced_json_schema_extra(schema: Dict[str, Any]) -> None:
            existing_json_schema_extra(schema)
            json_schema_extra_func(schema)

        # Copy our runtime data to the enhanced function
        enhanced_json_schema_extra._comparator_instance = actual_comparator
        enhanced_json_schema_extra._threshold = threshold
        enhanced_json_schema_extra._weight = weight
        enhanced_json_schema_extra._clip_under_threshold = clip_under_threshold
        enhanced_json_schema_extra._aggregate = aggregate
        enhanced_json_schema_extra._comparison_metadata = serializable_metadata
        final_json_schema_extra = enhanced_json_schema_extra
    elif isinstance(existing_json_schema_extra, dict):

        def enhanced_json_schema_extra(schema: Dict[str, Any]) -> None:
            schema.update(existing_json_schema_extra)
            json_schema_extra_func(schema)

        # Copy our runtime data to the enhanced function
        enhanced_json_schema_extra._comparator_instance = actual_comparator
        enhanced_json_schema_extra._threshold = threshold
        enhanced_json_schema_extra._weight = weight
        enhanced_json_schema_extra._clip_under_threshold = clip_under_threshold
        enhanced_json_schema_extra._aggregate = aggregate
        enhanced_json_schema_extra._comparison_metadata = serializable_metadata
        final_json_schema_extra = enhanced_json_schema_extra
    else:
        final_json_schema_extra = json_schema_extra_func

    # Remove json_schema_extra from field_kwargs to avoid duplication
    clean_field_kwargs = {
        k: v for k, v in field_kwargs.items() if k != "json_schema_extra"
    }

    # Create the Field
    field = Field(
        default=default,
        alias=alias,
        description=description,
        examples=examples,
        json_schema_extra=final_json_schema_extra,
        **clean_field_kwargs,
    )

    return field

stickler.structured_object_evaluator.models.non_match_field.NonMatchField

Bases: BaseModel

Model for documenting non-matches in structured object evaluation.

This class stores detailed information about each non-match detected during the evaluation process, enabling more thorough analysis and debugging of evaluation results.

Source code in stickler/structured_object_evaluator/models/non_match_field.py

class NonMatchField(BaseModel):
    """Model for documenting non-matches in structured object evaluation.

    This class stores detailed information about each non-match detected
    during the evaluation process, enabling more thorough analysis and
    debugging of evaluation results.
    """

    field_path: str = Field(
        description="Dot-notation path to the field (e.g., 'address.city')"
    )
    non_match_type: NonMatchType = Field(description="Type of non-match")
    ground_truth_value: Any = Field(description="Original ground truth value")
    prediction_value: Any = Field(description="Predicted value")
    similarity_score: Optional[float] = Field(
        default=None, description="Similarity score if available"
    )
    details: Dict[str, Any] = Field(
        default_factory=dict, description="Additional context or details"
    )
    document_id: Optional[str] = Field(
        default=None, description="ID of the document this non-match belongs to"
    )

    def __str__(self) -> str:
        """Return a string representation of the non-match document."""
        similarity_str = (
            f", similarity: {self.similarity_score:.4f}"
            if self.similarity_score is not None
            else ""
        )
        doc_id_str = f" (doc: {self.document_id})" if self.document_id else ""
        return (
            f"{self.non_match_type.value.upper()} at '{self.field_path}'{similarity_str}{doc_id_str}\n"
            f"  GT: {self.ground_truth_value}\n"
            f"  Pred: {self.prediction_value}"
        )

    @staticmethod
    def filter_by_type(
        documents: List["NonMatchField"], match_type: NonMatchType
    ) -> List["NonMatchField"]:
        """
        Filter non-match documents by their type.

        Args:
            documents: List of NonMatchField instances to filter
            match_type: Type of non-match to filter for

        Returns:
            Filtered list of NonMatchField instances
        """
        return [doc for doc in documents if doc.non_match_type == match_type]

    @staticmethod
    def export_to_dict(
        documents: List["NonMatchField"],
    ) -> Dict[str, List[Dict[str, Any]]]:
        """
        Export a list of non-match documents to a dictionary for serialization.

        Args:
            documents: List of NonMatchField instances

        Returns:
            Dictionary with categorized non-matches
        """
        result = {"false_alarms": [], "false_discoveries": [], "false_negatives": []}

        for doc in documents:
            # Create a simplified entry
            entry = {
                "field_path": doc.field_path,
                "ground_truth": str(doc.ground_truth_value),
                "prediction": str(doc.prediction_value),
            }

            if doc.similarity_score is not None:
                entry["similarity_score"] = doc.similarity_score

            if doc.details:
                entry["details"] = doc.details

            if doc.non_match_type == NonMatchType.FALSE_ALARM:
                result["false_alarms"].append(entry)
            elif doc.non_match_type == NonMatchType.FALSE_DISCOVERY:
                result["false_discoveries"].append(entry)
            elif doc.non_match_type == NonMatchType.FALSE_NEGATIVE:
                result["false_negatives"].append(entry)

        return result

    @staticmethod
    def export_to_json(documents: List["NonMatchField"], output_path: str):
        """
        Export a list of non-match documents to a JSON file.

        Args:
            documents: List of NonMatchField instances
            output_path: Path to save the JSON file
        """
        # Create parent directories if needed
        Path(output_path).parent.mkdir(parents=True, exist_ok=True)

        # Export as dictionary
        data = NonMatchField.export_to_dict(documents)

        # Write to file
        with open(output_path, "w", encoding="utf-8") as f:
            json.dump(data, f, indent=2)

    @staticmethod
    def print_summary(documents: List["NonMatchField"], detailed: bool = False):
        """
        Print a summary of non-match documents.

        Args:
            documents: List of NonMatchField instances
            detailed: Whether to print detailed information for each document
        """
        # Count by type
        false_alarms = NonMatchField.filter_by_type(documents, NonMatchType.FALSE_ALARM)
        false_discoveries = NonMatchField.filter_by_type(
            documents, NonMatchType.FALSE_DISCOVERY
        )
        false_negatives = NonMatchField.filter_by_type(
            documents, NonMatchType.FALSE_NEGATIVE
        )

        # Print summary counts
        print(f"Non-matches summary:")
        print(f"- False Alarms: {len(false_alarms)}")
        print(f"- False Discoveries: {len(false_discoveries)}")
        print(f"- False Negatives: {len(false_negatives)}")

        # Print details if requested
        if detailed and documents:
            print("\nDetailed non-matches:")
            for i, doc in enumerate(documents):
                print(f"\nNon-match #{i + 1}:")
                print(f"- Type: {doc.non_match_type}")
                print(f"- Field: {doc.field_path}")
                print(f"- Ground truth: {doc.ground_truth_value}")
                print(f"- Prediction: {doc.prediction_value}")
                if doc.similarity_score is not None:
                    print(f"- Similarity: {doc.similarity_score:.4f}")

__str__()

Return a string representation of the non-match document.

Source code in stickler/structured_object_evaluator/models/non_match_field.py

def __str__(self) -> str:
    """Return a string representation of the non-match document."""
    similarity_str = (
        f", similarity: {self.similarity_score:.4f}"
        if self.similarity_score is not None
        else ""
    )
    doc_id_str = f" (doc: {self.document_id})" if self.document_id else ""
    return (
        f"{self.non_match_type.value.upper()} at '{self.field_path}'{similarity_str}{doc_id_str}\n"
        f"  GT: {self.ground_truth_value}\n"
        f"  Pred: {self.prediction_value}"
    )

export_to_dict(documents) staticmethod

Export a list of non-match documents to a dictionary for serialization.

Parameters:

Name	Type	Description	Default
documents	List[NonMatchField]	List of NonMatchField instances	required

Returns:

Type	Description
Dict[str, List[Dict[str, Any]]]	Dictionary with categorized non-matches

Source code in stickler/structured_object_evaluator/models/non_match_field.py

@staticmethod
def export_to_dict(
    documents: List["NonMatchField"],
) -> Dict[str, List[Dict[str, Any]]]:
    """
    Export a list of non-match documents to a dictionary for serialization.

    Args:
        documents: List of NonMatchField instances

    Returns:
        Dictionary with categorized non-matches
    """
    result = {"false_alarms": [], "false_discoveries": [], "false_negatives": []}

    for doc in documents:
        # Create a simplified entry
        entry = {
            "field_path": doc.field_path,
            "ground_truth": str(doc.ground_truth_value),
            "prediction": str(doc.prediction_value),
        }

        if doc.similarity_score is not None:
            entry["similarity_score"] = doc.similarity_score

        if doc.details:
            entry["details"] = doc.details

        if doc.non_match_type == NonMatchType.FALSE_ALARM:
            result["false_alarms"].append(entry)
        elif doc.non_match_type == NonMatchType.FALSE_DISCOVERY:
            result["false_discoveries"].append(entry)
        elif doc.non_match_type == NonMatchType.FALSE_NEGATIVE:
            result["false_negatives"].append(entry)

    return result

export_to_json(documents, output_path) staticmethod

Export a list of non-match documents to a JSON file.

Parameters:

Name	Type	Description	Default
documents	List[NonMatchField]	List of NonMatchField instances	required
output_path	str	Path to save the JSON file	required

Source code in stickler/structured_object_evaluator/models/non_match_field.py

@staticmethod
def export_to_json(documents: List["NonMatchField"], output_path: str):
    """
    Export a list of non-match documents to a JSON file.

    Args:
        documents: List of NonMatchField instances
        output_path: Path to save the JSON file
    """
    # Create parent directories if needed
    Path(output_path).parent.mkdir(parents=True, exist_ok=True)

    # Export as dictionary
    data = NonMatchField.export_to_dict(documents)

    # Write to file
    with open(output_path, "w", encoding="utf-8") as f:
        json.dump(data, f, indent=2)

filter_by_type(documents, match_type) staticmethod

Filter non-match documents by their type.

Parameters:

Name	Type	Description	Default
documents	List[NonMatchField]	List of NonMatchField instances to filter	required
match_type	NonMatchType	Type of non-match to filter for	required

Returns:

Type	Description
List[NonMatchField]	Filtered list of NonMatchField instances

Source code in stickler/structured_object_evaluator/models/non_match_field.py

@staticmethod
def filter_by_type(
    documents: List["NonMatchField"], match_type: NonMatchType
) -> List["NonMatchField"]:
    """
    Filter non-match documents by their type.

    Args:
        documents: List of NonMatchField instances to filter
        match_type: Type of non-match to filter for

    Returns:
        Filtered list of NonMatchField instances
    """
    return [doc for doc in documents if doc.non_match_type == match_type]

print_summary(documents, detailed=False) staticmethod

Print a summary of non-match documents.

Parameters:

Name	Type	Description	Default
documents	List[NonMatchField]	List of NonMatchField instances	required
detailed	bool	Whether to print detailed information for each document	False

Source code in stickler/structured_object_evaluator/models/non_match_field.py

@staticmethod
def print_summary(documents: List["NonMatchField"], detailed: bool = False):
    """
    Print a summary of non-match documents.

    Args:
        documents: List of NonMatchField instances
        detailed: Whether to print detailed information for each document
    """
    # Count by type
    false_alarms = NonMatchField.filter_by_type(documents, NonMatchType.FALSE_ALARM)
    false_discoveries = NonMatchField.filter_by_type(
        documents, NonMatchType.FALSE_DISCOVERY
    )
    false_negatives = NonMatchField.filter_by_type(
        documents, NonMatchType.FALSE_NEGATIVE
    )

    # Print summary counts
    print(f"Non-matches summary:")
    print(f"- False Alarms: {len(false_alarms)}")
    print(f"- False Discoveries: {len(false_discoveries)}")
    print(f"- False Negatives: {len(false_negatives)}")

    # Print details if requested
    if detailed and documents:
        print("\nDetailed non-matches:")
        for i, doc in enumerate(documents):
            print(f"\nNon-match #{i + 1}:")
            print(f"- Type: {doc.non_match_type}")
            print(f"- Field: {doc.field_path}")
            print(f"- Ground truth: {doc.ground_truth_value}")
            print(f"- Prediction: {doc.prediction_value}")
            if doc.similarity_score is not None:
                print(f"- Similarity: {doc.similarity_score:.4f}")

stickler.structured_object_evaluator.models.non_match_field.NonMatchType

Bases: str, Enum

Enum defining the types of non-matches.

Source code in stickler/structured_object_evaluator/models/non_match_field.py

class NonMatchType(str, Enum):
    """Enum defining the types of non-matches."""

    FALSE_ALARM = "false_alarm"  # GT null, prediction non-null
    FALSE_DISCOVERY = "false_discovery"  # Both non-null but don't match
    FALSE_NEGATIVE = "false_negative"  # GT non-null, prediction null

stickler.structured_object_evaluator.evaluator.StructuredModelEvaluator

Evaluator for StructuredModel objects.

This evaluator computes comprehensive metrics for StructuredModel objects, leveraging their built-in comparison capabilities. It includes confusion matrix calculations, field-level metrics, non-match documentation, and memory optimization capabilities.

Source code in stickler/structured_object_evaluator/evaluator.py

class StructuredModelEvaluator:
    """
    Evaluator for StructuredModel objects.

    This evaluator computes comprehensive metrics for StructuredModel objects,
    leveraging their built-in comparison capabilities. It includes confusion matrix
    calculations, field-level metrics, non-match documentation, and memory optimization capabilities.
    """

    def __init__(
        self,
        model_class: Optional[Type[StructuredModel]] = None,
        threshold: float = 0.5,
        verbose: bool = False,
        document_non_matches: bool = True,
        recall_with_fd: bool = False
    ):
        """
        Initialize the evaluator.

        Args:
            model_class: Optional StructuredModel class for type checking
            threshold: Similarity threshold for considering a match
            verbose: Whether to print detailed progress information
            document_non_matches: Whether to document detailed non-match information
        """
        self.model_class = model_class
        self.threshold = threshold
        self.verbose = verbose
        self.peak_memory_usage = 0
        self.recall_with_fd = recall_with_fd
        self.start_memory = get_memory_usage()

        # New attributes for documenting non-matches
        self.document_non_matches = document_non_matches
        self.non_match_documents: List[NonMatchField] = []

        warnings.warn(
            "This module is going to be removed in future versions. Use the StructuredModel.compare_with() method.",
            DeprecationWarning,
            stacklevel=2,
        )



        if self.verbose:
            print(
                f"Initialized StructuredModelEvaluator. Starting memory: {self.start_memory:.2f} MB"
            )

    def _check_memory(self):
        """Check current memory usage and update peak memory."""
        current_memory = get_memory_usage()

        if current_memory > self.peak_memory_usage:
            self.peak_memory_usage = current_memory

        if self.verbose and current_memory > self.start_memory + 100:  # 100MB increase
            print(f"Memory usage increased: {current_memory:.2f} MB")

        return current_memory

    def _calculate_metrics_from_binary(
        self,
        tp: float,
        fp: float,
        fn: float,
        tn: float = 0.0,
        fd: float = 0.0,
        recall_with_fd: bool = False,
    ) -> Dict[str, float]:
        """
        Calculate metrics from binary classification counts.

        Args:
            tp: True positive count
            fp: False positive count
            fn: False negative count
            tn: True negative count (default 0)
            fd: False discovery count (default 0) - used only when recall_with_fd=True
            recall_with_fd: Whether to use alternative recall formula including FD in denominator

        Returns:
            Dictionary with precision, recall, F1, and accuracy
        """
        # Calculate precision
        precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0

        # Calculate recall based on the selected formula
        if recall_with_fd:
            # Alternative recall: TP / (TP + FN + FD)
            recall = tp / (tp + fn + fd) if (tp + fn + fd) > 0 else 0.0
        else:
            # Traditional recall: TP / (TP + FN)
            recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0

        # Calculate F1 score
        f1 = (
            2 * (precision * recall) / (precision + recall)
            if (precision + recall) > 0
            else 0.0
        )

        # Calculate accuracy
        accuracy = (tp + tn) / (tp + fp + fn + tn) if (tp + fp + fn + tn) > 0 else 0.0

        return {
            "precision": precision,
            "recall": recall,
            "f1": f1,
            "accuracy": accuracy,
        }

    def calculate_derived_confusion_matrix_metrics(
        self, cm_counts: Dict[str, Union[int, float]]
    ) -> Dict[str, float]:
        """
        Calculate derived metrics from confusion matrix counts.

        This method uses MetricsHelper to maintain consistency and avoid code duplication.

        Args:
            cm_counts: Dictionary with confusion matrix counts containing keys:
                      'tp', 'fp', 'tn', 'fn', and optionally 'fd', 'fa'

        Returns:
            Dictionary with derived metrics: cm_precision, cm_recall, cm_f1, cm_accuracy
        """
        # Use MetricsHelper for consistent metric calculation
        from stickler.structured_object_evaluator.models.metrics_helper import (
            MetricsHelper,
        )

        metrics_helper = MetricsHelper()

        # Convert counts to the format expected by MetricsHelper
        metrics_dict = {
            "tp": int(cm_counts.get("tp", 0)),
            "fp": int(cm_counts.get("fp", 0)),
            "tn": int(cm_counts.get("tn", 0)),
            "fn": int(cm_counts.get("fn", 0)),
            "fd": int(cm_counts.get("fd", 0)),
            "fa": int(cm_counts.get("fa", 0)),
        }

        # Use MetricsHelper to calculate derived metrics
        return metrics_helper.calculate_derived_metrics(metrics_dict)

    def _convert_score_to_binary(self, score: float) -> Dict[str, float]:
        """
        Convert an ANLS Star score to binary classification counts.

        Args:
            score: ANLS Star similarity score [0-1]

        Returns:
            Dictionary with TP, FP, FN, TN counts
        """
        # For a single field comparison, there are different approaches
        # to convert a similarity score to binary classification:

        # Approach used here: If score >= threshold, count as TP with
        # proportional value, otherwise count as partial FP and partial FN
        if score >= self.threshold:
            # Handle as true positive with proportional credit
            tp = score  # Proportional TP
            fp = (
                1 - score if score < 1.0 else 0
            )  # Proportional FP for imperfect matches
            fn = 0
            tn = 0
        else:
            # Handle as false classification
            tp = 0
            fp = score  # Give partial credit for similarity even if below threshold
            fn = 1 - score  # More different = higher FN
            tn = 0

        return {"tp": tp, "fp": fp, "fn": fn, "tn": tn}

    def _is_null_value(self, value: Any) -> bool:
        """
        Determine if a value should be considered null or empty.

        Args:
            value: The value to check

        Returns:
            True if the value is null/empty, False otherwise
        """
        if value is None:
            return True
        elif hasattr(value, "__len__") and not isinstance(
            value, (str, bytes, bytearray)
        ):
            # Consider empty lists/collections as null values
            return len(value) == 0
        elif isinstance(value, (str, bytes, bytearray)):
            return len(value.strip()) == 0
        return False

    def combine_cm_dicts(
        self, cm1: Dict[str, int], cm2: Dict[str, int]
    ) -> Dict[str, int]:
        """
        Combine two confusion matrix dictionaries by adding corresponding values.

        Args:
            cm1: First confusion matrix dictionary
            cm2: Second confusion matrix dictionary

        Returns:
            Combined confusion matrix dictionary
        """
        return {
            key: cm1.get(key, 0) + cm2.get(key, 0)
            for key in ["tp", "fa", "fd", "fp", "tn", "fn"]
        }

    def add_non_match(
        self,
        field_path: str,
        non_match_type: NonMatchType,
        gt_value: Any,
        pred_value: Any,
        similarity_score: Optional[float] = None,
        details: Optional[Dict[str, Any]] = None,
    ):
        """
        Document a non-match with detailed information.

        Args:
            field_path: Dot-notation path to the field (e.g., 'address.city')
            non_match_type: Type of non-match
            gt_value: Ground truth value
            pred_value: Predicted value
            similarity_score: Optional similarity score if available
            details: Optional additional context or details
            document_id: Optional ID of the document this non-match belongs to
        """
        if not self.document_non_matches:
            return

        self.non_match_documents.append(
            NonMatchField(
                field_path=field_path,
                non_match_type=non_match_type,
                ground_truth_value=gt_value,
                prediction_value=pred_value,
                similarity_score=similarity_score,
                details=details or {},
            )
        )

    def clear_non_match_documents(self):
        """Clear the stored non-match documents."""
        self.non_match_documents = []

    def _convert_enhanced_non_match_to_field(
        self, nm_dict: Dict[str, Any]
    ) -> Dict[str, Any]:
        """Convert enhanced non-match format to NonMatchField format.

        Args:
            nm_dict: Enhanced non-match dictionary from StructuredModel

        Returns:
            Dictionary in NonMatchField format
        """
        # Map enhanced format to NonMatchField format
        converted = {
            "field_path": nm_dict.get("field_path", ""),
            "ground_truth_value": nm_dict.get("ground_truth_value"),
            "prediction_value": nm_dict.get("prediction_value"),
            "similarity_score": nm_dict.get("similarity_score"),
            "details": nm_dict.get("details", {}),
        }

        # The non_match_type is already a NonMatchType enum from StructuredModel
        converted["non_match_type"] = nm_dict.get("non_match_type")

        return converted

    def _compare_models(
        self, gt_model: StructuredModel, pred_model: StructuredModel
    ) -> Dict[str, Any]:
        """
        Compare two StructuredModel instances and return metrics.

        Args:
            gt_model: Ground truth model
            pred_model: Predicted model

        Returns:
            Dict with comparison metrics including tp, fp, fn, tn, field_scores, overall_score
        """
        # Check if inputs are valid StructuredModel instances
        if not (
            isinstance(gt_model, StructuredModel)
            and isinstance(pred_model, StructuredModel)
        ):
            raise TypeError("Both models must be StructuredModel instances")

        # If model_class is specified, check type
        if self.model_class and not (
            isinstance(gt_model, self.model_class)
            and isinstance(pred_model, self.model_class)
        ):
            raise TypeError(
                f"Both models must be instances of {self.model_class.__name__}"
            )

        # Use the built-in compare_with method from StructuredModel
        comparison_result = gt_model.compare_with(pred_model)

        # Initialize metrics
        tp = fp = fn = tn = 0

        # Determine match status
        if comparison_result["overall_score"] >= self.threshold:
            # Good enough match
            tp = 1
        else:
            # Not a good enough match
            fp = 1

        # Prepare result
        result = {
            "tp": tp,
            "fp": fp,
            "fn": fn,
            "tn": tn,
            "field_scores": comparison_result["field_scores"],
            "overall_score": comparison_result["overall_score"],
            # match_status removed - now unnecessary
        }

        return result

    def evaluate(
        self,
        ground_truth: StructuredModel,
        predictions: StructuredModel,
        recall_with_fd: bool = False,
    ) -> Dict[str, Any]:
        """
        Evaluate predictions against ground truth and return comprehensive metrics.

        Args:
            ground_truth: Ground truth data (StructuredModel instance)
            predictions: Predicted data (StructuredModel instance)
            recall_with_fd: If True, include FD in recall denominator (TP/(TP+FN+FD))
                            If False, use traditional recall (TP/(TP+FN))

        Returns:
            Dictionary with the following structure:

            {
                "overall": {
                    "precision": float,     # Overall precision [0-1]
                    "recall": float,        # Overall recall [0-1]
                    "f1": float,           # Overall F1 score [0-1]
                    "accuracy": float,     # Overall accuracy [0-1]
                    "anls_score": float    # Overall ANLS similarity score [0-1]
                },

                "fields": {
                    "<field_name>": {
                        # For primitive fields (str, int, float, bool):
                        "precision": float,
                        "recall": float,
                        "f1": float,
                        "accuracy": float,
                        "anls_score": float
                    },

                    "<list_field_name>": {
                        # For list fields (e.g., products: List[Product]):
                        "overall": {
                            "precision": float,
                            "recall": float,
                            "f1": float,
                            "accuracy": float,
                            "anls_score": float
                        },
                        "items": [
                            # Individual metrics for each matched item pair
                            {
                                "overall": {...},  # Item-level overall metrics
                                "fields": {        # Field metrics within each item
                                    "<nested_field>": {...}
                                }
                            }
                        ]
                    }
                },

                "confusion_matrix": {
                    "fields": {
                        # AGGREGATED metrics for all field types
                        "<field_name>": {
                            "tp": int,          # True positives
                            "fp": int,          # False positives
                            "tn": int,          # True negatives
                            "fn": int,          # False negatives
                            "fd": int,          # False discoveries (non-null but don't match)
                            "fa": int,          # False alarms
                            "derived": {
                                "cm_precision": float,
                                "cm_recall": float,
                                "cm_f1": float,
                                "cm_accuracy": float
                            }
                        },

                        # For list fields with nested objects, aggregated field metrics:
                        "<list_field>.<nested_field>": {
                            # Aggregated counts across ALL instances in the list
                            "tp": int,    # Total true positives for this field across all items
                            "fp": int,    # Total false positives for this field across all items
                            "fn": int,    # Total false negatives for this field across all items
                            "fd": int,    # Total false discoveries for this field across all items
                            "fa": int,    # Total false alarms for this field across all items
                            "derived": {...}
                        }
                    },

                    "overall": {
                        # Overall confusion matrix aggregating all fields
                        "tp": int, "fp": int, "tn": int, "fn": int, "fd": int, "fa": int
                        "derived": {...}
                    }
                }
            }

        Key Usage Patterns:

        1. **Individual Item Metrics** (per-instance analysis):
           ```python
           # Access metrics for each individual item in a list
           for i, item_metrics in enumerate(results['fields']['products']['items']):
               print(f"Product {i}: {item_metrics['overall']['f1']}")
           ```

        2. **Aggregated Field Metrics** (recommended for field performance analysis):
           ```python
           # Access aggregated metrics across all instances of a field type
           cm_fields = results['confusion_matrix']['fields']
           product_id_performance = cm_fields['products.product_id']
           print(f"Product ID field: {product_id_performance['derived']['cm_precision']}")

           # Get all aggregated product field metrics
           product_fields = {k: v for k, v in cm_fields.items()
                           if k.startswith('products.')}
           ```

        3. **Helper Function for Aggregated Metrics**:
           ```python
           def get_aggregated_metrics(results, list_field_name):
               '''Extract aggregated field metrics for a list field.'''
               cm_fields = results['confusion_matrix']['fields']
               prefix = f"{list_field_name}."
               return {k.replace(prefix, ''): v for k, v in cm_fields.items()
                      if k.startswith(prefix)}

           # Usage:
           product_metrics = get_aggregated_metrics(results, 'products')
           print(f"Product name precision: {product_metrics['name']['derived']['cm_precision']}")
           ```

        Note:
            - Use `results['fields'][field]['items']` for per-instance analysis
            - Use `results['confusion_matrix']['fields'][field.subfield]` for aggregated field analysis
            - Aggregated metrics provide rolled-up performance across all instances of a field type
            - Confusion matrix metrics use standard TP/FP/TN/FN/FD classification with derived metrics
        """
        # Clear any existing non-match documents
        self.clear_non_match_documents()

        # Use StructuredModel's enhanced comparison with evaluator format
        # This pushes all the heavy lifting into the StructuredModel as requested
        result = ground_truth.compare_with(
            predictions,
            include_confusion_matrix=True,
            document_non_matches=self.document_non_matches,
            evaluator_format=True,  # This makes StructuredModel return evaluator-compatible format
            recall_with_fd=recall_with_fd,
        )

        # Add non-matches to evaluator's collection if they exist
        if result.get("non_matches"):
            for nm_dict in result["non_matches"]:
                # Convert enhanced non-match format to NonMatchField format
                converted_nm = self._convert_enhanced_non_match_to_field(nm_dict)
                self.non_match_documents.append(NonMatchField(**converted_nm))

        # Process derived metrics explicitly with recall_with_fd parameter
        if "confusion_matrix" in result and "overall" in result["confusion_matrix"]:
            overall_cm = result["confusion_matrix"]["overall"]

            # Update derived metrics directly in the result
            from stickler.structured_object_evaluator.models.metrics_helper import (
                MetricsHelper,
            )

            metrics_helper = MetricsHelper()

            # Apply correct recall_with_fd to overall metrics
            derived_metrics = metrics_helper.calculate_derived_metrics(
                overall_cm, recall_with_fd=recall_with_fd
            )
            result["confusion_matrix"]["overall"]["derived"] = derived_metrics

            # Copy these to the top-level metrics if needed
            if "overall" in result:
                result["overall"]["precision"] = derived_metrics["cm_precision"]
                result["overall"]["recall"] = derived_metrics["cm_recall"]
                result["overall"]["f1"] = derived_metrics["cm_f1"]

        return result

    def _format_evaluation_results(
        self, comparison_result: Dict[str, Any]
    ) -> Dict[str, Any]:
        """
        Format StructuredModel comparison results to match expected evaluator output format.

        Args:
            comparison_result: Result from StructuredModel.compare_with()

        Returns:
            Dictionary in the expected evaluator format
        """
        # Extract components from StructuredModel result
        field_scores = comparison_result["field_scores"]
        overall_score = comparison_result["overall_score"]
        confusion_matrix = comparison_result.get("confusion_matrix", {})
        non_matches = comparison_result.get("non_matches", [])

        # Calculate field metrics using existing logic for backward compatibility
        field_metrics = {}

        for field_name, score in field_scores.items():
            # Convert field score to binary metrics using existing method
            binary = self._convert_score_to_binary(score)
            # For field metrics, fd is often not available directly, so we ignore recall_with_fd
            metrics = self._calculate_metrics_from_binary(
                binary["tp"], binary["fp"], binary["fn"], binary["tn"]
            )
            metrics["anls_score"] = score
            field_metrics[field_name] = metrics

        # Calculate overall metrics
        binary = self._convert_score_to_binary(overall_score)
        # For overall metrics, use confusion_matrix data which should have fd
        overall_fd = confusion_matrix.get("overall", {}).get("fd", 0)
        overall_metrics = self._calculate_metrics_from_binary(
            binary["tp"],
            binary["fp"],
            binary["fn"],
            binary["tn"],
            fd=overall_fd,
            recall_with_fd=self.recall_with_fd,
        )
        overall_metrics["anls_score"] = overall_score

        # Add non-matches to evaluator's collection if they exist
        if non_matches:
            for nm_dict in non_matches:
                self.non_match_documents.append(NonMatchField(**nm_dict))

        # Prepare final result in expected format
        result = {
            "overall": overall_metrics,
            "fields": field_metrics,
            "confusion_matrix": confusion_matrix,
            "non_matches": non_matches,
        }

        return result

    def _compare_model_lists(
        self, gt_models: List[StructuredModel], pred_models: List[StructuredModel]
    ) -> Dict[str, Any]:
        """
        Compare two lists of StructuredModel instances using Hungarian matching.

        Args:
            gt_models: List of ground truth models
            pred_models: List of predicted models

        Returns:
            Dict with comparison metrics including tp, fp, fn, overall_score
        """
        # Handle empty lists
        if not gt_models and not pred_models:
            return {
                "tp": 0,
                "fp": 0,
                "fn": 0,
                "tn": 0,
                "overall_score": 1.0,  # Empty lists are a perfect match
            }

        if not gt_models:
            return {
                "tp": 0,
                "fp": len(pred_models),
                "fn": 0,
                "tn": 0,
                "overall_score": 0.0,  # All predictions are false positives
            }

        if not pred_models:
            return {
                "tp": 0,
                "fp": 0,
                "fn": len(gt_models),
                "tn": 0,
                "overall_score": 0.0,  # All ground truths are false negatives
            }

        # Ensure all items are StructuredModel instances
        if not all(
            isinstance(model, StructuredModel) for model in gt_models + pred_models
        ):
            raise TypeError("All items in both lists must be StructuredModel instances")

        # If model_class is specified, check type for all models
        if self.model_class:
            if not all(
                isinstance(model, self.model_class) for model in gt_models + pred_models
            ):
                raise TypeError(
                    f"All models must be instances of {self.model_class.__name__}"
                )

        # Create a Hungarian matcher with StructuredModelComparator
        hungarian = HungarianMatcher(StructuredModelComparator())

        # Run Hungarian matching
        tp, fp = hungarian(gt_models, pred_models)

        # Calculate false negatives
        fn = len(gt_models) - tp

        # Calculate overall score (proportion of correct matches)
        max_items = max(len(gt_models), len(pred_models))
        overall_score = tp / max_items if max_items > 0 else 1.0

        return {"tp": tp, "fp": fp, "fn": fn, "tn": 0, "overall_score": overall_score}

__init__(model_class=None, threshold=0.5, verbose=False, document_non_matches=True, recall_with_fd=False)

Initialize the evaluator.

Parameters:

Name	Type	Description	Default
model_class	Optional[Type[StructuredModel]]	Optional StructuredModel class for type checking	None
threshold	float	Similarity threshold for considering a match	0.5
verbose	bool	Whether to print detailed progress information	False
document_non_matches	bool	Whether to document detailed non-match information	True

Source code in stickler/structured_object_evaluator/evaluator.py

def __init__(
    self,
    model_class: Optional[Type[StructuredModel]] = None,
    threshold: float = 0.5,
    verbose: bool = False,
    document_non_matches: bool = True,
    recall_with_fd: bool = False
):
    """
    Initialize the evaluator.

    Args:
        model_class: Optional StructuredModel class for type checking
        threshold: Similarity threshold for considering a match
        verbose: Whether to print detailed progress information
        document_non_matches: Whether to document detailed non-match information
    """
    self.model_class = model_class
    self.threshold = threshold
    self.verbose = verbose
    self.peak_memory_usage = 0
    self.recall_with_fd = recall_with_fd
    self.start_memory = get_memory_usage()

    # New attributes for documenting non-matches
    self.document_non_matches = document_non_matches
    self.non_match_documents: List[NonMatchField] = []

    warnings.warn(
        "This module is going to be removed in future versions. Use the StructuredModel.compare_with() method.",
        DeprecationWarning,
        stacklevel=2,
    )



    if self.verbose:
        print(
            f"Initialized StructuredModelEvaluator. Starting memory: {self.start_memory:.2f} MB"
        )

add_non_match(field_path, non_match_type, gt_value, pred_value, similarity_score=None, details=None)

Document a non-match with detailed information.

Parameters:

Name	Type	Description	Default
field_path	str	Dot-notation path to the field (e.g., 'address.city')	required
non_match_type	NonMatchType	Type of non-match	required
gt_value	Any	Ground truth value	required
pred_value	Any	Predicted value	required
similarity_score	Optional[float]	Optional similarity score if available	None
details	Optional[Dict[str, Any]]	Optional additional context or details	None
document_id		Optional ID of the document this non-match belongs to	required

Source code in stickler/structured_object_evaluator/evaluator.py

def add_non_match(
    self,
    field_path: str,
    non_match_type: NonMatchType,
    gt_value: Any,
    pred_value: Any,
    similarity_score: Optional[float] = None,
    details: Optional[Dict[str, Any]] = None,
):
    """
    Document a non-match with detailed information.

    Args:
        field_path: Dot-notation path to the field (e.g., 'address.city')
        non_match_type: Type of non-match
        gt_value: Ground truth value
        pred_value: Predicted value
        similarity_score: Optional similarity score if available
        details: Optional additional context or details
        document_id: Optional ID of the document this non-match belongs to
    """
    if not self.document_non_matches:
        return

    self.non_match_documents.append(
        NonMatchField(
            field_path=field_path,
            non_match_type=non_match_type,
            ground_truth_value=gt_value,
            prediction_value=pred_value,
            similarity_score=similarity_score,
            details=details or {},
        )
    )

calculate_derived_confusion_matrix_metrics(cm_counts)

Calculate derived metrics from confusion matrix counts.

This method uses MetricsHelper to maintain consistency and avoid code duplication.

Parameters:

Name	Type	Description	Default
cm_counts	Dict[str, Union[int, float]]	Dictionary with confusion matrix counts containing keys: 'tp', 'fp', 'tn', 'fn', and optionally 'fd', 'fa'	required

Returns:

Type	Description
Dict[str, float]	Dictionary with derived metrics: cm_precision, cm_recall, cm_f1, cm_accuracy

Source code in stickler/structured_object_evaluator/evaluator.py

def calculate_derived_confusion_matrix_metrics(
    self, cm_counts: Dict[str, Union[int, float]]
) -> Dict[str, float]:
    """
    Calculate derived metrics from confusion matrix counts.

    This method uses MetricsHelper to maintain consistency and avoid code duplication.

    Args:
        cm_counts: Dictionary with confusion matrix counts containing keys:
                  'tp', 'fp', 'tn', 'fn', and optionally 'fd', 'fa'

    Returns:
        Dictionary with derived metrics: cm_precision, cm_recall, cm_f1, cm_accuracy
    """
    # Use MetricsHelper for consistent metric calculation
    from stickler.structured_object_evaluator.models.metrics_helper import (
        MetricsHelper,
    )

    metrics_helper = MetricsHelper()

    # Convert counts to the format expected by MetricsHelper
    metrics_dict = {
        "tp": int(cm_counts.get("tp", 0)),
        "fp": int(cm_counts.get("fp", 0)),
        "tn": int(cm_counts.get("tn", 0)),
        "fn": int(cm_counts.get("fn", 0)),
        "fd": int(cm_counts.get("fd", 0)),
        "fa": int(cm_counts.get("fa", 0)),
    }

    # Use MetricsHelper to calculate derived metrics
    return metrics_helper.calculate_derived_metrics(metrics_dict)

clear_non_match_documents()

Clear the stored non-match documents.

Source code in stickler/structured_object_evaluator/evaluator.py

def clear_non_match_documents(self):
    """Clear the stored non-match documents."""
    self.non_match_documents = []

combine_cm_dicts(cm1, cm2)

Combine two confusion matrix dictionaries by adding corresponding values.

Parameters:

Name	Type	Description	Default
cm1	Dict[str, int]	First confusion matrix dictionary	required
cm2	Dict[str, int]	Second confusion matrix dictionary	required

Returns:

Type	Description
Dict[str, int]	Combined confusion matrix dictionary

Source code in stickler/structured_object_evaluator/evaluator.py

def combine_cm_dicts(
    self, cm1: Dict[str, int], cm2: Dict[str, int]
) -> Dict[str, int]:
    """
    Combine two confusion matrix dictionaries by adding corresponding values.

    Args:
        cm1: First confusion matrix dictionary
        cm2: Second confusion matrix dictionary

    Returns:
        Combined confusion matrix dictionary
    """
    return {
        key: cm1.get(key, 0) + cm2.get(key, 0)
        for key in ["tp", "fa", "fd", "fp", "tn", "fn"]
    }

evaluate(ground_truth, predictions, recall_with_fd=False)

Evaluate predictions against ground truth and return comprehensive metrics.

Parameters:

Name	Type	Description	Default
ground_truth	StructuredModel	Ground truth data (StructuredModel instance)	required
predictions	StructuredModel	Predicted data (StructuredModel instance)	required
recall_with_fd	bool	If True, include FD in recall denominator (TP/(TP+FN+FD)) If False, use traditional recall (TP/(TP+FN))	False

Returns:

Type	Description
Dict[str, Any]	Dictionary with the following structure:
Dict[str, Any]	{ "overall": { "precision": float, # Overall precision [0-1] "recall": float, # Overall recall [0-1] "f1": float, # Overall F1 score [0-1] "accuracy": float, # Overall accuracy [0-1] "anls_score": float # Overall ANLS similarity score [0-1] }, "fields": { "": { # For primitive fields (str, int, float, bool): "precision": float, "recall": float, "f1": float, "accuracy": float, "anls_score": float }, "<list_field_name>": { # For list fields (e.g., products: List[Product]): "overall": { "precision": float, "recall": float, "f1": float, "accuracy": float, "anls_score": float }, "items": [ # Individual metrics for each matched item pair { "overall": {...}, # Item-level overall metrics "fields": { # Field metrics within each item "<nested_field>": {...} } } ] } }, "confusion_matrix": { "fields": { # AGGREGATED metrics for all field types "": { "tp": int, # True positives "fp": int, # False positives "tn": int, # True negatives "fn": int, # False negatives "fd": int, # False discoveries (non-null but don't match) "fa": int, # False alarms "derived": { "cm_precision": float, "cm_recall": float, "cm_f1": float, "cm_accuracy": float } }, # For list fields with nested objects, aggregated field metrics: "<list_field>.<nested_field>": { # Aggregated counts across ALL instances in the list "tp": int, # Total true positives for this field across all items "fp": int, # Total false positives for this field across all items "fn": int, # Total false negatives for this field across all items "fd": int, # Total false discoveries for this field across all items "fa": int, # Total false alarms for this field across all items "derived": {...} } }, "overall": { # Overall confusion matrix aggregating all fields "tp": int, "fp": int, "tn": int, "fn": int, "fd": int, "fa": int "derived": {...} } }
Dict[str, Any]	}

Key Usage Patterns:

1. Individual Item Metrics (per-instance analysis):

   # Access metrics for each individual item in a list
   for i, item_metrics in enumerate(results['fields']['products']['items']):
       print(f"Product {i}: {item_metrics['overall']['f1']}")
   

2. Aggregated Field Metrics (recommended for field performance analysis):

   # Access aggregated metrics across all instances of a field type
   cm_fields = results['confusion_matrix']['fields']
   product_id_performance = cm_fields['products.product_id']
   print(f"Product ID field: {product_id_performance['derived']['cm_precision']}")
   
   # Get all aggregated product field metrics
   product_fields = {k: v for k, v in cm_fields.items()
                   if k.startswith('products.')}
   

3. Helper Function for Aggregated Metrics:

   def get_aggregated_metrics(results, list_field_name):
       '''Extract aggregated field metrics for a list field.'''
       cm_fields = results['confusion_matrix']['fields']
       prefix = f"{list_field_name}."
       return {k.replace(prefix, ''): v for k, v in cm_fields.items()
              if k.startswith(prefix)}
   
   # Usage:
   product_metrics = get_aggregated_metrics(results, 'products')
   print(f"Product name precision: {product_metrics['name']['derived']['cm_precision']}")
   

Note

• Use results['fields'][field]['items'] for per-instance analysis
• Use results['confusion_matrix']['fields'][field.subfield] for aggregated field analysis
• Aggregated metrics provide rolled-up performance across all instances of a field type
• Confusion matrix metrics use standard TP/FP/TN/FN/FD classification with derived metrics

Source code in stickler/structured_object_evaluator/evaluator.py

def evaluate(
    self,
    ground_truth: StructuredModel,
    predictions: StructuredModel,
    recall_with_fd: bool = False,
) -> Dict[str, Any]:
    """
    Evaluate predictions against ground truth and return comprehensive metrics.

    Args:
        ground_truth: Ground truth data (StructuredModel instance)
        predictions: Predicted data (StructuredModel instance)
        recall_with_fd: If True, include FD in recall denominator (TP/(TP+FN+FD))
                        If False, use traditional recall (TP/(TP+FN))

    Returns:
        Dictionary with the following structure:

        {
            "overall": {
                "precision": float,     # Overall precision [0-1]
                "recall": float,        # Overall recall [0-1]
                "f1": float,           # Overall F1 score [0-1]
                "accuracy": float,     # Overall accuracy [0-1]
                "anls_score": float    # Overall ANLS similarity score [0-1]
            },

            "fields": {
                "<field_name>": {
                    # For primitive fields (str, int, float, bool):
                    "precision": float,
                    "recall": float,
                    "f1": float,
                    "accuracy": float,
                    "anls_score": float
                },

                "<list_field_name>": {
                    # For list fields (e.g., products: List[Product]):
                    "overall": {
                        "precision": float,
                        "recall": float,
                        "f1": float,
                        "accuracy": float,
                        "anls_score": float
                    },
                    "items": [
                        # Individual metrics for each matched item pair
                        {
                            "overall": {...},  # Item-level overall metrics
                            "fields": {        # Field metrics within each item
                                "<nested_field>": {...}
                            }
                        }
                    ]
                }
            },

            "confusion_matrix": {
                "fields": {
                    # AGGREGATED metrics for all field types
                    "<field_name>": {
                        "tp": int,          # True positives
                        "fp": int,          # False positives
                        "tn": int,          # True negatives
                        "fn": int,          # False negatives
                        "fd": int,          # False discoveries (non-null but don't match)
                        "fa": int,          # False alarms
                        "derived": {
                            "cm_precision": float,
                            "cm_recall": float,
                            "cm_f1": float,
                            "cm_accuracy": float
                        }
                    },

                    # For list fields with nested objects, aggregated field metrics:
                    "<list_field>.<nested_field>": {
                        # Aggregated counts across ALL instances in the list
                        "tp": int,    # Total true positives for this field across all items
                        "fp": int,    # Total false positives for this field across all items
                        "fn": int,    # Total false negatives for this field across all items
                        "fd": int,    # Total false discoveries for this field across all items
                        "fa": int,    # Total false alarms for this field across all items
                        "derived": {...}
                    }
                },

                "overall": {
                    # Overall confusion matrix aggregating all fields
                    "tp": int, "fp": int, "tn": int, "fn": int, "fd": int, "fa": int
                    "derived": {...}
                }
            }
        }

    Key Usage Patterns:

    1. **Individual Item Metrics** (per-instance analysis):
       ```python
       # Access metrics for each individual item in a list
       for i, item_metrics in enumerate(results['fields']['products']['items']):
           print(f"Product {i}: {item_metrics['overall']['f1']}")
       ```

    2. **Aggregated Field Metrics** (recommended for field performance analysis):
       ```python
       # Access aggregated metrics across all instances of a field type
       cm_fields = results['confusion_matrix']['fields']
       product_id_performance = cm_fields['products.product_id']
       print(f"Product ID field: {product_id_performance['derived']['cm_precision']}")

       # Get all aggregated product field metrics
       product_fields = {k: v for k, v in cm_fields.items()
                       if k.startswith('products.')}
       ```

    3. **Helper Function for Aggregated Metrics**:
       ```python
       def get_aggregated_metrics(results, list_field_name):
           '''Extract aggregated field metrics for a list field.'''
           cm_fields = results['confusion_matrix']['fields']
           prefix = f"{list_field_name}."
           return {k.replace(prefix, ''): v for k, v in cm_fields.items()
                  if k.startswith(prefix)}

       # Usage:
       product_metrics = get_aggregated_metrics(results, 'products')
       print(f"Product name precision: {product_metrics['name']['derived']['cm_precision']}")
       ```

    Note:
        - Use `results['fields'][field]['items']` for per-instance analysis
        - Use `results['confusion_matrix']['fields'][field.subfield]` for aggregated field analysis
        - Aggregated metrics provide rolled-up performance across all instances of a field type
        - Confusion matrix metrics use standard TP/FP/TN/FN/FD classification with derived metrics
    """
    # Clear any existing non-match documents
    self.clear_non_match_documents()

    # Use StructuredModel's enhanced comparison with evaluator format
    # This pushes all the heavy lifting into the StructuredModel as requested
    result = ground_truth.compare_with(
        predictions,
        include_confusion_matrix=True,
        document_non_matches=self.document_non_matches,
        evaluator_format=True,  # This makes StructuredModel return evaluator-compatible format
        recall_with_fd=recall_with_fd,
    )

    # Add non-matches to evaluator's collection if they exist
    if result.get("non_matches"):
        for nm_dict in result["non_matches"]:
            # Convert enhanced non-match format to NonMatchField format
            converted_nm = self._convert_enhanced_non_match_to_field(nm_dict)
            self.non_match_documents.append(NonMatchField(**converted_nm))

    # Process derived metrics explicitly with recall_with_fd parameter
    if "confusion_matrix" in result and "overall" in result["confusion_matrix"]:
        overall_cm = result["confusion_matrix"]["overall"]

        # Update derived metrics directly in the result
        from stickler.structured_object_evaluator.models.metrics_helper import (
            MetricsHelper,
        )

        metrics_helper = MetricsHelper()

        # Apply correct recall_with_fd to overall metrics
        derived_metrics = metrics_helper.calculate_derived_metrics(
            overall_cm, recall_with_fd=recall_with_fd
        )
        result["confusion_matrix"]["overall"]["derived"] = derived_metrics

        # Copy these to the top-level metrics if needed
        if "overall" in result:
            result["overall"]["precision"] = derived_metrics["cm_precision"]
            result["overall"]["recall"] = derived_metrics["cm_recall"]
            result["overall"]["f1"] = derived_metrics["cm_f1"]

    return result

stickler.structured_object_evaluator.utils.anls_score.compare_structured_models(gt, pred)

Compare a ground truth model with a prediction.

This function wraps the compare_with method of StructuredModel for a more explicit API.

Parameters:

Name	Type	Description	Default
gt	StructuredModel	Ground truth model	required
pred	StructuredModel	Prediction model	required

Returns:

Type	Description
Dict[str, Any]	Comparison result dictionary

Source code in stickler/structured_object_evaluator/utils/anls_score.py

def compare_structured_models(
    gt: StructuredModel, pred: StructuredModel
) -> Dict[str, Any]:
    """Compare a ground truth model with a prediction.

    This function wraps the compare_with method of StructuredModel for
    a more explicit API.

    Args:
        gt: Ground truth model
        pred: Prediction model

    Returns:
        Comparison result dictionary
    """
    return gt.compare_with(pred)

stickler.structured_object_evaluator.utils.anls_score.anls_score(gt, pred, return_gt=False, return_key_scores=False)

Calculate ANLS* score between two objects.

This function provides a simple API for getting an ANLS* score between any two objects, similar to the original anls_score function.

Parameters:

Name	Type	Description	Default
gt	Any	Ground truth object	required
pred	Any	Prediction object	required
return_gt	bool	Whether to return the closest ground truth	False
return_key_scores	bool	Whether to return detailed key scores	False

Returns:

Type	Description
Union[float, Tuple[float, Any], Tuple[float, Any, Dict[str, Any]]]	Either just the overall score (float), or a tuple with the score and
Union[float, Tuple[float, Any], Tuple[float, Any, Dict[str, Any]]]	closest ground truth, or a tuple with the score, closest ground truth,
Union[float, Tuple[float, Any], Tuple[float, Any, Dict[str, Any]]]	and key scores.

Source code in stickler/structured_object_evaluator/utils/anls_score.py

def anls_score(
    gt: Any, pred: Any, return_gt: bool = False, return_key_scores: bool = False
) -> Union[float, Tuple[float, Any], Tuple[float, Any, Dict[str, Any]]]:
    """Calculate ANLS* score between two objects.

    This function provides a simple API for getting an ANLS* score
    between any two objects, similar to the original anls_score function.

    Args:
        gt: Ground truth object
        pred: Prediction object
        return_gt: Whether to return the closest ground truth
        return_key_scores: Whether to return detailed key scores

    Returns:
        Either just the overall score (float), or a tuple with the score and
        closest ground truth, or a tuple with the score, closest ground truth,
        and key scores.
    """
    import warnings
    from ..trees.base import ANLSTree

    # Store original gt object for possible return
    original_gt = gt

    # Handle classical QA dataset compatibility
    gt_is_list_str = isinstance(gt, list) and all(isinstance(x, str) for x in gt)
    pred_is_str = isinstance(pred, str)
    if gt_is_list_str and pred_is_str:
        warnings.warn(
            "Treating ground truth as a list of options. This is a compatibility mode for ST-VQA-like datasets."
        )
        gt = tuple(gt)

    # Create trees from the objects
    gt_tree = ANLSTree.make_tree(gt, is_gt=True)
    pred_tree = ANLSTree.make_tree(pred, is_gt=False)

    # Calculate ANLS score
    score, closest_gt, key_scores = gt_tree.anls(pred_tree)

    # Determine what to return for gt (smart detection)
    gt_to_return = original_gt if hasattr(original_gt, "model_dump") else closest_gt

    # Return the requested information
    if return_gt and return_key_scores:
        from .key_scores import construct_nested_dict

        key_scores_dict = construct_nested_dict(key_scores)
        return score, gt_to_return, key_scores_dict
    elif return_gt:
        return score, gt_to_return
    elif return_key_scores:
        from .key_scores import construct_nested_dict

        key_scores_dict = construct_nested_dict(key_scores)
        return score, key_scores_dict
    else:
        return score

stickler.structured_object_evaluator.utils.compare_json.compare_json(gt_json, pred_json, model_cls)

Compare JSON objects using a StructuredModel.

This function is a utility for comparing raw JSON objects using a StructuredModel class. It handles missing fields and extra fields gracefully.

Parameters:

Name	Type	Description	Default
gt_json	Dict[str, Any]	Ground truth JSON	required
pred_json	Dict[str, Any]	Prediction JSON	required
model_cls	Type[StructuredModel]	StructuredModel class to use for comparison	required

Returns:

Type	Description
Dict[str, Any]	Dictionary with comparison results

Source code in stickler/structured_object_evaluator/utils/compare_json.py

def compare_json(
    gt_json: Dict[str, Any], pred_json: Dict[str, Any], model_cls: Type[StructuredModel]
) -> Dict[str, Any]:
    """Compare JSON objects using a StructuredModel.

    This function is a utility for comparing raw JSON objects using a
    StructuredModel class. It handles missing fields and extra fields gracefully.

    Args:
        gt_json: Ground truth JSON
        pred_json: Prediction JSON
        model_cls: StructuredModel class to use for comparison

    Returns:
        Dictionary with comparison results
    """
    try:
        # Try to convert both JSONs to structured models
        gt_model = model_cls.from_json(gt_json)
        pred_model = model_cls.from_json(pred_json)

        # Compare the models
        return gt_model.compare_with(pred_model)
    except Exception as e:
        # Return error details if conversion fails
        return {
            "error": str(e),
            "overall_score": 0.0,
            "field_scores": {},
            "all_fields_matched": False,
        }