Document Packet Splitting Evaluation

Stickler provides two complementary metric sets for evaluating document packet splitting — the task of separating a multi-page document packet into individual documents, classifying them, and ordering their pages.

Both metric sets are based on the DocSplit paper.

When to Use Which

Metric Set	Best For
Classical (`DocSplitClassificationMetrics`)	Quick pass/fail evaluation, section-level ground truth
Proposed (`evaluate_packet`)	Nuanced evaluation with partial credit, page-level ground truth

The classical metrics are binary (exact match or miss). The proposed metrics provide continuous scores that distinguish between near-correct and completely wrong predictions.

Classical Metrics

What They Measure

Metric	Description
Page Level Accuracy	What fraction of pages have the correct document class?
Split Accuracy (Without Order)	What fraction of GT sections have a predicted section with the same page set and class?
Split Accuracy (With Order)	Same as above, but page order must also match exactly.

Input Format

load_sections() takes two lists of section dicts — one for ground truth, one for predictions.

Each section dict represents one document within the packet:

{
    "section_id": "s1",           # Unique ID (optional, auto-generated if missing)
    "document_class": "invoice",  # Document type label
    "page_indices": [0, 1, 2]    # Zero-based page indices in this document
}

The document_class field also accepts a dict: {"type": "invoice"}.

The page_indices field can also be nested: {"split_document": {"page_indices": [0, 1, 2]}}.

Rules:

page_indices are zero-based integers
Pages can be non-sequential (e.g., [0, 2, 4] for interleaved documents)
A page index should appear in exactly one section per list
GT and predicted lists don't need the same number of sections
section_id values don't need to match between GT and pred — matching is by page set + class

Example

from stickler.doc_split import DocSplitClassificationMetrics

gt = [
    {"section_id": "s1", "document_class": "invoice", "page_indices": [0, 1, 2]},
    {"section_id": "s2", "document_class": "form", "page_indices": [3, 4]},
]

pred = [
    {"section_id": "p1", "document_class": "invoice", "page_indices": [0, 1, 2]},
    {"section_id": "p2", "document_class": "form", "page_indices": [3, 4]},
]

m = DocSplitClassificationMetrics()
m.load_sections(gt, pred)
results = m.calculate_all_metrics()

print(results["page_level_accuracy"]["accuracy"])           # 1.0
print(results["split_accuracy_without_order"]["accuracy"])   # 1.0
print(results["split_accuracy_with_order"]["accuracy"])      # 1.0

# Generate a markdown report
report = m.generate_markdown_report(results)

Proposed Metrics (Packet Score)

What They Measure

Metric	Range	Description
V-measure	[0, 1]	Are predicted clusters homogeneous and complete?
Rand Index	[0, 1]	Pairwise clustering agreement
Clustering Score	[0, 1]	`w × V-measure + (1-w) × Rand Index`
Kendall's Tau	[-1, 1]	Page ordering correlation per document
Ordering Score	[-1, 1]	Mean Tau across multi-page documents
Packet Score	[-0.5, 1]	`α × Clustering + β × Ordering`

Why These Are Better Than Classical

The paper demonstrates several failure modes of classical metrics:

No partial credit: A single boundary error cascades to 0% on all affected sections
No severity distinction: Over-segmentation and under-segmentation score identically
No ordering direction: Reversed pages and scrambled pages score the same

The proposed metrics address all of these with continuous scoring.

Input Format

evaluate_packet() accepts page-level data in three formats:

List[Dict] — list of page dicts (recommended)
str — path to a CSV file
pd.DataFrame — pandas DataFrame

Each row represents one page in the packet:

{
    "group_id": "inv-01",                # GT document group
    "group_id_predicted": "inv-01",      # Predicted document group
    "page_number": 1,                    # GT page order within document
    "page_number_predicted": 1,          # Predicted page order
    "class_label": "invoice",            # GT document type (strict mode only)
    "class_label_predicted": "invoice",  # Predicted document type (strict mode only)
}

Required columns (always):

Column	Type	Description
`group_id`	str/int	Ground truth document group
`group_id_predicted`	str/int	Predicted document group
`page_number`	int	GT page order within its document
`page_number_predicted`	int	Predicted page order

Additional columns (strict mode):

Column	Type	Description
`class_label`	str	GT document type
`class_label_predicted`	str	Predicted document type

Rules:

One row per page in the packet
group_id values don't need to match group_id_predicted — clustering metrics compare partition structure, not labels
page_number represents page ordering within a document. Both within-document numbering (1, 2, 3 per document) and global-sequential numbering (e.g., form-01 uses 4, 5 if it starts at the 4th page) are valid — Kendall's Tau only compares relative ordering
Single-page groups receive a perfect ordering score of 1.0 (trivially correct order)
strict_clustering=True penalizes misclassification by breaking clustering credit for misclassified pages

Example

from stickler.doc_split import evaluate_packet

pages = [
    {"group_id": "inv-01", "group_id_predicted": "inv-01",
     "page_number": 1, "page_number_predicted": 1,
     "class_label": "invoice", "class_label_predicted": "invoice"},
    {"group_id": "inv-01", "group_id_predicted": "inv-01",
     "page_number": 2, "page_number_predicted": 2,
     "class_label": "invoice", "class_label_predicted": "invoice"},
    {"group_id": "inv-01", "group_id_predicted": "inv-01",
     "page_number": 3, "page_number_predicted": 3,
     "class_label": "invoice", "class_label_predicted": "invoice"},
    {"group_id": "form-01", "group_id_predicted": "form-01",
     "page_number": 4, "page_number_predicted": 4,
     "class_label": "form", "class_label_predicted": "form"},
    {"group_id": "form-01", "group_id_predicted": "form-01",
     "page_number": 5, "page_number_predicted": 5,
     "class_label": "form", "class_label_predicted": "form"},
]

results = evaluate_packet(pages, strict_clustering=True)

print(f"Packet Score:    {results['final_score']:.4f}")
print(f"Clustering:      {results['clustering_score']:.4f}")
print(f"V-measure:       {results['v_measure']:.4f}")
print(f"Rand Index:      {results['rand_index']:.4f}")
print(f"Ordering:        {results['avg_ordering_score']:.4f}")

From CSV

evaluate_packet() can read directly from a CSV file. The CSV must have a header row with the column names listed above.

group_id,group_id_predicted,page_number,page_number_predicted,class_label,class_label_predicted
inv-01,inv-01,1,1,invoice,invoice
inv-01,inv-01,2,2,invoice,invoice
inv-01,inv-01,3,3,invoice,invoice
form-01,form-01,4,4,form,form
form-01,form-01,5,5,form,form

results = evaluate_packet("path/to/evaluation_data.csv", strict_clustering=True)

The class_label and class_label_predicted columns can be omitted from the CSV if you're not using strict_clustering=True.

From pandas DataFrame

If you already have your data in a DataFrame (common when working with evaluation pipelines), pass it directly:

import pandas as pd
from stickler.doc_split import evaluate_packet

df = pd.read_csv("evaluation_data.csv")

# The DataFrame must have these columns:
#   group_id, group_id_predicted, page_number, page_number_predicted
# And optionally (for strict_clustering=True):
#   class_label, class_label_predicted

results = evaluate_packet(df, strict_clustering=True)

The DataFrame column names must match exactly — they are case-sensitive. The column types are flexible: group_id and group_id_predicted can be strings or integers, and page_number / page_number_predicted can be any numeric type.

Here's the full column reference for the DataFrame:

Column	Required	Type	Description
`group_id`	Always	str or int	Ground truth document group identifier. Pages with the same `group_id` belong to the same document.
`group_id_predicted`	Always	str or int	Model's predicted document group. Does not need to use the same labels as `group_id` — only the partition structure matters.
`page_number`	Always	int	Ground truth page position within its document. Used for ordering evaluation via Kendall's Tau.
`page_number_predicted`	Always	int	Model's predicted page position within its document.
`class_label`	Strict mode	str	Ground truth document type (e.g., `"invoice"`, `"form"`, `"letter"`).
`class_label_predicted`	Strict mode	str	Model's predicted document type.

Tuning Weights

# Emphasize clustering over ordering
results = evaluate_packet(pages, alpha=0.7, beta=0.3)

# Emphasize V-measure over Rand Index in clustering
results = evaluate_packet(pages, v_measure_weight=0.8)

API Reference

`stickler.doc_split.DocSplitClassificationMetrics`

Calculator for document split classification accuracy metrics.

Evaluates three types of accuracy: 1. Page Level Accuracy: Classification accuracy for individual pages 2. Split Accuracy (Without Order): Correct page grouping regardless of order 3. Split Accuracy (With Order): Correct page grouping with exact order

Source code in stickler/doc_split/doc_split_classification_metrics.py

class DocSplitClassificationMetrics:
    """
    Calculator for document split classification accuracy metrics.

    Evaluates three types of accuracy:
    1. Page Level Accuracy: Classification accuracy for individual pages
    2. Split Accuracy (Without Order): Correct page grouping regardless of order
    3. Split Accuracy (With Order): Correct page grouping with exact order
    """

    def __init__(self):
        """Initialize the metrics calculator."""
        self.page_classifications_gt: Dict[int, str] = {}  # page_index -> class
        self.page_classifications_pred: Dict[int, str] = {}  # page_index -> class
        self.sections_gt: List[Dict[str, Any]] = []
        self.sections_pred: List[Dict[str, Any]] = []
        self.errors: List[str] = []

    def _get_document_class(self, section_data: Dict[str, Any]) -> str:
        """
        Safely extract document class from section data.

        Args:
            section_data: Section data dictionary

        Returns:
            Document class name or "Unknown" if missing/invalid
        """
        if not section_data:
            return "Unknown"

        doc_class = section_data.get("document_class")
        if not doc_class or doc_class is None:
            return "Unknown"

        if isinstance(doc_class, dict):
            return doc_class.get("type", "Unknown")

        return str(doc_class) if doc_class else "Unknown"

    def _get_page_indices(self, section_data: Dict[str, Any]) -> List[int]:
        """
        Safely extract page indices from section data.

        Args:
            section_data: Section data dictionary

        Returns:
            List of page indices (0-based, may be non-sequential) or empty list
        """
        if not section_data:
            return []

        # Support both flat format (page_indices at top level)
        # and nested format (page_indices under split_document)
        page_indices = section_data.get("page_indices")
        if page_indices is None:
            split_doc = section_data.get("split_document")
            if split_doc and isinstance(split_doc, dict):
                page_indices = split_doc.get("page_indices", [])

        if not page_indices or page_indices is None:
            return []

        # Ensure all indices are integers
        try:
            return [int(idx) for idx in page_indices]
        except (ValueError, TypeError) as e:
            logger.warning(f"Invalid page indices format: {e}")
            return []

    def load_sections(
        self,
        ground_truth_sections: List[Dict[str, Any]],
        predicted_sections: List[Dict[str, Any]],
    ) -> None:
        """
        Load and parse ground truth and predicted sections.

        Each section dict should have:
            - section_id: str identifier
            - document_class: str class name (or dict with "type" key)
            - page_indices: list of int page indices (0-based)

        Alternatively, the accelerator nested format is also supported:
            - section_id: str
            - document_class: {"type": "invoice"}
            - split_document: {"page_indices": [0, 1, 2]}

        Args:
            ground_truth_sections: List of section dicts with ground truth data
            predicted_sections: List of section dicts with predicted data
        """
        # Load ground truth sections
        for section in ground_truth_sections:
            section_id = section.get("section_id", f"gt_{len(self.sections_gt)}")
            doc_class = self._get_document_class(section)
            page_indices = self._get_page_indices(section)

            if not page_indices:
                logger.warning(
                    f"Section {section_id} has no page indices in ground truth"
                )

            self.sections_gt.append(
                {
                    "section_id": section_id,
                    "document_class": doc_class,
                    "page_indices": page_indices,
                }
            )

            for page_idx in page_indices:
                self.page_classifications_gt[page_idx] = doc_class

        # Load predicted sections
        for section in predicted_sections:
            section_id = section.get("section_id", f"pred_{len(self.sections_pred)}")
            doc_class = self._get_document_class(section)
            page_indices = self._get_page_indices(section)

            if not page_indices:
                logger.warning(
                    f"Section {section_id} has no page indices in prediction"
                )

            self.sections_pred.append(
                {
                    "section_id": section_id,
                    "document_class": doc_class,
                    "page_indices": page_indices,
                }
            )

            for page_idx in page_indices:
                self.page_classifications_pred[page_idx] = doc_class

        logger.info(
            f"Loaded {len(self.sections_gt)} ground truth sections "
            f"and {len(self.sections_pred)} predicted sections"
        )

    def calculate_page_level_accuracy(self) -> Dict[str, Any]:
        """
        Calculate page-level classification accuracy.

        Compares document_class for each page index individually, regardless of
        which section it appears in.

        Returns:
            Dictionary with page-level accuracy metrics and per-page details
        """
        all_pages = set(self.page_classifications_gt.keys()) | set(
            self.page_classifications_pred.keys()
        )

        if not all_pages:
            return {
                "accuracy": 0.0,
                "total_pages": 0,
                "correct_pages": 0,
                "page_details": [],
            }

        correct_count = 0
        page_details = []

        for page_idx in sorted(all_pages):
            gt_class = self.page_classifications_gt.get(page_idx, "Missing")
            pred_class = self.page_classifications_pred.get(page_idx, "Missing")

            is_correct = gt_class == pred_class
            if is_correct:
                correct_count += 1

            page_details.append(
                {
                    "page_index": page_idx,
                    "ground_truth_class": gt_class,
                    "predicted_class": pred_class,
                    "correct": is_correct,
                }
            )

        accuracy = correct_count / len(all_pages) if all_pages else 0.0

        return {
            "accuracy": accuracy,
            "total_pages": len(all_pages),
            "correct_pages": correct_count,
            "page_details": page_details,
        }

    def calculate_split_accuracy_without_order(self) -> Dict[str, Any]:
        """
        Calculate document split accuracy without considering page order.

        For each ground truth section, checks if there's a predicted section with:
        - Same set of page indices (regardless of order)
        - Same document_class

        Returns:
            Dictionary with split accuracy metrics and per-section details
        """
        if not self.sections_gt:
            return {
                "accuracy": 0.0,
                "total_sections": 0,
                "correct_sections": 0,
                "section_details": [],
            }

        correct_count = 0
        section_details = []

        for gt_section in self.sections_gt:
            gt_pages_set = set(gt_section["page_indices"])
            gt_class = gt_section["document_class"]
            section_id = gt_section["section_id"]

            # Find if any predicted section matches (same pages set + same class)
            matched = False
            matched_pred_section = None

            for pred_section in self.sections_pred:
                pred_pages_set = set(pred_section["page_indices"])
                pred_class = pred_section["document_class"]

                if gt_pages_set == pred_pages_set and gt_class == pred_class:
                    matched = True
                    matched_pred_section = pred_section
                    break

            if matched:
                correct_count += 1

            section_details.append(
                {
                    "section_id": section_id,
                    "ground_truth_class": gt_class,
                    "ground_truth_pages": sorted(gt_section["page_indices"]),
                    "matched": matched,
                    "matched_section_id": (
                        matched_pred_section["section_id"]
                        if matched_pred_section
                        else None
                    ),
                    "predicted_class": (
                        matched_pred_section["document_class"]
                        if matched_pred_section
                        else "No Match"
                    ),
                    "predicted_pages": (
                        sorted(matched_pred_section["page_indices"])
                        if matched_pred_section
                        else []
                    ),
                }
            )

        accuracy = correct_count / len(self.sections_gt) if self.sections_gt else 0.0

        return {
            "accuracy": accuracy,
            "total_sections": len(self.sections_gt),
            "correct_sections": correct_count,
            "section_details": section_details,
        }

    def calculate_split_accuracy_with_order(self) -> Dict[str, Any]:
        """
        Calculate document split accuracy with page order consideration.

        For each ground truth section, checks if there's a predicted section with:
        - Exact same page indices list (same order)
        - Same document_class

        Returns:
            Dictionary with split accuracy metrics and per-section details
        """
        if not self.sections_gt:
            return {
                "accuracy": 0.0,
                "total_sections": 0,
                "correct_sections": 0,
                "section_details": [],
            }

        correct_count = 0
        section_details = []

        for gt_section in self.sections_gt:
            gt_pages = gt_section["page_indices"]
            gt_class = gt_section["document_class"]
            section_id = gt_section["section_id"]

            # Find if any predicted section matches (exact page order + same class)
            matched = False
            matched_pred_section = None
            order_matched = False

            for pred_section in self.sections_pred:
                pred_pages = pred_section["page_indices"]
                pred_class = pred_section["document_class"]

                # Check if pages match as sets first
                pages_match_as_set = set(gt_pages) == set(pred_pages)
                # Check if pages match in order
                pages_match_in_order = gt_pages == pred_pages
                # Check if class matches
                class_matches = gt_class == pred_class

                if pages_match_as_set and class_matches:
                    matched_pred_section = pred_section
                    order_matched = pages_match_in_order

                    if pages_match_in_order:
                        matched = True
                        break

            if matched:
                correct_count += 1

            section_details.append(
                {
                    "section_id": section_id,
                    "ground_truth_class": gt_class,
                    "ground_truth_pages": gt_pages,
                    "matched": matched,
                    "order_matched": order_matched,
                    "matched_section_id": (
                        matched_pred_section["section_id"]
                        if matched_pred_section
                        else None
                    ),
                    "predicted_class": (
                        matched_pred_section["document_class"]
                        if matched_pred_section
                        else "No Match"
                    ),
                    "predicted_pages": (
                        matched_pred_section["page_indices"]
                        if matched_pred_section
                        else []
                    ),
                }
            )

        accuracy = correct_count / len(self.sections_gt) if self.sections_gt else 0.0

        return {
            "accuracy": accuracy,
            "total_sections": len(self.sections_gt),
            "correct_sections": correct_count,
            "section_details": section_details,
        }

    def calculate_all_metrics(self) -> Dict[str, Any]:
        """
        Calculate all document split classification metrics.

        Returns:
            Dictionary containing all three accuracy types and detailed results
        """
        page_level = self.calculate_page_level_accuracy()
        split_without_order = self.calculate_split_accuracy_without_order()
        split_with_order = self.calculate_split_accuracy_with_order()

        return {
            "page_level_accuracy": page_level,
            "split_accuracy_without_order": split_without_order,
            "split_accuracy_with_order": split_with_order,
            "errors": self.errors,
        }

    def generate_markdown_report(self, metrics: Dict[str, Any]) -> str:
        """
        Generate a markdown report for document split classification metrics.

        Args:
            metrics: Dictionary containing all calculated metrics
                (output of calculate_all_metrics)

        Returns:
            Formatted markdown report string
        """
        sections = []

        # Title
        sections.append("# Document Split Classification Evaluation")
        sections.append("")

        # Extract metrics
        page_level = metrics["page_level_accuracy"]
        split_no_order = metrics["split_accuracy_without_order"]
        split_with_order = metrics["split_accuracy_with_order"]

        # Helper function for visual indicator
        def get_indicator(accuracy: float) -> str:
            if accuracy >= 0.9:
                return "🟢"
            elif accuracy >= 0.7:
                return "🟡"
            elif accuracy >= 0.5:
                return "🟠"
            else:
                return "🔴"

        def get_rating(accuracy: float) -> str:
            if accuracy >= 0.9:
                return "🟢 Excellent"
            elif accuracy >= 0.7:
                return "🟡 Good"
            elif accuracy >= 0.5:
                return "🟠 Fair"
            else:
                return "🔴 Poor"

        # Summary section
        sections.append("## 🎯 Split Classification Summary")
        sections.append("")

        page_acc = page_level["accuracy"]
        split_no_ord_acc = split_no_order["accuracy"]
        split_ord_acc = split_with_order["accuracy"]

        # Create progress bars
        page_percent = int(page_acc * 100)
        page_progress = f"[{'█' * (page_percent // 5)}{'░' * (20 - page_percent // 5)}] {page_percent}%"

        split_no_ord_percent = int(split_no_ord_acc * 100)
        split_no_ord_progress = f"[{'█' * (split_no_ord_percent // 5)}{'░' * (20 - split_no_ord_percent // 5)}] {split_no_ord_percent}%"

        split_ord_percent = int(split_ord_acc * 100)
        split_ord_progress = f"[{'█' * (split_ord_percent // 5)}{'░' * (20 - split_ord_percent // 5)}] {split_ord_percent}%"

        sections.append(
            f"- **Page Level Accuracy**: {get_indicator(page_acc)} "
            f"{page_level['correct_pages']}/{page_level['total_pages']} pages "
            f"{page_progress}"
        )
        sections.append(
            f"- **Split Accuracy (Without Order)**: {get_indicator(split_no_ord_acc)} "
            f"{split_no_order['correct_sections']}/{split_no_order['total_sections']} sections "
            f"{split_no_ord_progress}"
        )
        sections.append(
            f"- **Split Accuracy (With Order)**: {get_indicator(split_ord_acc)} "
            f"{split_with_order['correct_sections']}/{split_with_order['total_sections']} sections "
            f"{split_ord_progress}"
        )
        sections.append("")

        # Detailed metrics table
        sections.append("## 📊 Split Classification Metrics")
        sections.append("")
        sections.append("| Metric | Accuracy | Rating | Correct/Total |")
        sections.append("| ------ | :------: | :----: | :-----------: |")
        sections.append(
            f"| Page Level Classification | {page_acc:.4f} | {get_rating(page_acc)} | "
            f"{page_level['correct_pages']}/{page_level['total_pages']} pages |"
        )
        sections.append(
            f"| Document Split (Without Page Order) | {split_no_ord_acc:.4f} | {get_rating(split_no_ord_acc)} | "
            f"{split_no_order['correct_sections']}/{split_no_order['total_sections']} sections |"
        )
        sections.append(
            f"| Document Split (With Page Order) | {split_ord_acc:.4f} | {get_rating(split_ord_acc)} | "
            f"{split_with_order['correct_sections']}/{split_with_order['total_sections']} sections |"
        )
        sections.append("")

        # Combined Section split analysis
        sections.append("## 📑 Section Split Analysis")
        sections.append("")

        if split_no_order["section_details"] or split_with_order["section_details"]:
            sections.append(
                "| Section Match | Page Order Match | Section ID | Expected Class | Expected Pages | Pred Class | Pred Pages | Matched Section |"
            )
            sections.append(
                "| :-----------: | :--------------: | ---------- | -------------- | -------------- | ---------- | ---------- | --------------- |"
            )

            # Track matched predicted section IDs
            matched_pred_section_ids = set()

            for idx, section_with_order in enumerate(
                split_with_order["section_details"]
            ):
                section_no_order = (
                    split_no_order["section_details"][idx]
                    if idx < len(split_no_order["section_details"])
                    else None
                )

                section_match = (
                    "✅" if section_no_order and section_no_order["matched"] else "❌"
                )
                page_order_match = (
                    "✅" if section_with_order.get("order_matched", False) else "❌"
                )

                gt_pages = str(section_with_order["ground_truth_pages"])
                pred_pages = (
                    str(section_with_order["predicted_pages"])
                    if section_with_order["predicted_pages"]
                    else "N/A"
                )
                matched_id = (
                    section_with_order["matched_section_id"]
                    if section_with_order["matched_section_id"]
                    else "N/A"
                )

                if section_with_order["matched_section_id"]:
                    matched_pred_section_ids.add(
                        section_with_order["matched_section_id"]
                    )

                sections.append(
                    f"| {section_match} | {page_order_match} | {section_with_order['section_id']} | "
                    f"{section_with_order['ground_truth_class']} | {gt_pages} | {section_with_order['predicted_class']} | "
                    f"{pred_pages} | {matched_id} |"
                )

            # Add unmatched predicted sections
            for pred_section in self.sections_pred:
                if pred_section["section_id"] not in matched_pred_section_ids:
                    pred_pages = str(pred_section["page_indices"])
                    sections.append(
                        f"| ❌ | ❌ | N/A | No Match | N/A | "
                        f"{pred_section['document_class']} | {pred_pages} | {pred_section['section_id']} |"
                    )
        else:
            sections.append("*No section data available*")

        sections.append("")

        # Errors section if any
        if metrics.get("errors"):
            sections.append("## ⚠️ Errors Encountered")
            sections.append("")
            for error in metrics["errors"]:
                sections.append(f"- {error}")
            sections.append("")

        # Explanation section
        sections.append("## 📖 Metrics Explanation")
        sections.append("")
        sections.append("### Page Level Accuracy")
        sections.append("- Evaluates classification accuracy for **individual pages**")
        sections.append(
            "- Each page index is checked: does its predicted `document_class` match ground truth?"
        )
        sections.append("- Does not consider page grouping into sections")
        sections.append("")
        sections.append("### Section Split Analysis Table")
        sections.append(
            "- Combines all section-level evaluation metrics into a single comprehensive view"
        )
        sections.append(
            "- Shows both ground truth sections and unmatched predicted sections"
        )
        sections.append("")
        sections.append("**Column Definitions:**")
        sections.append("")
        sections.append(
            "- **Section Match**: ✅ if the expected section's pages (as a set) match any predicted section with the same document class, regardless of page order"
        )
        sections.append(
            "- **Page Order Match**: ✅ if Section Match is true AND the page order also matches exactly"
        )
        sections.append(
            "- **Section ID**: Ground truth section identifier (N/A for unmatched predicted sections)"
        )
        sections.append("- **Expected Class**: Document class from ground truth")
        sections.append("- **Expected Pages**: Page indices from ground truth section")
        sections.append("- **Pred Class**: Predicted document class")
        sections.append("- **Pred Pages**: Predicted page indices")
        sections.append(
            "- **Matched Section**: ID of the predicted section that matched the ground truth section"
        )
        sections.append("")
        sections.append("**Unmatched Predicted Sections:**")
        sections.append("")
        sections.append(
            "- Rows with N/A values represent predicted sections that don't correspond to any ground truth section"
        )
        sections.append(
            "- These indicate potential over-segmentation or incorrect classifications"
        )
        sections.append("")
        sections.append("### Document Split Accuracy Metrics")
        sections.append("")
        sections.append(
            "- **Without Page Order**: Counts sections where pages and class match (order doesn't matter)"
        )
        sections.append(
            "- **With Page Order**: Counts sections where pages, class, and page order all match exactly"
        )
        sections.append("")

        sections.append("---")
        sections.append("")

        return "\n".join(sections)

`init()`

Initialize the metrics calculator.

Source code in stickler/doc_split/doc_split_classification_metrics.py

def __init__(self):
    """Initialize the metrics calculator."""
    self.page_classifications_gt: Dict[int, str] = {}  # page_index -> class
    self.page_classifications_pred: Dict[int, str] = {}  # page_index -> class
    self.sections_gt: List[Dict[str, Any]] = []
    self.sections_pred: List[Dict[str, Any]] = []
    self.errors: List[str] = []

`calculate_all_metrics()`

Calculate all document split classification metrics.

Returns:

Type	Description
`Dict[str, Any]`	Dictionary containing all three accuracy types and detailed results

Source code in stickler/doc_split/doc_split_classification_metrics.py

def calculate_all_metrics(self) -> Dict[str, Any]:
    """
    Calculate all document split classification metrics.

    Returns:
        Dictionary containing all three accuracy types and detailed results
    """
    page_level = self.calculate_page_level_accuracy()
    split_without_order = self.calculate_split_accuracy_without_order()
    split_with_order = self.calculate_split_accuracy_with_order()

    return {
        "page_level_accuracy": page_level,
        "split_accuracy_without_order": split_without_order,
        "split_accuracy_with_order": split_with_order,
        "errors": self.errors,
    }

`calculate_page_level_accuracy()`

Calculate page-level classification accuracy.

Compares document_class for each page index individually, regardless of which section it appears in.

Returns:

Type	Description
`Dict[str, Any]`	Dictionary with page-level accuracy metrics and per-page details

Source code in stickler/doc_split/doc_split_classification_metrics.py

def calculate_page_level_accuracy(self) -> Dict[str, Any]:
    """
    Calculate page-level classification accuracy.

    Compares document_class for each page index individually, regardless of
    which section it appears in.

    Returns:
        Dictionary with page-level accuracy metrics and per-page details
    """
    all_pages = set(self.page_classifications_gt.keys()) | set(
        self.page_classifications_pred.keys()
    )

    if not all_pages:
        return {
            "accuracy": 0.0,
            "total_pages": 0,
            "correct_pages": 0,
            "page_details": [],
        }

    correct_count = 0
    page_details = []

    for page_idx in sorted(all_pages):
        gt_class = self.page_classifications_gt.get(page_idx, "Missing")
        pred_class = self.page_classifications_pred.get(page_idx, "Missing")

        is_correct = gt_class == pred_class
        if is_correct:
            correct_count += 1

        page_details.append(
            {
                "page_index": page_idx,
                "ground_truth_class": gt_class,
                "predicted_class": pred_class,
                "correct": is_correct,
            }
        )

    accuracy = correct_count / len(all_pages) if all_pages else 0.0

    return {
        "accuracy": accuracy,
        "total_pages": len(all_pages),
        "correct_pages": correct_count,
        "page_details": page_details,
    }

`calculate_split_accuracy_with_order()`

Calculate document split accuracy with page order consideration.

For each ground truth section, checks if there's a predicted section with: - Exact same page indices list (same order) - Same document_class

Returns:

Type	Description
`Dict[str, Any]`	Dictionary with split accuracy metrics and per-section details

Source code in stickler/doc_split/doc_split_classification_metrics.py

def calculate_split_accuracy_with_order(self) -> Dict[str, Any]:
    """
    Calculate document split accuracy with page order consideration.

    For each ground truth section, checks if there's a predicted section with:
    - Exact same page indices list (same order)
    - Same document_class

    Returns:
        Dictionary with split accuracy metrics and per-section details
    """
    if not self.sections_gt:
        return {
            "accuracy": 0.0,
            "total_sections": 0,
            "correct_sections": 0,
            "section_details": [],
        }

    correct_count = 0
    section_details = []

    for gt_section in self.sections_gt:
        gt_pages = gt_section["page_indices"]
        gt_class = gt_section["document_class"]
        section_id = gt_section["section_id"]

        # Find if any predicted section matches (exact page order + same class)
        matched = False
        matched_pred_section = None
        order_matched = False

        for pred_section in self.sections_pred:
            pred_pages = pred_section["page_indices"]
            pred_class = pred_section["document_class"]

            # Check if pages match as sets first
            pages_match_as_set = set(gt_pages) == set(pred_pages)
            # Check if pages match in order
            pages_match_in_order = gt_pages == pred_pages
            # Check if class matches
            class_matches = gt_class == pred_class

            if pages_match_as_set and class_matches:
                matched_pred_section = pred_section
                order_matched = pages_match_in_order

                if pages_match_in_order:
                    matched = True
                    break

        if matched:
            correct_count += 1

        section_details.append(
            {
                "section_id": section_id,
                "ground_truth_class": gt_class,
                "ground_truth_pages": gt_pages,
                "matched": matched,
                "order_matched": order_matched,
                "matched_section_id": (
                    matched_pred_section["section_id"]
                    if matched_pred_section
                    else None
                ),
                "predicted_class": (
                    matched_pred_section["document_class"]
                    if matched_pred_section
                    else "No Match"
                ),
                "predicted_pages": (
                    matched_pred_section["page_indices"]
                    if matched_pred_section
                    else []
                ),
            }
        )

    accuracy = correct_count / len(self.sections_gt) if self.sections_gt else 0.0

    return {
        "accuracy": accuracy,
        "total_sections": len(self.sections_gt),
        "correct_sections": correct_count,
        "section_details": section_details,
    }

`calculate_split_accuracy_without_order()`

Calculate document split accuracy without considering page order.

For each ground truth section, checks if there's a predicted section with: - Same set of page indices (regardless of order) - Same document_class

Returns:

Type	Description
`Dict[str, Any]`	Dictionary with split accuracy metrics and per-section details

Source code in stickler/doc_split/doc_split_classification_metrics.py

def calculate_split_accuracy_without_order(self) -> Dict[str, Any]:
    """
    Calculate document split accuracy without considering page order.

    For each ground truth section, checks if there's a predicted section with:
    - Same set of page indices (regardless of order)
    - Same document_class

    Returns:
        Dictionary with split accuracy metrics and per-section details
    """
    if not self.sections_gt:
        return {
            "accuracy": 0.0,
            "total_sections": 0,
            "correct_sections": 0,
            "section_details": [],
        }

    correct_count = 0
    section_details = []

    for gt_section in self.sections_gt:
        gt_pages_set = set(gt_section["page_indices"])
        gt_class = gt_section["document_class"]
        section_id = gt_section["section_id"]

        # Find if any predicted section matches (same pages set + same class)
        matched = False
        matched_pred_section = None

        for pred_section in self.sections_pred:
            pred_pages_set = set(pred_section["page_indices"])
            pred_class = pred_section["document_class"]

            if gt_pages_set == pred_pages_set and gt_class == pred_class:
                matched = True
                matched_pred_section = pred_section
                break

        if matched:
            correct_count += 1

        section_details.append(
            {
                "section_id": section_id,
                "ground_truth_class": gt_class,
                "ground_truth_pages": sorted(gt_section["page_indices"]),
                "matched": matched,
                "matched_section_id": (
                    matched_pred_section["section_id"]
                    if matched_pred_section
                    else None
                ),
                "predicted_class": (
                    matched_pred_section["document_class"]
                    if matched_pred_section
                    else "No Match"
                ),
                "predicted_pages": (
                    sorted(matched_pred_section["page_indices"])
                    if matched_pred_section
                    else []
                ),
            }
        )

    accuracy = correct_count / len(self.sections_gt) if self.sections_gt else 0.0

    return {
        "accuracy": accuracy,
        "total_sections": len(self.sections_gt),
        "correct_sections": correct_count,
        "section_details": section_details,
    }

`generate_markdown_report(metrics)`

Generate a markdown report for document split classification metrics.

Parameters:

Name	Type	Description	Default
`metrics`	`Dict[str, Any]`	Dictionary containing all calculated metrics (output of calculate_all_metrics)	required

Returns:

Type	Description
`str`	Formatted markdown report string

Source code in stickler/doc_split/doc_split_classification_metrics.py

def generate_markdown_report(self, metrics: Dict[str, Any]) -> str:
    """
    Generate a markdown report for document split classification metrics.

    Args:
        metrics: Dictionary containing all calculated metrics
            (output of calculate_all_metrics)

    Returns:
        Formatted markdown report string
    """
    sections = []

    # Title
    sections.append("# Document Split Classification Evaluation")
    sections.append("")

    # Extract metrics
    page_level = metrics["page_level_accuracy"]
    split_no_order = metrics["split_accuracy_without_order"]
    split_with_order = metrics["split_accuracy_with_order"]

    # Helper function for visual indicator
    def get_indicator(accuracy: float) -> str:
        if accuracy >= 0.9:
            return "🟢"
        elif accuracy >= 0.7:
            return "🟡"
        elif accuracy >= 0.5:
            return "🟠"
        else:
            return "🔴"

    def get_rating(accuracy: float) -> str:
        if accuracy >= 0.9:
            return "🟢 Excellent"
        elif accuracy >= 0.7:
            return "🟡 Good"
        elif accuracy >= 0.5:
            return "🟠 Fair"
        else:
            return "🔴 Poor"

    # Summary section
    sections.append("## 🎯 Split Classification Summary")
    sections.append("")

    page_acc = page_level["accuracy"]
    split_no_ord_acc = split_no_order["accuracy"]
    split_ord_acc = split_with_order["accuracy"]

    # Create progress bars
    page_percent = int(page_acc * 100)
    page_progress = f"[{'█' * (page_percent // 5)}{'░' * (20 - page_percent // 5)}] {page_percent}%"

    split_no_ord_percent = int(split_no_ord_acc * 100)
    split_no_ord_progress = f"[{'█' * (split_no_ord_percent // 5)}{'░' * (20 - split_no_ord_percent // 5)}] {split_no_ord_percent}%"

    split_ord_percent = int(split_ord_acc * 100)
    split_ord_progress = f"[{'█' * (split_ord_percent // 5)}{'░' * (20 - split_ord_percent // 5)}] {split_ord_percent}%"

    sections.append(
        f"- **Page Level Accuracy**: {get_indicator(page_acc)} "
        f"{page_level['correct_pages']}/{page_level['total_pages']} pages "
        f"{page_progress}"
    )
    sections.append(
        f"- **Split Accuracy (Without Order)**: {get_indicator(split_no_ord_acc)} "
        f"{split_no_order['correct_sections']}/{split_no_order['total_sections']} sections "
        f"{split_no_ord_progress}"
    )
    sections.append(
        f"- **Split Accuracy (With Order)**: {get_indicator(split_ord_acc)} "
        f"{split_with_order['correct_sections']}/{split_with_order['total_sections']} sections "
        f"{split_ord_progress}"
    )
    sections.append("")

    # Detailed metrics table
    sections.append("## 📊 Split Classification Metrics")
    sections.append("")
    sections.append("| Metric | Accuracy | Rating | Correct/Total |")
    sections.append("| ------ | :------: | :----: | :-----------: |")
    sections.append(
        f"| Page Level Classification | {page_acc:.4f} | {get_rating(page_acc)} | "
        f"{page_level['correct_pages']}/{page_level['total_pages']} pages |"
    )
    sections.append(
        f"| Document Split (Without Page Order) | {split_no_ord_acc:.4f} | {get_rating(split_no_ord_acc)} | "
        f"{split_no_order['correct_sections']}/{split_no_order['total_sections']} sections |"
    )
    sections.append(
        f"| Document Split (With Page Order) | {split_ord_acc:.4f} | {get_rating(split_ord_acc)} | "
        f"{split_with_order['correct_sections']}/{split_with_order['total_sections']} sections |"
    )
    sections.append("")

    # Combined Section split analysis
    sections.append("## 📑 Section Split Analysis")
    sections.append("")

    if split_no_order["section_details"] or split_with_order["section_details"]:
        sections.append(
            "| Section Match | Page Order Match | Section ID | Expected Class | Expected Pages | Pred Class | Pred Pages | Matched Section |"
        )
        sections.append(
            "| :-----------: | :--------------: | ---------- | -------------- | -------------- | ---------- | ---------- | --------------- |"
        )

        # Track matched predicted section IDs
        matched_pred_section_ids = set()

        for idx, section_with_order in enumerate(
            split_with_order["section_details"]
        ):
            section_no_order = (
                split_no_order["section_details"][idx]
                if idx < len(split_no_order["section_details"])
                else None
            )

            section_match = (
                "✅" if section_no_order and section_no_order["matched"] else "❌"
            )
            page_order_match = (
                "✅" if section_with_order.get("order_matched", False) else "❌"
            )

            gt_pages = str(section_with_order["ground_truth_pages"])
            pred_pages = (
                str(section_with_order["predicted_pages"])
                if section_with_order["predicted_pages"]
                else "N/A"
            )
            matched_id = (
                section_with_order["matched_section_id"]
                if section_with_order["matched_section_id"]
                else "N/A"
            )

            if section_with_order["matched_section_id"]:
                matched_pred_section_ids.add(
                    section_with_order["matched_section_id"]
                )

            sections.append(
                f"| {section_match} | {page_order_match} | {section_with_order['section_id']} | "
                f"{section_with_order['ground_truth_class']} | {gt_pages} | {section_with_order['predicted_class']} | "
                f"{pred_pages} | {matched_id} |"
            )

        # Add unmatched predicted sections
        for pred_section in self.sections_pred:
            if pred_section["section_id"] not in matched_pred_section_ids:
                pred_pages = str(pred_section["page_indices"])
                sections.append(
                    f"| ❌ | ❌ | N/A | No Match | N/A | "
                    f"{pred_section['document_class']} | {pred_pages} | {pred_section['section_id']} |"
                )
    else:
        sections.append("*No section data available*")

    sections.append("")

    # Errors section if any
    if metrics.get("errors"):
        sections.append("## ⚠️ Errors Encountered")
        sections.append("")
        for error in metrics["errors"]:
            sections.append(f"- {error}")
        sections.append("")

    # Explanation section
    sections.append("## 📖 Metrics Explanation")
    sections.append("")
    sections.append("### Page Level Accuracy")
    sections.append("- Evaluates classification accuracy for **individual pages**")
    sections.append(
        "- Each page index is checked: does its predicted `document_class` match ground truth?"
    )
    sections.append("- Does not consider page grouping into sections")
    sections.append("")
    sections.append("### Section Split Analysis Table")
    sections.append(
        "- Combines all section-level evaluation metrics into a single comprehensive view"
    )
    sections.append(
        "- Shows both ground truth sections and unmatched predicted sections"
    )
    sections.append("")
    sections.append("**Column Definitions:**")
    sections.append("")
    sections.append(
        "- **Section Match**: ✅ if the expected section's pages (as a set) match any predicted section with the same document class, regardless of page order"
    )
    sections.append(
        "- **Page Order Match**: ✅ if Section Match is true AND the page order also matches exactly"
    )
    sections.append(
        "- **Section ID**: Ground truth section identifier (N/A for unmatched predicted sections)"
    )
    sections.append("- **Expected Class**: Document class from ground truth")
    sections.append("- **Expected Pages**: Page indices from ground truth section")
    sections.append("- **Pred Class**: Predicted document class")
    sections.append("- **Pred Pages**: Predicted page indices")
    sections.append(
        "- **Matched Section**: ID of the predicted section that matched the ground truth section"
    )
    sections.append("")
    sections.append("**Unmatched Predicted Sections:**")
    sections.append("")
    sections.append(
        "- Rows with N/A values represent predicted sections that don't correspond to any ground truth section"
    )
    sections.append(
        "- These indicate potential over-segmentation or incorrect classifications"
    )
    sections.append("")
    sections.append("### Document Split Accuracy Metrics")
    sections.append("")
    sections.append(
        "- **Without Page Order**: Counts sections where pages and class match (order doesn't matter)"
    )
    sections.append(
        "- **With Page Order**: Counts sections where pages, class, and page order all match exactly"
    )
    sections.append("")

    sections.append("---")
    sections.append("")

    return "\n".join(sections)

`load_sections(ground_truth_sections, predicted_sections)`

Load and parse ground truth and predicted sections.

Each section dict should have

section_id: str identifier
document_class: str class name (or dict with "type" key)
page_indices: list of int page indices (0-based)

Alternatively, the accelerator nested format is also supported: - section_id: str - document_class: {"type": "invoice"} - split_document: {"page_indices": [0, 1, 2]}

Parameters:

Name	Type	Description	Default
`ground_truth_sections`	`List[Dict[str, Any]]`	List of section dicts with ground truth data	required
`predicted_sections`	`List[Dict[str, Any]]`	List of section dicts with predicted data	required

Source code in stickler/doc_split/doc_split_classification_metrics.py

def load_sections(
    self,
    ground_truth_sections: List[Dict[str, Any]],
    predicted_sections: List[Dict[str, Any]],
) -> None:
    """
    Load and parse ground truth and predicted sections.

    Each section dict should have:
        - section_id: str identifier
        - document_class: str class name (or dict with "type" key)
        - page_indices: list of int page indices (0-based)

    Alternatively, the accelerator nested format is also supported:
        - section_id: str
        - document_class: {"type": "invoice"}
        - split_document: {"page_indices": [0, 1, 2]}

    Args:
        ground_truth_sections: List of section dicts with ground truth data
        predicted_sections: List of section dicts with predicted data
    """
    # Load ground truth sections
    for section in ground_truth_sections:
        section_id = section.get("section_id", f"gt_{len(self.sections_gt)}")
        doc_class = self._get_document_class(section)
        page_indices = self._get_page_indices(section)

        if not page_indices:
            logger.warning(
                f"Section {section_id} has no page indices in ground truth"
            )

        self.sections_gt.append(
            {
                "section_id": section_id,
                "document_class": doc_class,
                "page_indices": page_indices,
            }
        )

        for page_idx in page_indices:
            self.page_classifications_gt[page_idx] = doc_class

    # Load predicted sections
    for section in predicted_sections:
        section_id = section.get("section_id", f"pred_{len(self.sections_pred)}")
        doc_class = self._get_document_class(section)
        page_indices = self._get_page_indices(section)

        if not page_indices:
            logger.warning(
                f"Section {section_id} has no page indices in prediction"
            )

        self.sections_pred.append(
            {
                "section_id": section_id,
                "document_class": doc_class,
                "page_indices": page_indices,
            }
        )

        for page_idx in page_indices:
            self.page_classifications_pred[page_idx] = doc_class

    logger.info(
        f"Loaded {len(self.sections_gt)} ground truth sections "
        f"and {len(self.sections_pred)} predicted sections"
    )

`stickler.doc_split.packet_evaluation_metrics.evaluate_packet(data, v_measure_weight=0.5, alpha=0.5, beta=0.5, strict_clustering=False)`

Evaluate document packet splitting performance.

Accepts flexible input (list of dicts, CSV path, or DataFrame) and computes clustering + ordering metrics.

Required columns

group_id: Ground truth document group
group_id_predicted: Predicted document group
page_number: Ground truth page order within document
page_number_predicted: Predicted page order

For strict_clustering=True, also requires: - class_label: Ground truth document type - class_label_predicted: Predicted document type

Parameters:

Name	Type	Description	Default
`data`	`Union[List[Dict[str, Any]], str, DataFrame]`	Page-level evaluation data.	required
`v_measure_weight`	`float`	Weight for V-measure in clustering score.	`0.5`
`alpha`	`float`	Weight for clustering in final score.	`0.5`
`beta`	`float`	Weight for ordering in final score.	`0.5`
`strict_clustering`	`bool`	If True, penalize misclassification in clustering.	`False`

Returns:

Type	Description
`Dict[str, Any]`	Dict with final_score, clustering_score, v_measure, rand_index,
`Dict[str, Any]`	avg_ordering_score, group_ordering_scores, strict_clustering.

Source code in stickler/doc_split/packet_evaluation_metrics.py

def evaluate_packet(
    data: Union[List[Dict[str, Any]], str, pd.DataFrame],
    v_measure_weight: float = 0.5,
    alpha: float = 0.5,
    beta: float = 0.5,
    strict_clustering: bool = False,
) -> Dict[str, Any]:
    """
    Evaluate document packet splitting performance.

    Accepts flexible input (list of dicts, CSV path, or DataFrame) and
    computes clustering + ordering metrics.

    Required columns:
        - group_id: Ground truth document group
        - group_id_predicted: Predicted document group
        - page_number: Ground truth page order within document
        - page_number_predicted: Predicted page order

    For strict_clustering=True, also requires:
        - class_label: Ground truth document type
        - class_label_predicted: Predicted document type

    Args:
        data: Page-level evaluation data.
        v_measure_weight: Weight for V-measure in clustering score.
        alpha: Weight for clustering in final score.
        beta: Weight for ordering in final score.
        strict_clustering: If True, penalize misclassification in clustering.

    Returns:
        Dict with final_score, clustering_score, v_measure, rand_index,
        avg_ordering_score, group_ordering_scores, strict_clustering.
    """
    df = _to_dataframe(data)

    if df.empty:
        return {
            "final_score": 0.0,
            "clustering_score": 0.0,
            "v_measure": 0.0,
            "rand_index": 0.0,
            "avg_ordering_score": 0.0,
            "group_ordering_scores": {},
            "strict_clustering": strict_clustering,
        }

    clustering_score, v_measure, ri = calculate_clustering_score(
        df, v_measure_weight, strict_clustering
    )

    group_scores = calculate_ordering_score_per_group(df)
    avg_ordering = calculate_average_ordering_score(group_scores)

    final = calculate_final_score(
        clustering_score, v_measure, avg_ordering, alpha, beta
    )

    return {
        "final_score": final,
        "clustering_score": clustering_score,
        "v_measure": v_measure,
        "rand_index": ri,
        "avg_ordering_score": avg_ordering,
        "group_ordering_scores": group_scores,
        "strict_clustering": strict_clustering,
    }

Document Packet Splitting Evaluation

When to Use Which

Classical Metrics

What They Measure

Input Format

Example

Proposed Metrics (Packet Score)

What They Measure

Why These Are Better Than Classical

Input Format

Example

From CSV

From pandas DataFrame

Tuning Weights

API Reference

stickler.doc_split.DocSplitClassificationMetrics

__init__()

calculate_all_metrics()

calculate_page_level_accuracy()

calculate_split_accuracy_with_order()

calculate_split_accuracy_without_order()

generate_markdown_report(metrics)

load_sections(ground_truth_sections, predicted_sections)

stickler.doc_split.packet_evaluation_metrics.evaluate_packet(data, v_measure_weight=0.5, alpha=0.5, beta=0.5, strict_clustering=False)

`stickler.doc_split.DocSplitClassificationMetrics`

`init()`

`calculate_all_metrics()`

`calculate_page_level_accuracy()`

`calculate_split_accuracy_with_order()`

`calculate_split_accuracy_without_order()`

`generate_markdown_report(metrics)`

`load_sections(ground_truth_sections, predicted_sections)`

`stickler.doc_split.packet_evaluation_metrics.evaluate_packet(data, v_measure_weight=0.5, alpha=0.5, beta=0.5, strict_clustering=False)`