# CMM: A Scalable Framework for Table of Contents Extraction from Complex ESG Reports > CMM introduces a construction-modelling-modification pipeline for hierarchical Table of Contents extraction from long, structurally diverse ESG annual reports, paired with the new ESGDoc dataset of 1,093 reports. - Authors: Hunter Heidenreich - Published: 2023-10-27 - Updated: 2026-04-02 - Code: CMM Code - Paper: Preprint (arXiv) - Paper: Published (EMNLP 2023) - Source: Bevaya Labs, --- ## TL;DR CMM (Construction-Modelling-Modification) is a three-stage pipeline for extracting a hierarchical Table of Contents from long, structurally diverse PDF documents. It builds an initial document tree from reading order and font sizes, encodes each node independently using GRU and Graph Attention Networks over a node-centric subtree, then predicts a Keep/Delete/Move operation per node to produce the final ToC tree. The accompanying ESGDoc dataset contains 1,093 ESG annual reports averaging 72 pages (versus 19 pages for the prior HierDoc benchmark), with less regular structure and higher assumption-violation rates. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Method}}$: The core contribution is the CMM framework itself: a three-stage pipeline for ToC extraction that replaces pairwise heading-relationship modelling with per-node operation prediction over a graph-structured subtree. The paper presents ablations, runtime comparisons, and comparisons against the prior best-reported method, all orbiting the methodological contribution. **Secondary:** $\Psi\_{\text{Resource}}$: The paper also introduces ESGDoc, a new dataset of 1,093 ESG reports spanning 2001-2022. The dataset is a meaningful standalone contribution, but it is built and released primarily to benchmark CMM and expose the limitations of MTD (the prior best-reported method on HierDoc) on long, complex documents. ## What is the motivation? Most document understanding work targets well-structured, short scientific papers. The HierDoc benchmark (Hu et al., 2022), which MTD was built and evaluated on, consists of 650 arXiv papers averaging 19 pages. In such documents, section numbering (e.g., "5.1 Experimental Setup") encodes hierarchy directly in the heading text, making ToC extraction comparatively straightforward. ESG (Environmental, Social, and Governance) annual reports present a different challenge: they routinely exceed 100 pages, lack standardized section numbering, include landscape-oriented pages, embed heavy visual elements (charts, infographics, tables), and show wide variation in font choices across companies and years. No dataset or method addressed this regime at the time. The specific failure mode of MTD on long documents is also concrete: MTD must process all headings simultaneously to model pairwise relationships, which causes out-of-memory errors on documents exceeding approximately 50 pages. This makes MTD technically inapplicable to a substantial fraction of ESGDoc. ## What is the novelty? The central insight is to reformulate ToC extraction as a per-node classification problem on a pre-built document tree, rather than as pairwise heading-relationship classification over the full document. This decoupling enables linear scaling with document length. **Tree Construction.** Using the XY-Cut algorithm (Ha et al., 1995) to establish reading order, CMM builds a complete tree $T$ over all text blocks extracted by PyMuPDF. For each block $x\_i$, CMM finds the closest preceding block $x\_j \in \{x\_{ s\_i$ (where $s$ denotes font size) and sets $x\_i$ as a child of $x\_j$. This encodes the assumption that higher-level headings have larger font sizes; blocks with no larger-font predecessor become children of the pseudo root. **Node-Centric Subtree Extraction.** Rather than processing the entire tree at once, CMM extracts a local subtree $t\_i$ for each node $x\_i$ by running Breadth First Search to depth $n\_d$ (set to 2), capturing parent, sibling, and child nodes. This limits GPU memory footprint to a fixed-size neighbourhood regardless of document length, and allows multiple subtrees to be batched simultaneously. **Node Encoding.** Each block $x\_i$ is encoded by concatenating a pretrained language model representation (RoBERTa-base) with a handcrafted feature vector $f\_i$ covering page number, font and font size, colour (RGB), text line count and length, and bounding box coordinates: $$b\_i = \text{MLP}([\text{TextEncoder}(x\_i),\ f\_i])$$ A one-layer bidirectional GRU is then applied to the nodes of $t\_i$ in in-order traversal to incorporate sequential reading-order context: $$\\{v\_j\\}\_{t\_1}^{t\_l} = \text{GRU}(t\_i)$$ **Graph Attention over the Subtree.** The subtree is converted to a graph $G = (V, \mathcal{E})$ with nodes connected by parent, child, and sibling edges. Edge features $f\_{j,i}$ encode edge type, font size difference $s\_j - s\_i$, font/colour match, page difference, and bounding box position difference. Graph Attention Networks (GAT) with $n\_d$ layers propagate information across the graph: $$\\{h\_i\\}^{t\_i} = \text{GAT}(V, \mathcal{E})$$ **Tree Modification.** For each node $x\_i$, three operation scores are computed from the final GAT representation $h\_i$: $$\begin{aligned} o\_i^{[kp]} &= W\_{kp} h\_i + b\_{kp} \\\\ o\_i^{[de]} &= W\_{de} h\_i + b\_{de} \\\\ o\_i^{[mv]} &= W\_{mv}[\text{POOL}(\text{PRS}(h\_i)),\ h\_i] + b\_{mv} \end{aligned}$$ The Move score incorporates a max-pooled representation of all preceding sibling nodes $\text{PRS}(h\_i)$, allowing the model to compare a node against its already-seen siblings before deciding whether it should be re-parented. Operations are selected via: $$\hat{y}\_i = \text{argmax}(p\_i)$$ where $p\_i$ is the vector of softmax probabilities over the three operation scores. After all per-node predictions are made, deletions are executed first, then Move operations are applied in reading order: a moved node's parent is set to its immediately preceding sibling. Training uses cross-entropy loss over the three-class operation labels. Ground-truth labels are derived from the annotated ToC: non-headings receive Delete; headings with a higher-level heading preceding them in the document receive Move; remaining headings receive Keep. ## What experiments were performed? **Datasets.** Two benchmarks are used: ESGDoc (the new dataset) and HierDoc (the prior benchmark of arXiv papers). For ESGDoc, MTD cannot process documents longer than 50 pages due to memory constraints, so a filtered sub-dataset "ESGDoc (Partial)" of documents under 50 pages (274/40/78 train/dev/test) is constructed for fair MTD comparison. CMM is evaluated on the full ESGDoc (765/110/218 train/dev/test) as well. **Baseline.** MTD (Hu et al., 2022) is the only baseline. It fuses text, visual, and layout modalities using a pretrained language model for heading classification, then models all pairwise heading relationships with GRU and attention to decode the tree. The paper cites earlier rule-based and ML methods (Namboodiri and Jain, 2007; Tuarob et al., 2015) but does not compare against them, which is a gap: their inclusion would help calibrate how much the neural approach adds over classical heuristics on HierDoc where both models already score above 87 TEDS. **Metrics.** Heading Detection (HD) is measured by F1-score, capturing how well the model identifies which blocks are headings. ToC quality is measured by Tree Edit Distance Similarity (TEDS): $$\text{TEDS}(T\_p, T\_g) = 1 - \frac{\text{TreeEditDist}(T\_p, T\_g)}{\max(|T\_p|, |T\_g|)}$$ **Ablations.** Four conditions are compared: (1) full CMM, (2) page-based division instead of tree-based subtree segmentation, (3) CMM without GRU, (4) CMM without GNN. A hyperparameter sweep over $n\_d \in \\{1, 2, 3, 4\\}$ is also reported. **Implementation.** RoBERTa-base is the text encoder. BFS depth $n\_d = 2$. Hidden size for $b$, $v$, and $h$ is 128. Training uses Adam with learning rate $10^{-5}$ for pretrained parameters and $10^{-3}$ for randomly initialized parameters, batch size 32, on a single NVIDIA A100 80GB GPU. ## What are the outcomes/conclusions? **Main results.** On HierDoc, CMM slightly outperforms MTD (HD: 97.0 vs. 96.1 F1; ToC: 88.1 vs. 87.2 TEDS). On ESGDoc (Partial), CMM substantially outperforms MTD (HD: 53.2 vs. 40.4; ToC: 30.0 vs. 26.9). On ESGDoc (Full), MTD achieves only 12.7/12.8 HD/ToC due to out-of-memory failures on long documents; CMM achieves 55.6/33.2. The gap on ToC is smaller than on HD, which the authors attribute to assumption violations (see below). **Runtime.** CMM trains 4.6$\times$ faster than MTD on HierDoc and 2.1$\times$ faster on ESGDoc (Partial). Inference is 4.2$\times$ and 1.3$\times$ faster, respectively. The smaller inference gap on ESGDoc is attributed to the higher number of graph edges in ESGDoc, which contains many small numeric text blocks that become individual nodes. **Ablation findings.** Removing the GNN causes the largest performance drop: 0.6 points on HierDoc and 11.6 points on ESGDoc (HD F1). Removing GRU costs 0.2 and 4.8 points respectively. Replacing tree-based segmentation with page-based windowing (6-page window, 2-page overlap) costs 0.3 points on HierDoc and 3.3 points on ESGDoc. All components matter more on ESGDoc, confirming that the GNN's ability to capture long-distance relationships is particularly important when heading hierarchy cannot be inferred from heading text alone. BFS depth $n\_d = 2$ offers the best accuracy-efficiency tradeoff; gains plateau beyond $n\_d = 2$. **Assumption violations.** CMM relies on three assumptions: (1) reading order is left-to-right, top-to-bottom; (2) higher-level headings have larger or equal font size; (3) headings at the same level share the same font size. In ESGDoc, 10.8% of headings violate at least one assumption, versus 4.6% in HierDoc. Violations arise from decorative typography, emphasis styling, and XY-Cut errors. These cases are the primary source of ToC prediction errors, since the initial tree construction may not position these headings correctly. **Limitations acknowledged.** CMM requires font size extraction from PDFs; scanned or photographed documents need an additional OCR step to recover font information. The model cannot recover from tree construction errors caused by XY-Cut failures or assumption violations. The paper also notes that CMM's advantage over MTD is smaller on the ToC metric than on HD, partly because assumption violations propagate into the modification step even when heading detection is correct. ## Reproducibility ### Models - Text encoder: RoBERTa-base (12-layer, 768-dimensional hidden, 125M parameters) - GRU: one-layer bidirectional; hidden size 128 - GAT: $n\_d = 2$ layers; hidden size 128; uses edge embeddings in addition to node embeddings (following Brody et al., 2021 formulation of GATv2) - MLP for block encoding: maps from [RoBERTa hidden + feature dim] to 128 - Output heads: three separate linear layers (Keep, Delete, Move) from 128-dimensional $h\_i$; Move additionally uses a max-pooled preceding-sibling representation concatenated before the linear layer - No pretrained weights are reported as released; the code repository has no license file (all rights reserved by default) ### Algorithms - Optimizer: Adam - Learning rate: $10^{-5}$ for RoBERTa parameters, $10^{-3}$ for randomly initialized parameters - Batch size: 32 - Training epochs/steps: not specified - Loss: cross-entropy over three operation classes (Keep, Delete, Move) - No warmup schedule, gradient clipping, or mixed precision details are mentioned - BFS depth $n\_d = 2$; texts with very small sizes are deleted automatically during tree modification ### Data - ESGDoc: 1,093 ESG annual reports from 563 companies (2001-2022), sourced from ResponsibilityReports.com; split 765/110/218 (train/dev/test) - Documents range from 4 to 521 pages; average 72 pages - Text extracted using PyMuPDF; annotations derived by using the embedded ToC of each PDF as ground truth (documents without an embedded ToC were excluded from the original 10,639 downloads) - ESGDoc is not directly redistributed due to copyright concerns over the original PDFs; the GitHub repository provides metadata and download scripts (via Google Drive) that re-crawl reports from ResponsibilityReports.com, so reproducibility depends on that site remaining accessible and the download links staying live - No explicit license is stated for the ESGDoc metadata or annotations in the paper or code repository - HierDoc: 650 arXiv papers (350/300 train/test), from Hu et al. (2022); font sizes re-extracted from PDFs using PyMuPDF since the HierDoc release does not include them ### Evaluation - HD metric: F1-score over heading block identification - ToC metric: TEDS (tree edit distance similarity) averaged over all documents in the test set - ESGDoc (Partial) used for MTD comparison to avoid out-of-memory exclusions affecting MTD - No error bars, significance tests, or multiple-run seeds are reported - Assumption violation statistics (Table 1) are computed automatically from the labeled data, not from human inspection of every case ### Hardware - Training hardware: single NVIDIA A100 80GB GPU - Training time: 525.4 minutes on HierDoc, 241.4 minutes on ESGDoc (Partial) for CMM; MTD requires 2420.6 and 513.5 minutes respectively - Inference time: 3.8 minutes on HierDoc, 2.1 minutes on ESGDoc (Partial) for CMM - No CPU-only or deployment considerations reported ## BibTeX ```bibtex @inproceedings{wang-etal-2023-cmm, title = "A Scalable Framework for Table of Contents Extraction from Complex {ESG} Annual Reports", author = "Wang, Xinyu and Gui, Lin and He, Yulan", booktitle = "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing", month = dec, year = "2023", address = "Singapore", publisher = "Association for Computational Linguistics", pages = "13215--13229", doi = "10.18653/v1/2023.emnlp-main.816" } ```