# Capturing Logical Structure of Visually Structured Documents with Multimodal Transition Parser

> A feature-based transition parser that fuses visual, textual, and semantic cues to recover the hierarchical logical structure of PDFs and other visually structured documents, substantially outperforming rule-based and heuristic baselines on legal documents.

- Authors: Bevaya AI Team
- Published: 2026-04-02
- Paper: Preprint (arXiv) <https://arxiv.org/abs/2105.00150>
- Paper: Published (NLLP@EMNLP 2021) <https://doi.org/10.18653/v1/2021.nllp-1.15>
- Code: pdf-struct <https://github.com/stanfordnlp/pdf-struct>
- Dataset: pdf-struct dataset <https://stanfordnlp.github.io/pdf-struct-dataset/>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## TL;DR

This paper proposes a transition-label formulation for recovering the logical hierarchy of visually structured documents (VSDs) such as PDFs. Given a sequence of text blocks extracted by existing layout tools, a Random Forest classifier predicts one of five transition labels between each consecutive pair of blocks, building a tree that captures paragraph boundaries and their nesting. The multimodal feature set (visual, textual, and semantic cues including a GPT-2 coherence signal) substantially outperforms visual-only and numbering-only baselines, achieving a paragraph boundary detection F1 of 0.953 on English NDAs compared to 0.739 for PDFMiner.

## What kind of paper is this?

**Dominant:** $\Psi\_{\text{Method}}$: The primary contribution is a complete algorithmic system, the transition parser with its multimodal feature set, formulation, and two-pass decoding procedure. The paper is organized around algorithm description, ablation-style feature importance analysis, and cross-dataset baseline comparisons.

**Secondary:** $\Psi\_{\text{Resource}}$: The authors release both the code (pdf-struct, Apache-2.0) and the annotated dataset covering four document types (English and Japanese NDAs in PDF and plain text, plus English executive orders in PDF). The dataset and system together constitute a reusable infrastructure contribution for legal document preprocessing research.

## What is the motivation?

Most NLP pipelines assume clean, linear text, but a substantial fraction of real-world documents are visually structured: PDFs produced by word processors or scanners, plain text with indentation-based formatting, and so on. Existing preprocessing tools (PDFMiner, pdftotext, etc.) concentrate on two extremes: word-level segmentation and coarse layout analysis (figure/body/table detection). Neither step recovers the logical hierarchy inside the body text: which lines belong to the same paragraph, which paragraphs are children of a section header, which blocks are debris (page numbers, running headers) that should be discarded.

This gap matters especially in the legal domain. Contracts and executive orders are heavily hierarchical: a confidentiality obligation may be defined as a numbered clause containing lettered sub-clauses, and extracting the definition requires understanding that hierarchy. The best available PDF-to-text tool at the time of publication had a 17% newline detection error rate (Bast and Korzen, 2017), and none of these tools could recover hierarchy at all.

The paper targets single-column VSDs, for which coarse layout analysis is straightforward, and focuses exclusively on the logical structure within the body text.

## What is the novelty?

### Transition label formulation

The core insight is to reduce tree construction to a local, sequential classification problem. Given text blocks $b\_1, b\_2, \ldots, b\_n$ in document order, the system predicts a transition label $trans\_i$ for each consecutive pair $(b\_i, b\_{i+1})$:

$$trans\_i \in \\{\texttt{continuous},\, \texttt{consecutive},\, \texttt{down},\, \texttt{up},\, \texttt{omitted}\\}$$

- **continuous**: $b\_{i+1}$ continues the same paragraph as $b\_i$.
- **consecutive**: $b\_{i+1}$ starts a new sibling paragraph at the same level.
- **down**: $b\_{i+1}$ starts a new paragraph that is a child of $b\_i$'s paragraph (one level deeper).
- **up**: $b\_{i+1}$ returns to an ancestor level; accompanied by a pointer $ptr\_i = b\_j$ that identifies which prior block's level $b\_{i+1}$ rejoins.
- **omitted**: $b\_i$ is debris (a page number, running header, etc.) and is dropped; $trans\_{i-1}$ is carried over to bridge $b\_{i-1}$ and $b\_{i+1}$.

The five labels are sufficient to reconstruct the full tree. Because **up** can be ambiguous about how many levels to ascend, the authors introduce a separate pointer identification module (Section 4.2) that selects among all candidate **down** blocks preceding $b\_i$.

### Two-pass decoding

Because the **omitted** label changes which blocks participate in subsequent feature extraction, the system runs a first pass dedicated to detecting **omitted** blocks, then a second pass that extracts features from the correct four-block context window $[b\_{i-1}, b\_i, b\_j, b\_{j+1}]$ (where $b\_j$ is the next non-omitted block after $b\_i$) to predict the remaining four labels.

### Multimodal feature set

The classifier receives hand-crafted features spanning three modalities (Table 2 in the paper lists 12 visual features V1-V12, 12 textual features T1-T12, and 1 semantic feature S1):

- **Visual**: indentation relative to adjacent blocks, indentation after stripping numbering prefixes, centering, line break before the estimated right margin, page transitions, header/footer zone membership, line spacing relative to the modal spacing, justification with internal spaces, positionally repeated text (header/footer detection via edit distance across pages), and emphasis via spaced characters or parentheses.
- **Textual**: a heuristic numbering-transition sub-parser (T1) that tracks numbering type memories to output a predicted transition as a feature, terminal punctuation, list-start and list-element patterns, strict and tolerant page-number regexes, domain-specific legal phrasing triggers ("whereas", "now, therefore"), dictionary-like formatting, all-capitals, blank fill-in fields, and horizontal rule characters.
- **Semantic**: a GPT-2 language model coherence signal (S1) computed as $l(i, i-1) - l(i+1, i-1)$, where $l(a, b)$ is the language model loss for block $b\_a$ given block $b\_b$ as context, with a second variant $l(i+1, i) - l(i+1, i-1)$. A more negative value indicates $b\_i$ is more coherent after $b\_{i-1}$ than $b\_{i+1}$ would be, suggesting $b\_i$ is not debris.

### Pointer identification

For **up** transitions, the pointer module selects among candidate blocks $C\_i = \\{b\_j : trans\_j = \texttt{down}, j < i\\}$. Features for each candidate pair $(b\_j, b\_i)$ include: consecutive numbering between $b\_i$ and $b\_j$ or the paragraph head $b\_{head(j)}$, indentation changes, left-alignment flags, and counts of intervening **down** and **up** transitions. A binary Random Forest classifier scores each candidate and the argmax is taken.

### Customizable design

Feature extractors are implemented as Python classes with decorator-annotated methods (`@single_input_feature`, `@pairwise_feature`, `@pointer_feature`). New document types can subclass an existing extractor and override or extend individual features. The Contract$^{pdf}$ and Contract$^{jaf}$ (Japanese) extractors both inherit from a base PDF extractor, with Japanese-specific adaptations (e.g., a Japanese-language GPT-2 model and Japanese numbering detection).

## What experiments were performed?

### Datasets

Four document types are evaluated, all annotated by a single author:

| Type | Format | Language | Documents | Avg. blocks/doc |
|:-----|:-------|:---------|----------:|----------------:|
| Contract$^{pdf}$ | PDF | English | 40 | 137.9 |
| Law$^{pdf}$ | PDF | English | 40 | 165.9 |
| Contract$^{txt}$ | Plain text | English | 22 | 142.0 |
| Contract$^{jaf}$ | PDF | Japanese | 40 | 73.7 |

PDFs were downloaded from Google search results targeting NDAs and executive orders. Plain-text NDAs were retrieved from SEC EDGAR. All documents are single-column. Blocks were extracted using PDFMiner (`LTTextLine` objects with vertical overlap merging) for PDFs, and individual non-blank lines for plain text. Evaluation used five-fold cross-validation.

### Evaluation metrics

The authors report two families of metrics:

1. **Information extraction (IE) perspective**: For each pair of blocks, classify their pairwise relationship as same-paragraph, sibling, or ancestor-descendant. Precision, recall, and F1 are computed for each relationship type, then micro- and macro-averaged.
2. **Preprocessing perspective**: Paragraph boundary detection F1 (both strict and lenient variants), and F1 for **omitted** block elimination (debris removal). PDFMiner is additionally included as a baseline for the boundary task.

### Baselines

- **Numbering baseline** (Hatsutori et al., 2017): uses regular-expression-based numbering detection and type changes to infer hierarchy. Equivalent to the T1 feature used in isolation.
- **Visual baseline**: predicts from indentation direction and line spacing only.
- **PDFMiner**: purely geometric heuristics for paragraph break detection; available only for PDF inputs.

### Classifier

Random Forest (scikit-learn defaults, no hyperparameter tuning) trained separately for each document type. GPT-2 medium is used for English; `rinna/japanese-gpt2-medium` for Japanese. The paper explicitly cites the choice of Random Forest as being well suited to the predominantly categorical feature set and the small training set sizes typical in the legal domain.

## What are the outcomes/conclusions?

### Quantitative results

On the IE perspective metrics, the system achieves micro-average F1 of:

| Type | Visual | Numbering | Ours |
|:-----|-------:|----------:|-----:|
| Contract$^{pdf}$ | 0.772 | 0.778 | **0.914** |
| Law$^{pdf}$ | 0.827 | 0.685 | **0.908** |
| Contract$^{txt}$ | 0.587 | 0.674 | **0.828** |
| Contract$^{jaf}$ | 0.618 | 0.623 | **0.940** |

On paragraph boundary detection (micro-average F1):

| Type | PDFMiner | Visual | Numbering | Ours |
|:-----|:--------:|:------:|:---------:|-----:|
| Contract$^{pdf}$ | 0.739 | 0.712 | 0.793 | **0.953** |
| Law$^{pdf}$ | 0.667 | 0.676 | 0.750 | **0.948** |
| Contract$^{txt}$ | -- | 0.633 | 0.702 | **0.950** |
| Contract$^{jaf}$ | 0.653 | 0.632 | 0.759 | **0.980** |

Transition label prediction accuracy (micro-average) ranges from 0.923 to 0.955 across document types. The system outperforms all baselines on all but one relationship type across all four document types.

### Feature importance

Greedy forward selection and backward elimination (Table 5) reveal that the system makes balanced use of both visual and textual cues. Indentation (V1), larger line spacing (V8), and numbering hierarchy (T1) rank highly, as do all-capitals (T10) and terminal punctuation (T2). The semantic GPT-2 feature (S1) ranks lower than most visual and textual features, which the authors attribute to the model not being fine-tuned on legal text, causing the coherence signal to sometimes ignore context.

### Error analysis

Two systematic error modes are identified: (1) for Contract$^{pdf}$, documents with bold or underlined section headers followed by unindented paragraphs confuse the system into predicting **continuous** instead of **down**; typographic features (bold, underline detection) would address this. (2) For Contract$^{jaf}$, all-capital blocks and underbar sequences that function as section titles or input fields are misclassified as **omitted**; the authors attribute this to insufficient training data for those patterns rather than a feature design failure.

The authors note an interesting finding: the system tends to perform better on hierarchically more complex documents, possibly because such documents use more varied visual and textual cues to signal structure, giving the classifier more signal to work from.

### Limitations

- Single-column documents only. Multi-column layouts require prior layout analysis that is out of scope.
- Single annotator, no inter-annotator agreement measurement.
- Small dataset sizes (22-40 documents per type); results are cross-validated but no significance tests or confidence intervals are reported.
- The GPT-2 semantic feature performs weakly due to domain mismatch; future work could use legal domain-adapted language models.
- Typographic features (bold, italic, underline) are not included despite evidence they would help.
- The system processes blocks in linear order: it cannot revisit earlier predictions. Errors in the first (**omitted** detection) pass propagate to the second pass.

## Reproducibility

### Models

- The transition and pointer classifiers are Random Forest models using scikit-learn 0.x default hyperparameters. No model weights are needed beyond what scikit-learn trains from scratch.
- The semantic feature requires GPT-2 medium (English) and `rinna/japanese-gpt2-medium` (Japanese), both available on Hugging Face. These are used for inference only (language model loss computation), not fine-tuned.
- No pretrained weights specific to this paper are released.

### Algorithms

- Two-pass classification: first pass predicts **omitted** labels; second pass uses corrected context windows to predict the four structural labels.
- Pointer identification: among all candidate **down** blocks, a separate binary Random Forest selects the argmax probability prediction.
- Feature extraction is block-level with a four-block context window. The right margin is estimated per-document by 1D clustering on block right-edge positions (minimum 6 members per page). Normal line spacing is estimated by 1D clustering on inter-block spacings (largest cluster).
- GPT-2 coherence feature uses log-loss differences; no gradient or fine-tuning involved.
- All implementation details are open-sourced at [github.com/stanfordnlp/pdf-struct](https://github.com/stanfordnlp/pdf-struct).

### Data

- The annotated dataset (142 documents across four types) is released at a dedicated page ([stanfordnlp.github.io/pdf-struct-dataset/](https://stanfordnlp.github.io/pdf-struct-dataset/)) under CC-BY-4.0, separate from the code repository (which is Apache-2.0).
- PDFs were collected from Google search results; plain-text NDAs from SEC EDGAR. No license is specified for the source documents themselves, which are legal contracts and government orders from various jurisdictions.
- Annotation was performed by one author with no majority vote. The authors argue that labels are unambiguous enough that single-annotator annotation is reliable, but no inter-annotator agreement metric is reported.

### Evaluation

- Five-fold cross-validation throughout.
- IE perspective: pairwise relationship matrices compared for same-paragraph, sibling, and ancestor-descendant categories; precision, recall, and F1 reported per category plus micro/macro averages.
- Preprocessing perspective: paragraph boundary F1 and **omitted** block elimination F1.
- Transition label accuracy (micro) also reported as a diagnostic.
- No error bars or significance tests reported.
- PDFMiner baseline is applicable only to PDF document types; not included for Contract$^{txt}$.

### Hardware

- Training and inference used ABCI (AI Bridging Cloud Infrastructure) provided by Japan's AIST. Specific GPU types and counts are not reported.
- Random Forest training on small datasets (22-40 documents) is fast; the main computational cost is GPT-2 inference for the S1 feature.
- The system is CPU-feasible for both training and inference given the classical ML classifier; GPU acceleration is needed only for the GPT-2 feature.

## BibTeX

```bibtex
@inproceedings{koreeda-manning-2021-capturing,
  title     = {Capturing Logical Structure of Visually Structured Documents with Multimodal Transition Parser},
  author    = {Koreeda, Yuta and Manning, Christopher D.},
  booktitle = {Proceedings of the Natural Legal Language Processing Workshop 2021},
  month     = nov,
  year      = {2021},
  address   = {Punta Cana, Dominican Republic},
  publisher = {Association for Computational Linguistics},
  pages     = {144--154},
  doi       = {10.18653/v1/2021.nllp-1.15},
  url       = {https://aclanthology.org/2021.nllp-1.15},
}
```