# Reading Order Matters: Information Extraction from Visually-rich Documents by Token Path Prediction > Token Path Prediction (TPP) reformulates VrD-NER as predicting directed token paths in a graph, sidestepping the reading order dependency of BIO-tagging. The paper also releases FUNSD-r and CORD-r, two revised benchmarks that reflect realistic OCR-induced reading order disorder. - Authors: Hunter Heidenreich - Published: 2026-04-02 - Paper: Preprint (arXiv) - Paper: Published (EMNLP 2023) - Dataset: FUNSD-r / CORD-r (datasets) - Code: TPP (code) - Source: Bevaya Labs, --- ## TL;DR BIO-tagging for named entity recognition assumes that entity tokens appear in a clean, contiguous, front-to-back order in the model input. On scanned documents, OCR-driven input order routinely violates this assumption. Token Path Prediction (TPP) avoids the assumption entirely by modeling entities as directed paths in a complete token graph, making it robust to disordered OCR inputs without requiring any pre-computed reading order. The authors also release FUNSD-r and CORD-r, revised versions of popular NER benchmarks that better reflect the disordered layouts seen in practice. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Method}}$: The central contribution is TPP, a new prediction head that replaces the token classification head used by sequence-labeling NER models. The paper includes ablations, SOTA comparisons on multiple tasks, and a case study, all focused on demonstrating the method's effectiveness. **Secondary:** $\Psi\_{\text{Resource}}$: FUNSD-r and CORD-r are re-annotation efforts requiring manual word-level entity labeling on OCR-relaid layouts, released as community benchmarks. They also expose a data quality problem in the original FUNSD and CORD annotations that affects fair evaluation. ## What is the motivation? Information extraction from visually-rich documents (VrDs) such as forms, receipts, and contracts typically relies on pre-trained document transformers (LayoutLM family, BROS, LayoutMask) fine-tuned with a BIO token classification head for named entity recognition. The BIO scheme requires that each entity spans a contiguous, correctly ordered slice of the input token sequence. In practice, OCR systems processing scanned documents read top-to-bottom, left-to-right and produce segment-level annotations that need not align with logical entity boundaries. Complex layouts, including multi-column text, tables, and elements that span multiple rows, cause OCR outputs to interleave tokens from different entities or split a single entity across non-adjacent positions. The BIO tagging scheme cannot represent such entities correctly, regardless of how powerful the underlying encoder is. The authors note a secondary problem: the standard FUNSD and CORD benchmarks have segment annotations manually aligned with entity boundaries, so each entity always maps to a contiguous span in the input. Models trained and evaluated on these benchmarks are never tested on the disorder that appears in real scanned documents. This makes existing leaderboard numbers misleading as a measure of real-world capability. ## What is the novelty? ### Token Path Prediction (TPP) Rather than tagging each input token with a BIO label, TPP treats entity extraction as a graph path prediction problem. A visually-rich document with $N\_D$ words is represented as: $$D = \\{(w\_i, b\_i)\\}\_{i=1,\ldots,N\_D}$$ where $w\_i$ is the $i$-th word and $b\_i = (x\_i^0, y\_i^0, x\_i^1, y\_i^1)$ is its bounding box. An entity of type $e\_j$ is a word sequence $(w\_{j\_1}, \ldots, w\_{j\_k})$ where the words need not be adjacent in the input order. TPP constructs a complete directed graph over all $N\_D$ tokens. For each entity type, the token paths that form entities are encoded as an $n \times n$ binary grid label, where entry $(i, j)$ is 1 if token $w\_i$ is immediately followed by token $w\_j$ in any entity of that type. This transforms NER into $N\_e$ independent binary classification problems on $n^2$ token-pair samples. The classifier is implemented using Global Pointer (Su et al., 2022), which computes a bilinear score for each directed token pair: $$s(w\_i \to w\_j) = \text{GlobalPointer}(h\_i, h\_j)$$ Training uses the class-imbalance loss from Global Pointer (Su et al., 2022). For each entity type grid, let $\mathcal{P}$ be the set of positive token pairs and $\mathcal{N}$ the set of negative pairs. The loss penalizes positive pairs with low score and negative pairs with high score: $$\mathcal{L} = \log\!\left(1 + \sum\_{(i,j)\in\mathcal{P}} e^{-s(w\_i \to w\_j)}\right) + \log\!\left(1 + \sum\_{(i,j)\in\mathcal{N}} e^{s(w\_i \to w\_j)}\right)$$ This formulation handles the extreme class imbalance naturally, since at most $n$ of the $n^2$ pairs per grid are positive. During inference, positive-scored pairs are collected per entity type, and a greedy search decodes them into token paths (entity mentions), keeping the highest-confidence successor when multiple outgoing edges exist from the same token. Two enhancements improve performance over the base design: - **Positional residual linking**: adds a residual connection from the backbone's 1D positional embeddings into the GlobalPointer inputs, strengthening the ordering signal available to the classifier. - **Multi-dropout**: duplicates the final fully-connected layer and trains each copy with a different dropout mask while sharing weights, following Inoue (2019), to improve robustness. ### Adaptation to other VrD tasks The same grid-label design is adapted to entity linking (VrD-EL) by marking all token pairs between linked entity spans, and to reading order prediction (VrD-ROP) by predicting a global path over all tokens starting from a special `` token, decoded via beam search. ### FUNSD-r and CORD-r The revised benchmarks are re-annotated from the original FUNSD (199 documents) and CORD (999 documents) images using an automatic OCR pass with PP-OCRv3 for layout re-annotation, followed by manual re-labeling of entity mentions as word sequences on the new layouts. Key differences from the originals: (1) every segment is constrained to a single row, matching real OCR behavior; (2) entity boundaries are annotated at the word level and are not forced to align with segment boundaries; (3) entities may span multiple segments (and hence be discontinuous in OCR-order input). The continuous entity rate, the fraction of entities that form a contiguous span in OCR-order input, is reported as a diagnostic: FUNSD-r has 95.74% and CORD-r has 92.10% when no reordering is applied. ## What experiments were performed? ### VrD-NER on FUNSD-r and CORD-r Two document transformer backbones are evaluated: LayoutLMv3-base (vision + text + layout) and LayoutMask (text + layout only). For each backbone, three conditions are compared: 1. **Sequence labeling, no pre-processing**: vanilla BIO tagging on OCR-ordered input. 2. **Sequence labeling + reordering pre-processing**: inputs are reordered by either LayoutReader or TPP-for-VrD-ROP before BIO tagging. 3. **TPP-for-VrD-NER**: graph-based prediction with no reading order assumption. The primary metric is entity-level F1 (strict boundary + type match), which the authors argue is more meaningful than the word-level F1 used in prior FUNSD/CORD work. Results from Table 1, including the continuous entity rate (Cont.) which measures how well each pre-processing step restores contiguous entity spans in the input: **FUNSD-r** | Backbone | Method | Pre-processing | Cont. (%) | F1 | |:---|:---|:---|:---|:---| | LayoutLMv3 | Seq. Labeling | None | 95.74 | 78.77 | | LayoutLMv3 | Seq. Labeling | LayoutReader | 95.53 | 78.37 | | LayoutLMv3 | Seq. Labeling | TPPR | **97.29** | 79.72 | | LayoutLMv3 | **TPP** | None | N/A | **80.40** | | LayoutMask | Seq. Labeling | None | 95.74 | 77.10 | | LayoutMask | Seq. Labeling | LayoutReader | 95.53 | 77.24 | | LayoutMask | Seq. Labeling | TPPR | **97.29** | **80.70** | | LayoutMask | **TPP** | None | N/A | 78.19 | **CORD-r** | Backbone | Method | Pre-processing | Cont. (%) | F1 | |:---|:---|:---|:---|:---| | LayoutLMv3 | Seq. Labeling | None | 92.10 | 82.72 | | LayoutLMv3 | Seq. Labeling | LayoutReader | 82.10 | 70.33 | | LayoutLMv3 | Seq. Labeling | TPPC | **92.43** | 83.24 | | LayoutLMv3 | **TPP** | None | N/A | **91.85** | | LayoutMask | Seq. Labeling | None | 92.10 | 81.84 | | LayoutMask | Seq. Labeling | LayoutReader | 82.10 | 68.05 | | LayoutMask | Seq. Labeling | TPPC | **92.43** | 81.90 | | LayoutMask | **TPP** | None | N/A | **89.34** | TPPR/TPPC denotes TPP-for-VrD-ROP trained on ReadingBank/CORD respectively. On CORD-r, LayoutReader is trained on ReadingBank, which uses column-by-column reading order; the CORD documents use row-by-row order. This domain mismatch causes LayoutReader to actively reduce the continuous entity rate from 92.10% to 82.10%, which translates into large F1 drops (+12 and +14 points below baseline for LayoutLMv3 and LayoutMask respectively). This is the strongest quantitative evidence against using generic reading order models as pre-processing. The CORD-r F1 gains for TPP itself are also large: +9.13 and +7.50 points over vanilla sequence labeling with LayoutLMv3 and LayoutMask respectively, with no pre-processing step at all. LayoutReader used as a pre-processing step either fails to improve or actively degrades performance on both benchmarks. The authors attribute this to LayoutReader's seq2seq decoding, which can produce duplicate or missing tokens and is optimized for BLEU on ReadingBank rather than for segment-level entity continuity. ### VrD-EL and VrD-ROP TPP is also evaluated on entity linking (FUNSD, F1) and reading order prediction (ReadingBank, BLEU and ARD). The authors report that TPP outperforms MSAU-PAF on entity linking (FUNSD, F1) and LayoutReader on reading order prediction (ReadingBank) in five of six BLEU/ARD settings. ### Ablation studies Ablations on positional encoding choices and TPP design components (Table 5a/5b) show that: - Using global 1D positional encoding alongside segment-level 2D positions gives the best result for LayoutLMv3 on FUNSD-r. - Both positional residual linking and multi-dropout contribute positively; their combined removal causes a drop of up to 12.66 F1 points on FUNSD-r with LayoutMask. ## What are the outcomes/conclusions? TPP consistently outperforms BIO-based sequence labeling on both new benchmarks, suggesting that order-invariant entity extraction is better suited to real-world scanned VrDs. As a secondary use mode, TPP-for-VrD-ROP also serves as a better pre-processing reordering mechanism than LayoutReader, as measured by the continuous entity rate after reordering. The case study (Figure 6) illustrates qualitative strengths: TPP correctly identifies multi-row, multi-column, and long entities that sequence-labeling methods fragment or miss. A notable failure mode is entity type misclassification: TPP sometimes finds the correct tokens but assigns the wrong entity type label, suggesting that layout features dominate text signals in some cases. The authors acknowledge two limitations. First, the $n^2$ grid label causes quadratic memory and compute scaling: training with TPP requires 2.45-2.75x more time and 1.47-1.56x more peak memory than vanilla token classification. This cost is manageable for fine-tuning on datasets of a few hundred to a few thousand samples but would be more problematic for large-scale pre-training. Second, the revised benchmarks cover English only, and similar disorder-reflecting benchmarks are not yet available for other languages or for IE tasks beyond NER. ## Reproducibility ### Models - **Architecture**: Document transformer backbone (LayoutLMv3-base or LayoutMask) with a GlobalPointer classification head per entity type. Backbone parameter counts follow the original papers (LayoutLMv3-base: 125M parameters). - **Weights**: Not released. The GitHub repository (https://github.com/chongzhangFDU/TPP) contains only a README; the authors note that open-sourcing of the code is pending approval from Ant Group. No pre-trained checkpoints are available. - **Backbone initialization**: Fine-tuned from publicly available LayoutLMv3-base and LayoutMask checkpoints. ### Algorithms - **Optimizer**: Adam with 1% linear warm-up and weight decay of 1e-5, dropout 0.1. - **Learning rate**: Grid-searched over {3e-5, 5e-5, 8e-5}; best values are backbone- and dataset-specific (e.g., 5e-5 / 5e-5 for LayoutLMv3 / LayoutMask on FUNSD; 3e-5 / 8e-5 on CORD). - **Batch size**: 16. - **Steps**: 1,000 on FUNSD-r; 2,500 on CORD-r. For VrD-ROP: 100 epochs on ReadingBank. - **Loss**: Class-imbalance loss (from Global Pointer, Su et al., 2022) applied to the $n^2$ binary classification problem per entity type. - **Decoding**: Greedy path search for NER/EL; beam search (beam size 8) for ROP. - **Maximum entities decoded**: 100. ### Data - **Training data**: FUNSD-r (199 documents) and CORD-r (999 documents), re-annotated from the original FUNSD and CORD images using PP-OCRv3 for layout and manual annotation for entity mentions. - **Splits**: Inherited from original FUNSD and CORD train/val/test splits after filtering for OCR confidence (> 0.8) and minimum document length (> 20 valid words). - **Availability**: Released in a separate GitHub repository (https://github.com/chongzhangFDU/Token-Path-Prediction-Datasets) under CC-BY-4.0. Note that the TPP code itself has not been released; the dataset repo is independent of the code repo. - **Annotation**: Semi-automatic (automated OCR re-annotation + manual entity labeling). No inter-annotator agreement figure is reported. - **Contamination**: Not discussed. The revised datasets derive from the same document images as the originals, so any overlap with pre-training data for LayoutLMv3 or LayoutMask is inherited. ### Evaluation - **Primary metric**: Entity-level F1 (exact match on entity type and all constituent tokens). This is stricter than the word-level F1 used in most prior FUNSD/CORD work. - **Baselines**: Sequence labeling with LayoutLMv3-base and LayoutMask; LayoutReader as pre-processing. Comparisons use the same backbones and fine-tuning budget, which is a reasonable basis for comparison. - **Limitations acknowledged**: Quadratic complexity; English-only benchmarks; no other-language or non-NER IE tasks evaluated with disorder-reflecting benchmarks. - **Limitations not acknowledged**: No error bars or multiple runs reported. The continuous entity rate metric, used to evaluate pre-processing quality, measures a proxy (contiguous spans) rather than direct downstream F1 impact at token level. The 92-96% continuous entity rates in the revised datasets mean that a majority of entities are still contiguous even after re-annotation, limiting how much the benchmarks actually stress-test non-sequential methods. ### Hardware - **Training hardware**: 8 Tesla A100 GPUs. - **Training time**: 2.45x (LayoutLMv3) to 2.75x (LayoutMask) longer than vanilla token classification. - **Peak memory**: 1.47x (LayoutMask) to 1.56x (LayoutLMv3) higher than vanilla token classification. - **Inference hardware**: Not reported. ## BibTeX ```bibtex @inproceedings{zhang-etal-2023-reading, title = "Reading Order Matters: Information Extraction from Visually-rich Documents by Token Path Prediction", author = "Zhang, Chong and Guo, Ya and Tu, Yi and Chen, Huan and Tang, Jinyang and Zhu, Huijia and Zhang, Qi and Gui, Tao", booktitle = "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing", year = "2023", publisher = "Association for Computational Linguistics", doi = "10.18653/v1/2023.emnlp-main.846" } ```