# ROOR: Modeling Layout Reading Order as Ordering Relations > Analysis of ROOR (EMNLP 2024), which argues that representing reading order as a single permutation is lossy and proposes a directed acyclic relation formulation over layout elements, along with a new benchmark and a downstream enhancement pipeline. - Authors: Bevaya AI Team - Published: 2026-04-02 - Paper: Preprint (arXiv) - Code: Code (ROOR) - Dataset: ROOR Dataset - Source: Bevaya Labs, --- ## TL;DR ROOR argues that representing layout reading order as a single permutation of elements is inherently lossy for complex documents, and proposes instead to model reading order as a directed acyclic relation (Immediate Succession During Reading, ISDR) over layout elements. The authors introduce a 199-sample benchmark dataset with ISDR annotations built on EC-FUNSD, a LayoutLMv3-based relation extraction baseline, and a downstream enhancement pipeline that injects reading order relations into any layout-aware model. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Method}}$: The headline contribution is the reformulation of ROP as a relation extraction task and the accompanying relation-aware attention pipeline for enhancing downstream VrD tasks. The method is the core novelty; the dataset and benchmark are released to enable it. **Secondary:** $\Psi\_{\text{Resource}}$: Introduces the ROOR benchmark dataset (199 samples, 10,662 segments, 10,967 annotated reading order relation pairs built on EC-FUNSD). **Secondary:** $\Psi\_{\text{Evaluation}}$: Proposes a new task formulation that challenges the standard permutation-based evaluation setup, and uses F1 over relation pairs as the primary metric rather than BLEU/ARD. ## What is the motivation? Reading Order Prediction (ROP) for visually-rich document layouts has historically been framed as predicting a permutation of layout elements. The authors identify a structural problem with this framing: for complex document layouts (multi-column pages, sidebars, tables, floating elements), multiple valid reading orders exist simultaneously, and no single permutation can capture all of them without discarding information or injecting annotation noise. Concretely, they identify three failure modes of the permutation formulation: 1. A layout element can have multiple valid immediate successors (e.g., the last cell in a table row can be followed by the cell below or the first cell in the next row). 2. Some layout elements (headers, footers, watermarks) are logically disconnected from the main reading flow and cannot be cleanly inserted into a linear sequence. 3. Indirect reading order relationships (e.g., a question and its distant answer) are not captured by immediate succession alone. The downstream motivation is practical: prior work has shown reading order information improves VrD-IE and VrD-QA tasks, but this benefit is limited by the quality of the reading order representation. ## What is the novelty? ### Reformulation: ISDR as a Directed Acyclic Relation The paper defines two order-theoretic concepts: **Immediate Succession During Reading (ISDR):** A directed acyclic relation (DAR) over the set of layout elements. Given a document layout $D = \\{(w\_i, b\_i)\\}\_{i=1,\ldots,N\_D}$, the reading order is represented as a set of directed pairs: $$ P\_D = \\{(k\_s, k\_o) \mid w\_{k\_o} \text{ immediately follows } w\_{k\_s} \text{ during reading}\\} $$ This is a DAR (not a total order), which means a single element may have multiple successors, and disconnected subgraphs are permitted. **Generalized Succession During Reading (GSDR):** The transitive closure of ISDR, forming a strict partial order (SPO). GSDR captures indirect ordering relationships (e.g., a question element always precedes its distant answer). ### Relation Extraction Baseline ROP is reformulated as a relation extraction task. The model uses LayoutLMv3 as the backbone encoder, pools per-element token embeddings via average pooling: $$ h\_i = \text{AvgPool1d}(h\_i^1, \ldots, h\_i^{n\_i}) $$ Then applies a global pointer network for pair scoring: $$ \begin{aligned} q\_i &= W\_q h\_i + b\_q \\\\ k\_i &= W\_k h\_i + b\_k \\\\ s\_{ij} &= q\_i^T k\_j \end{aligned} $$ Training uses a class-imbalance loss: $$ \mathcal{L} = \log\!\left(1 + \sum\_{\substack{(i,j) \notin L}} e^{s\_{ij}}\right) + \log\!\left(1 + \sum\_{\substack{(i,j) \in L}} e^{-s\_{ij}}\right) $$ Inference extracts pairs above a zero threshold: $\hat{L} = \\{(i,j) \mid s\_{ij} > 0\\}$. ### Downstream Enhancement Pipeline (RORE) Reading order relations are encoded as an $n \times n$ binary matrix over input tokens and injected into downstream layout-aware models via a relation-aware attention module: $$ a\_{ij}^l = \frac{\exp\!\left(\left(q\_i^T k\_j^l + \lambda^l \rho\_{ij}\right) / \sqrt{d\_k}\right)}{\sum\_{j'} \exp\!\left(\left(q\_i^T k\_{j'}^l + \lambda^l \rho\_{ij'}\right) / \sqrt{d\_k}\right)} $$ where $\rho\_{ij}$ is the binary reading order relation matrix and $\lambda^l$ is a learnable per-layer scalar weight. ## What experiments were performed? ### Dataset ROOR is built on EC-FUNSD: 199 single-page scanned VrD samples, 10,662 segments, 31,297 words, 10,967 annotated ISDR relation pairs. Two domain experts annotated independently and resolved conflicts by discussion. 23.76% of segments are involved in non-linear reading order. ### ROP Baseline Evaluation (Table 1) Evaluated on word-level and segment-level ROP with F1 score. LayoutLMv3-large achieves 93.01 (word-level) and 82.38 (segment-level) F1. Human inter-annotator agreement is 99.28 (segment-level). Prior sequence-based methods (LayoutReader: 9.44 word-level F1; Token Path Prediction: 42.96 segment-level F1) perform dramatically worse, illustrating the incompatibility of sequence-based methods with the relation-based evaluation. ### Downstream Enhancement on EC-FUNSD (Table 2) Ground truth ISDR injected into LayoutLMv3-base/large and GeoLayoutLM for SER and EL tasks. EL benefits substantially more than SER (up to +6.17 F1 for LayoutLMv3-base). ### Cross-Domain VrD-IE with Pseudo Labels (Table 3) An off-the-shelf LayoutLMv3-large ROP model trained only on ROOR generates pseudo labels for FUNSD, CORD, and SROIE. RORE variants improve over baselines across all settings; RORE-GeoLayoutLM achieves the best comprehensive VrD-IE performance (93.94 overall). ### VrD-QA with Pseudo Labels (Table 4) Tested on DocVQA, InfoVQA, WTQ, TextVQA using ANLS. Gains range from +0.09 (TextVQA, simple layouts) to +6.48 (InfoVQA, layout-heavy). TextVQA's minimal gain is attributed to its simple top-to-bottom layouts providing little new signal. ## What are the outcomes/conclusions? 1. **Permutation formulation is insufficient.** 23.76% of segments in ROOR are in non-linear reading positions. A single sequence cannot represent them without noise. 2. **Relation-based models dramatically outperform sequence-based models on ROOR.** LayoutLMv3-large at 93.01 F1 versus LayoutReader-style baselines below 10 F1 at word-level; though this comparison is partly an artifact of task reformulation (the prior models were not designed for this metric). 3. **Reading order relations improve downstream VrD tasks universally**, including via pseudo labels generated by the ROP model, without task-specific tuning. This is the most practically significant finding. 4. **EL benefits more than SER from reading order.** Entity linking requires understanding logical spatial flow; SER relies more on local entity semantics. ### Limitations - The ROOR benchmark is small (199 samples) and built from a single source dataset (EC-FUNSD). Generalization of the benchmark itself is not validated. - The baseline ROP model over-relies on spatial layout features relative to textual features, as shown in case studies in the appendix. - Only one fusion approach (relation-aware attention additive bias) is tested; graph-based or pre-training-based alternatives are acknowledged but not compared. - Search space increases from $O(n!)$ (permutation) to $O(2^{n^2})$ (DAR), which the authors flag as an inherent cost of the richer representation. ## Reproducibility ### Models - Backbone: LayoutLMv3-base (133M) and LayoutLMv3-large (368M). Standard HuggingFace releases. - Downstream baselines: GeoLayoutLM (399M), generative LayoutLMv3 + BART decoder. - No new model weights released; code for training and inference is available. ### Algorithms - Global pointer network for relation extraction (Su et al., 2022). - Relation-aware attention: additive bias with learnable per-layer scalar $\lambda^l$, initialized from pre-trained backbone. - Training details in Appendix D of the paper; specific hyperparameters not reported in main text. ### Data - ROOR dataset: 199 samples from EC-FUNSD, train/validation split follows EC-FUNSD. Released at [github.com/chongzhangFDU/ROOR-Datasets](https://github.com/chongzhangFDU/ROOR-Datasets) under CC-BY-4.0. - Annotation: two domain experts, conflict resolution by discussion, no formal inter-annotator agreement metric reported beyond 99.28 human F1 on the task. - Downstream benchmarks: FUNSD, CORD, SROIE, DocVQA, InfoVQA, WTQ, TextVQA (all publicly available, used as-is). ### Evaluation - Primary ROP metric: F1 over ISDR relation pairs (not BLEU or ARD). - Downstream metrics: F1 for SER/EL, ANLS for VrD-QA. - No error bars or significance tests reported. ### Hardware - Not explicitly reported in the main paper or appendix. ## BibTeX ```bibtex @inproceedings{zhang2024roor, title={Modeling Layout Reading Order as Ordering Relations for Visually-rich Document Understanding}, author={Zhang, Chong and Tu, Yi and Zhao, Yixi and Yuan, Chenshu and Chen, Huan and Zhang, Yue and Chai, Mingxu and Guo, Ya and Zhu, Huijia and Zhang, Qi and Gui, Tao}, booktitle={Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing}, year={2024} } ```