# LayoutMask: Local 1D Position and Dual Masking for Text-Layout Pre-training > LayoutMask replaces global reading-order positions with local, in-segment 1D positions and introduces two masking objectives (Masked Language Modeling with whole-word and layout-aware strategies, and Masked Position Modeling) to learn more robust text-layout representations. - Authors: Bevaya AI Team - Published: 2023-05-30 - Updated: 2026-04-02 - Paper: Preprint (arXiv) - Paper: Published (ACL 2023) - Source: Bevaya Labs, --- ## TL;DR LayoutMask is a text-and-layout-only pre-training model for visually-rich document understanding (VrDU). It replaces the global 1D token positions used by prior models with local, in-segment positions, forcing reading order to be inferred jointly from 1D and 2D layout signals. Two objectives drive pre-training: a Masked Language Model augmented with Whole Word Masking and Layout-Aware Masking, and a new Masked Position Modeling task that recovers masked 2D bounding boxes via GIoU loss. The authors report the base model leads all prior methods benchmarked on FUNSD, including those that incorporate image modality. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Method}}$: the paper proposes a new pre-training architecture and two new pre-training objectives. The central claim is that local 1D position, combined with targeted masking strategies, produces more adaptive and robust document representations than global reading-order positions. The main content is an ablation grid across position choices and pre-training configurations, with baseline comparison tables across multiple VrDU benchmarks. **Secondary:** $\Psi\_{\text{Evaluation}}$: the authors conduct a robustness analysis by simulating segment-swap layout disturbances on held-out test sets, quantifying the sensitivity of global vs. local 1D positions to OCR ordering errors. ## What is the motivation? Pre-trained models for VrDU tasks (forms, receipts, document classification) routinely encode reading order as a globally ascending sequence of 1D position integers: token 0, 1, 2, ..., 511 across the whole document. This representation carries two structural problems. First, plain text has a single linear reading order, but document layouts do not. A receipt with a vertical price column alongside a horizontal total row cannot be faithfully serialized by a single linear ordering. Any such ordering is an approximation, and different OCR tools or heuristic rules may produce different approximations for the same document. Second, the global order depends on consistent, stable OCR results and on empirical serialization rules (e.g., top-down, left-right). When OCR segments are detected out of order due to document rotation or scanning artifacts, the global 1D positions encode incorrect cross-segment precedence, which then contaminates fine-tuned models at inference time. The authors argue that requiring a model to infer global reading order from local and 2D positional signals, rather than receiving it as an explicit input, forces richer text-layout interactions and produces representations that generalize better across layout styles. ## What is the novelty? ### Local 1D Position Rather than assigning each token a globally unique position index reflecting its cross-document reading rank, LayoutMask assigns positions that restart at 1 for each OCR segment. A segment might be a line or a bounding box returned by the OCR engine. Within a segment, words are ordered 1, 2, 3, ... as usual. Across segments, no ordering information is conveyed by the 1D position. The model is instead expected to recover cross-segment ordering from segment-level 2D bounding boxes (shared by all tokens in a segment) and from semantic context. Table 1 in the paper compares position choices across prior work; LayoutMask is the only model using local 1D position. ### Masked Language Modeling with Two Masking Strategies The standard MLM loss averages cross-entropy over masked tokens: $$\mathcal{L}\_{\text{mlm}} = -\frac{1}{M}\sum\_{i=1}^{M} \text{CE}(y\_i, \hat{y}\_i)$$ where $M$ is the number of masked tokens and $y\_i$, $\hat{y}\_i$ are the ground truth and predicted token for position $i$. **Whole Word Masking (WWM)** applies masks at word level rather than subword level. All subword tokens of a word are masked together, which removes within-word context and forces the model to rely on surrounding words and layout signals. **Layout-Aware Masking (LAM)** targets cross-segment boundaries. Because local 1D positions do not encode cross-segment order, the model has to infer that order from segment 2D positions and semantics. LAM increases the masking probability for the first and last word of each segment from $P\_{\text{mlm}}$ to $3 \times P\_{\text{mlm}}$, specifically forcing the model to reason about what precedes or follows a given segment. ### Masked Position Modeling Masked Position Modeling (MPM) is an auxiliary objective that predicts the 2D bounding box of a randomly selected word whose spatial information has been removed. The procedure involves two steps: 1. **Box Split:** The selected word is separated from its segment. The segment is split into two or three pieces around the selected word, and local 1D positions are recomputed for each piece. 2. **Box Masking:** The word's 2D position is replaced with a pseudo-box $[0, 0, 0, n]$ where $n$ is a random integer, acting as a unique identifier to distinguish multiple masked positions. The model predicts the original 2D box for each masked word. The MPM loss is GIoU averaged over $N$ masked positions: $$\mathcal{L}\_{\text{mpm}} = \frac{1}{N} \sum\_{i=1}^{N} \left( \frac{|B\_i \cap \hat{B}\_i|}{|B\_i \cup \hat{B}\_i|} - \frac{|C\_i \setminus (B\_i \cup \hat{B}\_i)|}{|C\_i|} \right)$$ where $B\_i$ is the ground-truth box (normalized to $[0,1]$), $\hat{B}\_i$ is the predicted box, and $C\_i$ is the smallest convex shape covering both. The combined training objective is: $$\mathcal{L}\_{\text{total}} = \mathcal{L}\_{\text{mlm}} + \lambda \mathcal{L}\_{\text{mpm}}$$ with $\lambda = 1$ in practice. ## What experiments were performed? ### Pre-training LayoutMask uses a spatial-aware self-attention backbone from LayoutLMv2. It is initialized from XLM-RoBERTa weights and pre-trained on 10 million pages sampled from the IIT-CDIP Test Collection (42 million scanned pages total). OCR is performed with PaddleOCR. Two sizes are trained: Base (12 layers, 16 heads, hidden size 768, 182M parameters) and Large (24 layers, 16 heads, hidden size 1024, 404M parameters). Masking probabilities are $P\_{\text{mlm}} = 25\%$ and $P\_{\text{mpm}} = 15\%$. ### Downstream Tasks **Form and receipt understanding** (entity extraction / named entity recognition): FUNSD (199 forms), CORD (1000 receipts, 30 entity types), and SROIE (973 receipts, 4 entity types). Word-level F1 for FUNSD and CORD; entity-level F1 for SROIE. Each experiment is repeated 10 times; the paper reports mean and standard error. **Document image classification:** RVL-CDIP (400,000 images, 16 categories; 320k train / 40k val / 40k test). Overall classification accuracy is the metric. ### Ablation Studies The authors systematically ablate: - 1D position type (global vs. local) crossed with 2D position type (word-level vs. segment-level), reported as average F1 across FUNSD, CORD, and SROIE (Table 4). - All combinations of the four training components: no pre-training, MLM only, MLM+WWM, MLM+LAM, MLM+WWM+LAM+MPM (Table 6 across all four benchmarks). - Robustness to segment-swap perturbations: for Global+Segment models, segments in the same line are swapped with probability $P\_{\text{swap}} \in \\{10\%, 20\%, 30\%\\}$ at test time (Table 5). - Masking probabilities $P\_{\text{mlm}}$ and $P\_{\text{mpm}}$ (Appendix A). ## What are the outcomes/conclusions? ### Main Results On FUNSD, CORD, and SROIE (entity extraction), the authors report LayoutMask Base achieves 92.91, 96.99, and 96.87 F1 respectively, exceeding all prior base models the paper benchmarks against, including those that incorporate image modality (e.g., LayoutLMv3 Base at 90.29 on FUNSD). LayoutMask Large leads on FUNSD (93.20) and is competitive on CORD and SROIE. On RVL-CDIP (document classification), LayoutMask Base and Large reach 93.26% and 93.80% accuracy. These results are competitive with text-only and text+layout baselines, but trail trimodal (T+L+I) models. The authors attribute this gap to RVL-CDIP containing figures, table lines, and orientation anomalies that cannot be captured without image features. ### Ablation Findings The Local+Segment position combination is consistently best or near-best. The performance advantage of local over global 1D is most pronounced on FUNSD (where complex spatial layouts dominate) and on SROIE for the "Total" entity (which appears in mixed vertical/horizontal layouts with many semantically identical number strings nearby). Adding WWM, LAM, and MPM each contributes incremental gains across all four benchmarks. The full model improves over the naive version by 3.18% on FUNSD, 0.67% on CORD, 1.11% on SROIE, and 1.09% on RVL-CDIP. The robustness experiment shows that global 1D positions degrade substantially under segment-swap perturbations: the "Address" entity on SROIE drops by 4.81%, 6.51%, and 8.42% at swap rates of 10%, 20%, and 30%. Local 1D positions are not affected by this perturbation because they carry no cross-segment ordering signal. ### Limitations The authors identify two limitations. First, the evaluation datasets are small and domain-limited, so it is unclear how well the conclusions transfer to more diverse real-world document collections. Second, LayoutMask deliberately omits image modality, leaving a gap on tasks where visual elements (figures, ruling lines, handwriting) carry essential information not captured by OCR. Integrating image modality within the local-position framework is left as future work. ## Reproducibility ### Models LayoutMask is a transformer with spatial-aware self-attention (following LayoutLMv2). The Base variant has 12 layers, 16 heads, and hidden dimension 768 (182M parameters). The Large variant has 24 layers, 16 heads, and hidden dimension 1024 (404M parameters). Both are initialized from XLM-RoBERTa (Base and Large respectively). No model weights appear to have been publicly released with the paper. ### Algorithms Pre-training uses the combined loss $\mathcal{L}\_{\text{total}} = \mathcal{L}\_{\text{mlm}} + \lambda \mathcal{L}\_{\text{mpm}}$ with $\lambda = 1$, $P\_{\text{mlm}} = 25\%$, and $P\_{\text{mpm}} = 15\%$. Masking is applied at word level for both MLM and MPM. LAM triples the masking probability for first and last words of each segment. Optimizer, batch size, learning rate schedule, and total training steps are not detailed in the main paper or appendix. ### Data Pre-training uses 10 million pages sampled from the IIT-CDIP Test Collection. OCR is produced by PaddleOCR. IIT-CDIP is publicly available for research use. No data filtering or augmentation details beyond OCR are provided. Downstream evaluation uses FUNSD, CORD, SROIE, and RVL-CDIP, all publicly available benchmarks. ### Evaluation Entity extraction uses word-level F1 (FUNSD, CORD) and entity-level F1 (SROIE). Results are averaged over 10 runs with standard errors reported; this is more statistically careful than most prior work in the space, which typically reports single runs. Document classification uses top-1 accuracy on RVL-CDIP. No cross-dataset evaluation or domain generalization tests are performed. ### Hardware Hardware details are not reported in the paper or appendix. ## BibTeX ```bibtex @inproceedings{tu-etal-2023-layoutmask, title = "{L}ayout{M}ask: Enhance Text-Layout Interaction in Multi-modal Pre-training for Document Understanding", author = "Tu, Yi and Guo, Ya and Chen, Huan and Tang, Jinyang", booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)", year = "2023", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2023.acl-long.847", doi = "10.18653/v1/2023.acl-long.847", } ```