# ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understanding > ERNIE-Layout is a document pre-training model from Baidu that enhances layout awareness through reading-order-corrected serialization, spatial-aware disentangled attention, and two pre-training tasks: reading order prediction and replaced region prediction. - Authors: Bevaya AI Team - Published: 2022-10-12 - Updated: 2026-04-02 - Paper: Preprint (arXiv) - Paper: Published (Findings of EMNLP 2022) - Code: ERNIE-Layout (code and models) - Source: Bevaya Labs, --- ## TL;DR ERNIE-Layout is a multi-modal pre-trained model for visually-rich document understanding that treats layout as a first-class modality rather than a positional add-on. The two core contributions are: (1) correcting the serialization order of document tokens using a layout-aware parser before pre-training, coupled with a reading order prediction pre-training task; and (2) a spatial-aware disentangled attention mechanism that incorporates 1D and 2D relative positions directly into attention weight computation. On downstream key information extraction, document question answering, and document image classification tasks, the large variant improves substantially over prior methods. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Method}}$: The paper introduces a new model architecture, attention mechanism, pre-training objectives, and serialization pipeline. Performance on VrDU benchmarks is the central evaluation, with ablations isolating each design choice. **Secondary:** $\Psi\_{\text{Evaluation}}$: The ablation study systematically measures the contribution of each pre-training task and attention variant, providing quantitative comparisons across serialization strategies. ## What is the motivation? Visually-rich document understanding (VrDU) involves forms, invoices, receipts, and other documents where text, spatial layout, and visual appearance jointly determine meaning. Models like LayoutLM and LayoutLMv2 showed that incorporating 2D bounding box coordinates can substantially improve over text-only baselines. The authors identify two systematic gaps in prior approaches. First, serialization: existing methods pass OCR output to the model in raster-scan order (left to right, top to bottom), which is incorrect for documents with multi-column layouts, tables, or parallel text blocks. Human reading order groups words by their logical region first, then proceeds from region to region. Raster-scan serialization breaks this grouping and forces the model to learn reading order from corrupted sequences. Second, attention integration: prior work treats layout coordinates as extra input embeddings (appended to token representations) rather than as an independent factor in pairwise token similarity. The cross-modal interaction between layout and content is therefore shallow. ## What is the novelty? ERNIE-Layout addresses both gaps through a unified pre-training framework with four interconnected components. **Serialization module.** Before training, documents are passed through Document-Parser, a layout analysis toolkit built on top of Layout-Parser. Document-Parser detects paragraph boundaries, tables, and figures, then applies element-type-specific heuristics to assign a logical reading order. The result is that tokens belonging to the same table cell or paragraph stay contiguous, even when their pixel positions would place them in an interleaved raster order. The authors validate this with perplexity measured by GPT-2: for complex-layout documents, Document-Parser serialization reduces perplexity substantially compared to raster-scan order. **Input representation.** Each token (textual or visual) receives three additive embeddings: $$T = E\_{tk}(T) + E\_{1p}(T) + E\_{tp}(T)$$ $$V = F\_{vs}(V) + E\_{1p}(V) + E\_{tp}([V])$$ $$L = E\_{2x}(x\_0, x\_1, w) + E\_{2y}(y\_0, y\_1, h)$$ The final combined representation is: $$H = [T + L \;;\; V + L]$$ Visual tokens come from a Faster-RCNN visual encoder that produces a $7 \times 7 = 49$-token sequence from a $224 \times 224$ image. **Spatial-aware disentangled attention.** Inspired by DeBERTa's disentangled attention, ERNIE-Layout decomposes each pairwise attention score into four components: content-to-content, content-to-1D-position, content-to-2D-x, and content-to-2D-y. Relative position indices are computed for each axis with a clipping range $k$: $$\delta\_{1p}(i, j) = \begin{cases} 0 & \text{for } i - j \le -k \\\\ 2k - 1 & \text{for } i - j \ge k \\\\ i - j + k & \text{otherwise} \end{cases}$$ The four attention score components are: $$A\_{ij}^{ct,ct} = Q\_i^{ct} K\_j^{ct\,\mathsf{T}}$$ $$A\_{ij}^{ct,1p} = Q\_i^{ct} K\_{\delta\_{1p}(i,j)}^{1p\,\mathsf{T}} + K\_j^{ct} Q\_{\delta\_{1p}(j,i)}^{1p\,\mathsf{T}}$$ $$A\_{ij}^{ct,2x} = Q\_i^{ct} K\_{\delta\_{2x}(i,j)}^{2x\,\mathsf{T}} + K\_j^{ct} Q\_{\delta\_{2x}(j,i)}^{2x\,\mathsf{T}}$$ $$A\_{ij}^{ct,2y} = Q\_i^{ct} K\_{\delta\_{2y}(i,j)}^{2y\,\mathsf{T}} + K\_j^{ct} Q\_{\delta\_{2y}(j,i)}^{2y\,\mathsf{T}}$$ The final attention matrix and output are: $$A\_{ij} = A\_{ij}^{ct,ct} + A\_{ij}^{ct,1p} + A\_{ij}^{ct,2x} + A\_{ij}^{ct,2y}$$ $$H\_{out} = \text{softmax}\!\left(\frac{\hat{A}}{\sqrt{3d}}\right) V^{ct}$$ This means layout directly influences which pairs of tokens attend strongly to each other, rather than layout affecting only the query/key vectors through the input embedding. **Pre-training tasks.** ERNIE-Layout uses four tasks jointly: Reading Order Prediction (ROP): given the attention matrix $\hat{A}$, the model learns to assign $A\_{ij}$ the additional meaning of "probability that token $j$ is the next token after token $i$ in reading order." Ground truth is a binary matrix $G$ where $G\_{ij} = 1$ when there is a reading-order successor relationship. The loss is: $$\mathcal{L}\_{\text{ROP}} = -\sum\_{0 < i < N} \sum\_{0 < j < N} G\_{ij} \log(\hat{A}\_{ij})$$ Replaced Region Prediction (RRP): 10% of visual patches are replaced with patches from another document image. The [CLS] vector is used to predict which patches were replaced, trained with binary cross-entropy: $$\mathcal{L}\_{\text{RRP}} = \sum\_{0 \le i < HW} \left[ G\_i \log(P\_i) + (1 - G\_i) \log(1 - P\_i) \right]$$ Masked Visual-Language Modeling (MVLM) and Text-Image Alignment (TIA) are carried over from LayoutLMv2. The overall pre-training objective sums all four losses: $$\mathcal{L} = \mathcal{L}\_{\text{ROP}} + \mathcal{L}\_{\text{RRP}} + \mathcal{L}\_{\text{MVLM}} + \mathcal{L}\_{\text{TIA}}$$ ## What experiments were performed? **Pre-training data.** The model is pre-trained on 10 million scanned document pages sampled from the IIT-CDIP Test Collection, a large tobacco industry document archive. The layout-aware serialization is applied during pre-training. Notably, the fine-tuning datasets retain their original raster-scan OCR order, so the serialization benefit comes entirely from improved pre-training representations. **Downstream tasks and datasets.** Three task types are evaluated across six datasets: - Key information extraction (sequence labeling): FUNSD, CORD, SROIE, Kleister-NDA. Metric: entity-level F1. - Document question answering: DocVQA. Metric: Average Normalized Levenshtein Similarity (ANLS). - Document image classification: RVL-CDIP. Metric: accuracy. **Baselines.** The comparisons include text-only models (BERT-large, RoBERTa-large, UniLMv2-large) and multi-modal document models (LayoutLM, LayoutLMv2, StructuralLM, DocFormer, TILT). **Ablations.** Table 6 in the paper shows the incremental effect of adding RRP, ROP, and switching attention mechanisms. Table 7 compares raster-scan, Layout-Parser, and Document-Parser serialization on FUNSD and CORD. ## What are the outcomes/conclusions? On key information extraction, ERNIE-Layout-large achieves F1 of 0.9312 on FUNSD (7.98 points above the previous best), 0.9721 on CORD, 0.9755 on SROIE, and 0.8810 on Kleister-NDA. On document image classification (RVL-CDIP), accuracy is 0.9627, edging past StructuralLM. On DocVQA, ANLS is 0.8321 with the train split alone, and 0.8841 on the leaderboard with ensemble; this is competitive but not uniformly best, which the authors attribute to ERNIE-Layout being initialized from RoBERTa while LayoutLMv2 starts from the stronger UniLMv2 question-answering backbone. The ablation results are informative. Adding RRP improves FUNSD F1 by about 0.95 points over the MVLM + TIA baseline. Adding ROP on top of that adds roughly 1.3 more points. Switching from LayoutLMv2's spatial-aware self-attention to ERNIE-Layout's disentangled variant adds a further gain. The serialization ablation suggests that layout-aware parsing helps even at fine-tuning time when the fine-tuning data itself is still raster-scanned, confirming that the pre-training representation carries useful reading-order knowledge. The base-size model (12 layers, 768 hidden, initialized from RoBERTa-base) reproduces the same pattern and on several datasets outperforms large-scale baselines. One limitation is that Document-Parser is a proprietary internal Baidu toolkit. While the paper describes it conceptually and compares it against open-source Layout-Parser, the exact implementation is not released. Downstream fine-tuning also uses raster-scan OCR rather than the improved serializer, which limits the benefit to pre-training only. The authors do not report error bars or statistical significance tests for the main results, only for the FUNSD ablation where they average five random seeds. ## Reproducibility ### Models - ERNIE-Layout-large: 24 transformer layers, 1024 hidden units, 16 attention heads. Initialized from RoBERTa-large. - ERNIE-Layout-base: 12 transformer layers, 768 hidden units, 12 attention heads. Initialized from RoBERTa-base. - Visual encoder: Faster-RCNN, initialized from a pretrained Faster-RCNN checkpoint. Output is an adaptive-pooled $7 \times 7 = 49$ token sequence. - Code and model weights are available via PaddleNLP at the artifact URL above (Apache-2.0 license). ### Algorithms - Optimizer: Adam with learning rate 1e-4, weight decay 0.01 during pre-training; learning rate 2e-5 with weight decay 0.01 during fine-tuning. - Learning rate schedule: linear warm-up over the first 10% of steps, then linear decay to zero. - Pre-training batch size: 576; trained for 20 epochs. - Maximum textual sequence length: 512 tokens; visual sequence length: 49 tokens. - Fine-tuning hyperparameters (epochs, batch size, weight decay) vary per dataset and are reported in Table 2 of the paper. - No mixed precision or gradient clipping details are reported. ### Data - Pre-training: 10 million pages sampled from the IIT-CDIP Test Collection (tobacco documents), serialized with Document-Parser. - Fine-tuning: FUNSD (199 documents), CORD (1000 receipts), SROIE (973 receipts), Kleister-NDA (540 documents), RVL-CDIP (400k document images), DocVQA (50k questions on 12,767 images). - OCR: for FUNSD, CORD, SROIE, and Kleister-NDA, the official OCR annotations are used. For RVL-CDIP and DocVQA, Microsoft OCR tools are used. - Fine-tuning data is not re-serialized; only pre-training data uses Document-Parser ordering. ### Evaluation - Key information extraction: entity-level F1 (BIO labeling, sequence labeling framework). - Document QA: ANLS, as defined in the DocVQA competition. - Document classification: accuracy on RVL-CDIP (16 classes). - The main results compare model families at the large scale. Base-scale results are in the appendix. - Statistical rigor: five-run averaging with standard deviation reported only for the FUNSD ablation (standard deviation 0.0011 for the best-vs-prior comparison). No significance tests for other results. - The Kleister-NDA test set is not publicly available; results are reported on the validation set. ### Hardware - Pre-training: 24 Tesla A100 GPUs, 20 epochs. Total GPU-hours are not stated. - Fine-tuning hardware is not reported. - Inference requirements are not discussed. ## BibTeX ```bibtex @inproceedings{peng-etal-2022-ernie, title = "{ERNIE}-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understanding", author = "Peng, Qiming and Pan, Yinxu and Wang, Wenjin and Luo, Bin and Zhang, Zhenyu and Huang, Zhengjie and Cao, Yuhui and Yin, Weichong and Chen, Yongfeng and Zhang, Yin and Feng, Shikun and Sun, Yu and Tian, Hao and Wu, Hua and Wang, Haifeng", booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2022", month = dec, year = "2022", address = "Abu Dhabi, United Arab Emirates", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2022.findings-emnlp.274/", doi = "10.18653/v1/2022.findings-emnlp.274", pages = "3744--3756" } ```