# LayoutLM: Pre-training of Text and Layout for Document Image Understanding > Research note on LayoutLM, the first model to jointly pre-train text and 2D layout information for document image understanding, achieving strong results on form understanding, receipt extraction, and document classification. - Authors: Hunter Heidenreich - Published: 2026-03-13 - Paper: Preprint (arXiv) - Paper: Published (KDD 2020) - Code: LayoutLM Code - Model: layoutlm-base-uncased - Model: layoutlm-large-uncased - Source: Bevaya Labs, --- ## TL;DR LayoutLM extends BERT by adding 2D position embeddings (encoding each token's bounding box on the page) and optional image embeddings (via Faster R-CNN). Pre-trained on 11M scanned document images from IIT-CDIP with a Masked Visual-Language Model objective, it was the first model to jointly learn text and layout in a single framework. It improved form understanding (FUNSD F1: 70.72 $\rightarrow$ 79.27), receipt extraction (SROIE F1: 94.02 $\rightarrow$ 95.24), and document classification (RVL-CDIP: 93.07% $\rightarrow$ 94.42%). ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Method}}$. The primary contribution is a pre-training architecture that extends BERT with 2D layout position embeddings and image embeddings for document image understanding. The paper is organized around the model architecture, pre-training objectives, and SOTA comparisons on three downstream tasks. Ablations over data scale, pre-training epochs, initialization methods, and modality combinations take up most of the experimental section. **Secondary:** None. Code and model weights are released, but the paper frames them as supporting the method rather than as standalone contributions. ## What is the motivation? Prior approaches to document AI had two key limitations: 1. **Limited use of unlabeled data.** Models relied on small labeled datasets without leveraging self-supervised pre-training on large-scale document collections. 2. **Disconnected modalities.** Existing methods used either pre-trained CV models or pre-trained NLP models, but never jointly trained text and layout information together. The authors observed that in document images, spatial position carries substantial semantic signal (e.g., a form value is usually to the right of or below its key). Standard NLP pre-training (BERT, RoBERTa) discards this spatial information entirely. LayoutLM addresses this by encoding 2D bounding box coordinates directly into the token representation. ## What is the novelty? ### 2D Position Embedding LayoutLM represents each token's location using its bounding box coordinates $(x\_0, y\_0, x\_1, y\_1)$, where $(x\_0, y\_0)$ is the upper-left corner and $(x\_1, y\_1)$ is the lower-right corner. All coordinates are normalized to a $[0, 1000]$ virtual coordinate system. Four position embedding layers are added, with $x$ and $y$ dimensions sharing separate embedding tables: $$\text{emb}\_{\text{2D}} = E\_X(x\_0) + E\_Y(y\_0) + E\_X(x\_1) + E\_Y(y\_1)$$ This is concatenated with the standard token, 1D position, and segment embeddings from BERT. ### Image Embedding For the image modality, each token's bounding box crop is passed through a pre-trained Faster R-CNN (ResNet-101 backbone, pre-trained on Visual Genome) to produce a region feature vector. The $[\text{CLS}]$ token receives the whole-page image embedding as its ROI. Image embeddings are added during fine-tuning (not during pre-training in the released version). ### Pre-training Objectives Two objectives are used: 1. **Masked Visual-Language Model (MVLM):** 15% of input tokens are masked (following the BERT 80/10/10 scheme), but their 2D position embeddings are kept. The model must recover masked tokens using both textual context and layout position, bridging the two modalities. 2. **Multi-label Document Classification (MDC):** Using IIT-CDIP's document tags, the $[\text{CLS}]$ representation is trained to predict document categories. This is optional and only applicable when document labels are available. ## What experiments were performed? ### Benchmarks | Benchmark | Task | Metric | Domain | |-----------|------|--------|--------| | FUNSD | Form understanding (sequence labeling) | F1 | Noisy scanned forms (199 pages) | | SROIE | Receipt key info extraction (sequence labeling) | F1 | Scanned receipts (973 pages) | | RVL-CDIP | Document image classification (16 classes) | Accuracy | Diverse scanned documents (400K pages) | ### Key Results **FUNSD (Form Understanding):** | Model | Modality | F1 | Params | |-------|----------|----|--------| | BERT$\_{\text{BASE}}$ | Text | 0.6026 | 110M | | RoBERTa$\_{\text{LARGE}}$ | Text | 0.7072 | 355M | | LayoutLM$\_{\text{BASE}}$ (11M, MVLM) | Text + Layout | 0.7866 | 113M | | **LayoutLM$\_{\text{BASE}}$ (11M, MVLM+Image)** | **Text + Layout + Image** | **0.7927** | **160M** | **SROIE (Receipt Understanding):** | Model | F1 | Params | |-------|----|--------| | BERT$\_{\text{LARGE}}$ | 0.9200 | 340M | | Competition 1st place | 0.9402 | -- | | **LayoutLM$\_{\text{LARGE}}$ (11M, MVLM)** | **0.9524** | **343M** | **RVL-CDIP (Document Classification):** | Model | Accuracy | Params | |-------|----------|--------| | Multimodal Ensemble | 93.07% | -- | | LayoutLM$\_{\text{BASE}}$ (11M, Text+Layout) | 91.78% | 113M | | **LayoutLM$\_{\text{BASE}}$ (11M, Text+Layout+Image)** | **94.42%** | **160M** | ### Scaling Behavior The paper includes a thorough study of how pre-training data scale and epochs affect downstream performance on FUNSD. F1 increases monotonically with both more data (500K $\rightarrow$ 11M pages) and more epochs, though gains diminish at larger scales. This is among the more useful analyses in the paper, confirming that layout-aware pre-training benefits substantially from scale even on a tiny downstream dataset (149 training forms). ### Initialization Ablation RoBERTa initialization outperforms BERT initialization by 2.1 F1 points (BASE) and 1.3 F1 points (LARGE) on FUNSD. Training from scratch yields significantly worse results, confirming the value of warm-starting from a text-only pre-trained model. ## What are the outcomes/conclusions? **Strengths:** - The 2D position embedding idea is simple, well-motivated, and adds minimal parameter overhead. Subsequent work adopted this design (LayoutLMv2, LayoutLMv3), and other multimodal document models incorporated similar spatial encoding approaches. - The scaling study (Table 2) is informative: it shows clear log-linear improvement with pre-training data size, useful for practitioners deciding how much pre-training is worth the compute. - Strong results on three diverse tasks (forms, receipts, classification) demonstrate the generality of the approach. - The FUNSD improvement (+8.6 F1 percentage points over RoBERTa$\_{\text{LARGE}}$ with a smaller model) is substantial and practically significant for form understanding workflows. **Limitations and caveats:** - Image embeddings are only used during fine-tuning, not during pre-training. The authors acknowledge this and note plans to incorporate image features into pre-training in future work (which was realized in LayoutLMv2). - The Faster R-CNN backbone adds 47M parameters (113M $\rightarrow$ 160M for BASE) and requires an additional pre-trained object detector. This is a notable complexity and inference cost penalty. - Only three downstream tasks are evaluated. No layout detection or reading order tasks are included. - The MDC pre-training objective relies on IIT-CDIP metadata tags, which the authors note are noisy and inconsistent. The marginal benefit of MDC is modest (e.g., +0.73 F1 percentage points on FUNSD for the 1M/6-epoch setting). - No error bars or multi-run statistics are reported. - OCR quality dependency: the method assumes reasonable OCR output is available. Errors in OCR (bounding boxes or text) propagate directly into the model's input representation. The paper does not evaluate robustness to OCR noise. ## Reproducibility ### Models - **LayoutLM$\_{\text{BASE}}$:** 12-layer Transformer, 12 heads, hidden size 768. ~113M parameters (text + layout) or ~160M with image embeddings. - **LayoutLM$\_{\text{LARGE}}$:** 24-layer Transformer, 16 heads, hidden size 1024. ~343M parameters. - Initialized from BERT (base/large). RoBERTa initialization also tested and shown to be superior. - Image encoder: Faster R-CNN with ResNet-101, pre-trained on Visual Genome. Used only at fine-tuning time. - Pre-trained checkpoints (base, large) publicly available on Hugging Face under MIT license. ### Algorithms - **Optimizer:** Adam, initial LR $5 \times 10^{-5}$, linear decay schedule. - **Pre-training:** Total batch size 80 across 8 GPUs. BASE takes ~80 hours/epoch on 11M images; LARGE takes ~170 hours/epoch. - **MVLM masking:** 15% of tokens masked with BERT's 80/10/10 replacement scheme. 2D position embeddings are preserved for masked tokens. - **Fine-tuning hyperparameters:** - FUNSD: 100 epochs, batch size 16, LR $5 \times 10^{-5}$. - SROIE: specific settings not detailed in paper. - RVL-CDIP: 30 epochs, batch size 40, LR $2 \times 10^{-5}$. - 2D coordinates normalized to $[0, 1000]$ virtual coordinate space. ### Data - **Pre-training:** IIT-CDIP Test Collection 1.0. 6M+ documents, 11M scanned images. OCR re-processed with Tesseract (version not specified); results stored in hOCR format. IIT-CDIP is a gated dataset requiring a formal data use agreement through UCSF Industry Documents Library; the original NIST distribution URL is no longer active. - **FUNSD:** 199 forms (149 train / 50 test). 9,707 entities, 31,485 words. Publicly available. - **SROIE:** 626 train / 347 test receipts. Ground truth OCR used for experiments. Publicly available via ICDAR 2019 competition. - **RVL-CDIP:** 400K images in 16 classes (320K train / 40K val / 40K test). Subset of IIT-CDIP with document-type labels. Publicly available. ### Evaluation - **FUNSD:** Word-level F1 on semantic entity labeling. - **SROIE:** Exact-match entity F1 (Task 3: Key Information Extraction). Ground truth OCR used to isolate model contribution from OCR quality. - **RVL-CDIP:** Overall classification accuracy across 16 classes. - Baselines include BERT (base/large), RoBERTa (base/large), and image-only models (VGG-16, InceptionResNetV2, LadderNet, etc.). - No error bars, significance tests, or multi-run statistics reported. ### Hardware - **Pre-training:** 8 NVIDIA Tesla V100 32GB GPUs. BASE: ~80 hours/epoch; LARGE: ~170 hours/epoch. - No cost estimates or energy consumption figures reported. - Inference hardware requirements not specified. The Faster R-CNN component (when image embeddings are used) adds meaningful inference latency. ## BibTeX ```bibtex @inproceedings{xu2020layoutlm, title={LayoutLM: Pre-training of Text and Layout for Document Image Understanding}, author={Xu, Yiheng and Li, Minghao and Cui, Lei and Huang, Shaohan and Wei, Furu and Zhou, Ming}, booktitle={Proceedings of the 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining}, pages={1192--1200}, year={2020}, doi={10.1145/3394486.3403172} } ```