# LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking > Research note on LayoutLMv3, a multimodal pre-training approach for Document AI that unifies text and image masking objectives without relying on pre-trained CNN or Faster R-CNN visual backbones. - Authors: Hunter Heidenreich - Published: 2025-03-13 - Updated: 2026-03-13 - Paper: Preprint (arXiv) - Paper: Published (ACM MM 2022) - Code: LayoutLMv3 Code & Models - Model: layoutlmv3-base - Model: layoutlmv3-large - Model: layoutlmv3-base-chinese - Source: Bevaya Labs, --- ## TL;DR LayoutLMv3 is a multimodal Transformer for Document AI that replaces CNN-based image embeddings with simple linear patch projections and unifies text and image pre-training through masked language modeling (MLM), masked image modeling (MIM), and a word-patch alignment (WPA) objective. The authors report it is the first multimodal Document AI model to operate without CNN or Faster R-CNN backbones, and it performs competitively on both text-centric tasks (form understanding, receipt parsing, document VQA) and image-centric tasks (document classification, layout analysis) with fewer parameters than prior approaches. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Method}}$. The headline contribution is a new multimodal pre-training architecture and training recipe. The paper dedicates most of its pages to describing the unified masking objectives (MLM, MIM, WPA), the CNN-free image embedding design, and ablation studies that isolate the contribution of each component. SOTA comparisons across five benchmarks are the primary evidence. **Secondary:** None. While the authors release code and model weights, the paper does not frame these as the primary contribution; the models serve as validation of the method. ## What is the motivation? Prior multimodal Document AI models (LayoutLM, LayoutLMv2, DocFormer, SelfDoc) all relied on CNN backbones (ResNet, ResNeXt, Faster R-CNN) to extract visual features. This had several downsides: - **Parameter overhead:** CNN backbones added tens of millions of parameters (e.g., 44M for ResNet-101) on top of the Transformer. - **Region supervision:** Models using Faster R-CNN required object detection pre-training with bounding box annotations. - **Objective mismatch:** Text and image modalities used fundamentally different pre-training objectives (discrete token prediction for text vs. pixel reconstruction or region feature regression for images), making cross-modal alignment difficult. The gap between text objectives (discrete, symbolic) and image objectives (continuous, pixel-level) hindered multimodal representation learning. The authors sought to unify the pre-training approach across modalities. ## What is the novelty? ### CNN-free Image Embedding LayoutLMv3 replaces CNN/Faster R-CNN image encoders with a linear projection of image patches, following ViT and ViLT. A document image is resized to $224 \times 224$, split into $16 \times 16$ patches yielding $M = 196$ patch tokens, and each patch is linearly projected to $D$ dimensions. This adds only 0.6M parameters versus 44M+ for CNN backbones. ### Unified Masking Objectives The model is pre-trained with three objectives: **Masked Language Modeling (MLM):** 30% of text tokens are masked using span masking ($\lambda = 3$). The objective minimizes: $$L\_{\text{MLM}}(\theta) = -\sum\_{l=1}^{L'} \log p\_\theta(y\_l \mid X^{M'}, Y^{L'})$$ **Masked Image Modeling (MIM):** Approximately 40% of image tokens are masked with blockwise masking. Target labels come from a discrete VAE image tokenizer (vocabulary size 8,192, initialized from DiT). The objective minimizes: $$L\_{\text{MIM}}(\theta) = -\sum\_{m=1}^{M'} \log p\_\theta(x\_m \mid X^{M'}, Y^{L'})$$ **Word-Patch Alignment (WPA):** A binary classification task predicting whether a text token's corresponding image patch is masked. Only unmasked text tokens are used: $$L\_{\text{WPA}}(\theta) = -\sum\_{l=1}^{L-L'} \log p\_\theta(z\_l \mid X^{M'}, Y^{L'})$$ The total loss is $L = L\_{\text{MLM}} + L\_{\text{MIM}} + L\_{\text{WPA}}$. The key insight is that both MLM and MIM now operate over discrete token vocabularies, making the pre-training objectives symmetric across modalities. WPA provides fine-grained cross-modal alignment by leveraging the natural correspondence between text words and their spatial image patches in documents. ### Segment-Level Layout Positions Instead of word-level bounding boxes (as in LayoutLM/LayoutLMv2), LayoutLMv3 uses segment-level layout positions where words in the same text segment share 2D coordinates. This follows StructuralLM's observation that words within a segment share semantic meaning. ## What experiments were performed? ### Benchmarks The authors evaluated on five public benchmarks spanning both text-centric and image-centric tasks: | Benchmark | Task Type | Task | Metric | |-----------|-----------|------|--------| | FUNSD | Text-centric | Form understanding (sequence labeling) | F1 | | CORD | Text-centric | Receipt key info extraction (sequence labeling) | F1 | | DocVQA | Text-centric | Document visual QA (extractive) | ANLS | | RVL-CDIP | Image-centric | Document image classification (16 classes) | Accuracy | | PubLayNet | Image-centric | Document layout analysis (object detection) | mAP@IoU[0.50:0.95] | ### Key Results (BASE models) | Model | Params | FUNSD F1 | CORD F1 | RVL-CDIP Acc | DocVQA ANLS | |-------|--------|----------|---------|--------------|-------------| | LayoutLMv2 | 200M | 82.76 | 94.95 | 95.25 | 78.08 | | DocFormer | 183M | 83.34 | 96.33 | 96.17 | -- | | **LayoutLMv3** | **133M** | **90.29** | **96.56** | **95.44** | **78.76** | ### Key Results (LARGE models) | Model | Params | FUNSD F1 | CORD F1 | RVL-CDIP Acc | DocVQA ANLS | |-------|--------|----------|---------|--------------|-------------| | LayoutLMv2 | 426M | 84.20 | 96.01 | 95.64 | 83.48 | | DocFormer | 536M | 84.55 | 96.99 | 95.50 | -- | | **LayoutLMv3** | **368M** | **92.08** | **97.46** | **95.93** | **83.37** | ### Document Layout Analysis (PubLayNet) | Model | Framework | Overall mAP | |-------|-----------|-------------| | Mask R-CNN + ResNet-101 | Mask R-CNN | 91.0 | | DiT$\_{\text{BASE}}$ | Cascade R-CNN | 94.5 | | **LayoutLMv3$\_{\text{BASE}}$** | **Cascade R-CNN** | **95.1** | LayoutLMv3 was used as a feature backbone in a Cascade R-CNN detector with FPN, extracting single-scale features from Transformer layers 4, 6, 8, and 12. It outperformed DiT (the then-concurrent vision-only document Transformer) by 0.6 mAP. ### Ablation Study The ablation (Table 3) isolated each component's contribution at BASE size, trained on 1M samples for 150K steps: - **Without image embedding** (text+layout only, MLM): strong text-centric results but cannot do layout analysis. - **Linear image embedding + MLM only:** PubLayNet loss diverges; the model cannot learn visual representations without an image-specific objective. - **+ MIM:** PubLayNet loss converges (94.38 mAP); image-centric tasks improve. - **+ WPA:** Consistent improvement across all tasks (94.43 mAP on PubLayNet, 89.78 F1 on FUNSD). This ablation is the most informative part of the paper: it demonstrates that MIM is essential for learning visual representations from linear patches, and WPA provides complementary cross-modal signal. ### Chinese Model A LayoutLMv3-Chinese model (BASE size) was pre-trained on 50M Chinese document pages, initialized from XLM-R. On the EPHOIE visual information extraction benchmark, it achieved 99.21% mean F1, outperforming all compared methods including StrucTexT (97.95%). ## What are the outcomes/conclusions? **Strengths:** - The unified masking approach (MLM + MIM over discrete tokens) is clean and well-motivated. Symmetric objectives across modalities reduce architectural complexity. - Significant parameter savings: LayoutLMv3$\_{\text{BASE}}$ (133M) outperforms LayoutLMv2$\_{\text{BASE}}$ (200M) across most benchmarks while using a simpler image encoder. - The ablation clearly isolates each component's contribution. The finding that linear patches + MLM alone causes divergence on vision tasks, while MIM resolves this, is an important practical insight. - Generality across both text-centric and image-centric tasks is well-demonstrated. **Limitations and caveats:** - The FUNSD comparison is not entirely apples-to-apples: LayoutLMv3 and StructuralLM use segment-level layout positions, while most baselines use word-level positions. The authors acknowledge this but the magnitude of the FUNSD gain (90.29 vs. 83.34 for DocFormer) likely reflects both the method and the position granularity choice. - Pre-training data is IIT-CDIP (11M images), which is English-centric scanned documents. Generalization to born-digital, multilingual, or domain-specific documents is not assessed (except for the Chinese appendix). - The image tokenizer is initialized from DiT, which itself was pre-trained on IIT-CDIP. The paper does not ablate how much the quality of the image tokenizer matters. - Layout analysis is only evaluated on PubLayNet (research papers). Performance on more diverse document types (forms, invoices, historical documents) is not tested in the vision-only setting. - No error bars or variance across runs are reported for any experiments. - DocVQA LARGE result (83.37 ANLS) slightly trails LayoutLMv2 LARGE (83.48), though the comparison is complicated by the fact that some baselines use extra data. ## Reproducibility ### Models - **LayoutLMv3$\_{\text{BASE}}$:** 12-layer Transformer, 12 heads, hidden size 768, FFN intermediate 3072. Total: 133M parameters. - **LayoutLMv3$\_{\text{LARGE}}$:** 24-layer Transformer, 16 heads, hidden size 1024, FFN intermediate 4096. Total: 368M parameters. - Image input: $3 \times 224 \times 224$, patch size $16 \times 16$, yielding 196 image tokens. - Text input: BPE tokenized, max sequence length 512. - Text embeddings initialized from RoBERTa. Image tokenizer initialized from DiT (vocabulary size 8,192). - Pre-trained checkpoints (base, large, base-chinese) are publicly available on Hugging Face under CC-BY-NC-SA-4.0. ### Algorithms - **Optimizer:** Adam, $\beta\_1 = 0.9$, $\beta\_2 = 0.98$, weight decay $1 \times 10^{-2}$. - **Pre-training:** 500K steps, batch size 2,048. - BASE: LR $1 \times 10^{-4}$, warmup 4.8% of steps. - LARGE: LR $5 \times 10^{-5}$, warmup 10%. - **MLM masking:** 30% of text tokens, span masking with Poisson $\lambda = 3$. - **MIM masking:** ~40% of image tokens, blockwise masking strategy from BEiT. - **Attention stabilization:** PB-Relax technique from CogView with $\alpha = 32$. - Distributed training, mixed precision, gradient accumulation, gradient checkpointing (for layout analysis). ### Data - **Pre-training:** IIT-CDIP Test Collection 1.0 (11M scanned document images out of available 42M pages). English-centric. Access requires a formal data use agreement through the University of California, San Francisco (UCSF) Industry Documents Library; the dataset is not freely downloadable. - **Chinese pre-training:** 50M Chinese document pages collected from publicly available digital-born documents following Common Crawl principles. No public download link or further specification is provided; this data is not reproducible without additional information. - OCR is obtained from an off-the-shelf toolkit (Microsoft Read API for fine-tuning benchmarks). This is a commercial API, which introduces a cost dependency for reproduction. - No image augmentation is applied during pre-training, consistent with prior LayoutLM models. ### Evaluation - **FUNSD:** 149 train / 50 test documents. F1 on semantic entity labeling. Fine-tuned for 1,000 steps, LR $1 \times 10^{-5}$, batch size 16. - **CORD:** 800 train / 100 val / 100 test receipts. F1 on key information extraction. Fine-tuned for 1,000 steps, LR $5 \times 10^{-5}$, batch size 64. - **DocVQA:** 10,194 train / 1,286 val / 1,287 test images. ANLS on test set (submitted to official leaderboard). BASE fine-tuned for 100K steps, LR $3 \times 10^{-5}$, batch size 128, warmup 4.8%. LARGE fine-tuned for 200K steps, LR $1 \times 10^{-5}$, batch size 32, warmup 10%. - **RVL-CDIP:** 320K train / 40K val / 40K test. Overall classification accuracy. Fine-tuned for 20,000 steps, LR $2 \times 10^{-5}$, batch size 64. OCR from Microsoft Read API. - **PubLayNet:** 335,703 train / 11,245 val. mAP@IoU[0.50:0.95] on validation set. Fine-tuned for 60,000 steps using AdamW, LR $2 \times 10^{-4}$, batch size 32, weight decay 0.05, 1,000 warmup steps. Uses Cascade R-CNN with FPN via Detectron2. No flipping or cropping augmentation. - No error bars, confidence intervals, or multi-run statistics reported for any benchmark. - FUNSD comparisons are complicated by the segment-level vs. word-level position distinction. ### Hardware - Not explicitly reported. The scale of pre-training (batch size 2,048 for 500K steps on 11M images) implies multi-GPU clusters, likely 32+ V100 or A100 GPUs, but no concrete figures are given. - No training time, GPU-hours, or cost estimates provided. ## BibTeX ```bibtex @inproceedings{huang2022layoutlmv3, title={LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking}, author={Huang, Yupan and Lv, Tengchao and Cui, Lei and Lu, Yutong and Wei, Furu}, booktitle={Proceedings of the 30th ACM International Conference on Multimedia}, pages={4083--4091}, year={2022}, doi={10.1145/3503161.3548112} } ```