# UDOP: Unifying Vision, Text, and Layout for Universal Document Processing > UDOP is a generative document AI foundation model that unifies vision, text, and layout modalities through a Vision-Text-Layout Transformer, reporting strong results on 8 document understanding tasks. - Authors: Bevaya AI Team - Published: 2026-03-15 - Paper: Preprint (arXiv) - Paper: Published (CVPR 2023) - Model: UDOP (Hugging Face) - Code: Code (GitHub) - Source: Bevaya Labs, --- ## TL;DR UDOP (Universal Document Processing) is a generative foundation model for document AI that unifies vision, text, and layout in a single encoder-decoder architecture. It converts all document tasks (understanding, classification, QA, layout analysis) into a sequence-to-sequence generation format, including generating document images from text and layout. UDOP ranked first on the Document Understanding Benchmark (DUE) at the time of publication and reported leading results on 8 tasks across diverse document domains. --- ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Method}}$. The headline contribution is the Vision-Text-Layout (VTL) Transformer architecture and unified generative framework that handles diverse document tasks without task-specific heads. **Secondary:** $\Psi\_{\text{Evaluation}}$. Comprehensive evaluation on 8+ benchmarks (DUE-Benchmark, FUNSD, CORD, RVL-CDIP, DocVQA) with ablation studies. ## What is the motivation? Prior document AI models (LayoutLMv2, LayoutLMv3) treat layout as shallow positional embeddings added to text tokens, underutilizing the strong spatial correlation between text content and its visual appearance. They also require task-specific heads for each downstream task (classification head for RVL-CDIP, sequence labeling head for FUNSD, etc.), limiting flexibility. UDOP addresses both issues: (1) a layout-induced representation that directly fuses text tokens with the image patches where they appear, and (2) a unified seq2seq generation framework where all tasks use the same decoder. ## What is the novelty? ### Layout-Induced Vision-Text Representation Instead of encoding text and image separately, UDOP adds each text token's embedding to the visual features of the image patch where that token is located. This creates a unified representation where each token carries both its textual semantics and its visual appearance. ### Vision-Text-Layout Transformer A T5-based encoder-decoder with: - **Modality-agnostic encoder:** Processes the fused vision-text tokens. - **Text-Layout decoder:** Generates text and bounding box tokens autoregressively. - **Vision decoder:** Reconstructs masked image patches (MAE-style). Bounding boxes are discretized into tokens and added to the text vocabulary, allowing the model to generate layout as part of its output sequence. ### Unified Pre-training Four self-supervised objectives: 1. **Joint Text-Layout Reconstruction:** Reconstruct masked text tokens and their bounding boxes. 2. **Visual Text Recognition:** Predict text from image patches (OCR-like objective). 3. **Layout Modeling:** Predict bounding boxes from text. 4. **Masked Autoencoding:** Reconstruct masked image patches. Plus 11 supervised datasets (1.8M examples) incorporated into pre-training via task prompts, including FUNSD, CORD, DocVQA, and others. ## What experiments were performed? ### DUE-Benchmark (7 tasks) UDOP achieves first place overall with an average score of 64.8, compared to 62.9 for LayoutLMv3 LARGE. Strong results on DocVQA (84.7), InfoVQA (47.4), KLC (82.8), and DeepForm (85.5). ### Additional Benchmarks | Dataset | Task | UDOP | Previous SOTA | | :--- | :--- | ---: | ---: | | FUNSD | Info Extraction (F1) | **91.62** | 88.41 (LiLT) | | CORD | Info Extraction (F1) | **97.58** | 97.40 (BROS) | | RVL-CDIP | Classification (Acc) | 96.00 | **96.08** (StructuralLM) | | DocVQA | VQA (ANLS) | **84.7** | 83.4 (LayoutLMv3) | ### Document Generation UDOP can generate document images from text and layout, enabling document editing and content customization. The authors describe this as the first document AI model with this capability, though the generated images are at MAE resolution (not publication quality). ## What are the outcomes/conclusions? The authors argue that UDOP demonstrates a unified generative framework can match or exceed task-specific models across diverse document AI tasks. The layout-induced representation is effective at exploiting the spatial correlation unique to documents. Incorporating supervised datasets into pre-training alongside self-supervised objectives provides additional gains. ### Limitations - 794M parameters (LARGE size only); no BASE variant released. - Requires OCR as preprocessing. - Document generation quality is limited to MAE reconstruction resolution. - Pre-trained on 11M IIT-CDIP pages + 1.8M supervised examples; substantial compute requirements. ## Reproducibility ### Models - **UDOP:** ~794M parameters. Based on T5 LARGE architecture with added vision encoder (ViT) and vision decoder. - Checkpoint released on Hugging Face under MIT license. - **Vision decoder weights are withheld.** The authors chose not to release the vision decoder due to ethical concerns around potential misuse for document forgery. The released checkpoint includes the encoder and text-layout decoder only. ### Algorithms - **Pre-training:** 11M IIT-CDIP pages (self-supervised) + 11 supervised datasets (1.8M examples). - **Optimizer:** Adam with $\beta\_1 = 0.9$, $\beta\_2 = 0.98$, weight decay $1 \times 10^{-2}$. - **Learning rate:** $5 \times 10^{-5}$ with 1000 warmup steps. - **Batch size:** 512. - **Curriculum:** 3-stage resolution scaling ($224 \rightarrow 512 \rightarrow 1024$), 1 epoch per stage. - **Self-supervised tasks:** Joint text-layout reconstruction, visual text recognition, layout modeling, masked autoencoding. ### Data - **Pre-training (unlabeled):** IIT-CDIP Test Collection (11M documents). - **Pre-training (labeled):** FUNSD, CORD, RVL-CDIP, DocVQA, InfoVQA, KLC, PWC, DeepForm, WTQ, TabFact, SQA (11 datasets, 1.8M examples total). ### Evaluation - **Metrics:** F1 for entity extraction tasks (FUNSD, CORD), accuracy for document classification (RVL-CDIP), ANLS for visual question answering (DocVQA, InfoVQA). - **Benchmarks:** DUE-Benchmark (7 tasks), plus standalone evaluation on FUNSD, CORD, RVL-CDIP, and DocVQA. - **Baselines:** LayoutLMv3 LARGE, LiLT, BROS, StructuralLM, and other document AI models. - **Statistical reporting:** The authors report results over 5 runs with different random seeds and include standard deviations (Tables 9-10 in the paper), which is notable for this domain. ### Hardware - Not specified in the main paper. A model of this scale (794M params with vision encoder) typically requires multi-GPU training on A100-class hardware. ## BibTeX ```bibtex @inproceedings{tang2023udop, title={Unifying Vision, Text, and Layout for Universal Document Processing}, author={Tang, Zineng and Yang, Ziyi and Wang, Guoxin and Fang, Yuwei and Liu, Yang and Zhu, Chenguang and Zeng, Michael and Zhang, Cha and Bansal, Mohit}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages={15254--15264}, year={2023}, doi={10.1109/CVPR52729.2023.01462} } ```