# VGT: Vision Grid Transformer for Document Layout Analysis > VGT introduces a two-stream Vision Grid Transformer with a pre-trained Grid Transformer (GiT) for document layout analysis, alongside D4LA, a 27-class diverse benchmark sourced from RVL-CDIP. - Authors: Bevaya AI Team - Published: 2023-08-29 - Updated: 2026-03-12 - Paper: Preprint (arXiv) - Paper: Published (ICCV 2023) - Code: VGT (AdvancedLiterateMachinery) - Dataset: D4LA - Source: Bevaya Labs, --- ## TL;DR VGT is a two-stream multimodal architecture for document layout analysis that pairs a Vision Transformer (ViT) for image features with a Grid Transformer (GiT) for 2D text grid features. GiT is pre-trained on ~4 million IIT-CDIP document images using two objectives: token-level Masked Grid Language Modeling (MGLM) and segment-level contrastive Segment Language Modeling (SLM). The paper also introduces D4LA, a 27-class, 11,092-image benchmark spanning 12 real-world document types from RVL-CDIP. At the time of publication, VGT set new results on PubLayNet (96.2% mAP), DocBank (84.1% mAP), and D4LA (68.8% mAP). --- ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Method}}$. The core contribution is the VGT architecture: a two-stream ViT + GiT design where GiT is pre-trained with novel objectives (MGLM and SLM) specifically for 2D text grid understanding in document layout analysis. **Secondary:** $\Psi\_{\text{Resource}}$. The paper introduces D4LA (Diverse and Detailed Dataset for Document Layout Analysis), a manually annotated benchmark with 27 categories across 12 real-world document types. --- ## What is the motivation? Document layout analysis (DLA) sits at an awkward intersection of two paradigms, and neither makes full use of what is available in documents. **Pure visual methods** (CNN-based detectors, DiT) treat the page as an image and leverage pre-training well (DiT uses image-based BEiT-style pre-training), but they ignore the text content that is almost always present and semantically informative for distinguishing layout categories. **Grid-based methods** (VSR, sentence-grid methods) construct a 2D representation of text tokens aligned with the image and feed both streams into a detector. They use textual information at inference time, but do so without any textual pre-training objective. The grid branch is trained only on visual detection supervision, which limits how well it learns semantic text representations. No existing approach at the time combined: (1) multi-modal inputs with both image and text grid, and (2) dedicated pre-training for the text grid stream. In parallel, existing large-scale DLA datasets (PubLayNet with 5 classes, DocBank with 13 classes) are almost entirely scientific papers. The layout categories they define are poorly suited to real-world document types like invoices, letters, forms, and budget sheets, which have their own distinct structural elements. --- ## What is the novelty? ### Two-Stream Architecture: ViT + GiT VGT runs two parallel ViT-Base encoders on the same document: - **ViT stream:** Encodes the document image as patch embeddings with learnable 1D position embeddings, following DiT [25] initialization. - **GiT stream:** Encodes a 2D text grid $\mathbf{G} \in \mathbb{R}^{H \times W \times C\_G}$, where each pixel position is filled with the sub-word token embedding of whichever OCR token covers that pixel: $$G\_{i,j} = \begin{cases} E(c\_k) & \text{if } (i,j) \in \mathbf{b}\_k \\\\ E(\texttt{[PAD]}) & \text{otherwise} \end{cases}$$ Here $c\_k$ is the $k$-th sub-word token with bounding box $\mathbf{b}\_k$, and $E(\cdot)$ maps tokens to a $C\_G = 64$ dimensional embedding space initialized from LayoutLM. Background pixels with no text are assigned the [PAD] token embedding. The grid is then split into patches and processed identically to ViT. Multi-scale features from both streams are fused element-wise at each FPN scale $i$: $$\mathbf{Z}\_i = \mathbf{V}\_i \oplus \mathbf{S}\_i$$ where $\mathbf{V}\_i$ and $\mathbf{S}\_i$ are the ViT and GiT feature maps at scale $i$, and $\oplus$ is element-wise summation. The fused pyramid is passed through FPN and a Cascade R-CNN detection head. ### Pre-Training for GiT: MGLM and SLM The authors decouple visual and language pre-training: ViT is initialized from DiT-base, while GiT is separately pre-trained on ~4 million IIT-CDIP document images using two objectives. **Masked Grid Language Modeling (MGLM)** adapts the BERT MLM objective to the 2D grid space. Some tokens in $\mathbf{G}$ are replaced with [MASK], and the model must recover the original token identities. Because GiT produces 2D feature maps rather than 1D sequences, token features are extracted via RoIAlign on the FPN output rather than direct index lookup. The objective is: $$\mathcal{L}\_{\text{MGLM}}(\theta) = -\sum\_{k=1}^{N\_M} \log p\_{\theta}(c\_k \mid \mathbf{e}\_{c\_k})$$ where $N\_M$ is the number of masked tokens, $\theta$ is the parameters of GiT and FPN, and $\mathbf{e}\_{c\_k}$ is the RoIAlign-extracted feature for token $c\_k$. **Segment Language Modeling (SLM)** targets segment-level (text-line-level) representations via contrastive learning. For each text line segment $l\_i$, the model produces a segment feature $\mathbf{e}\_{l\_i}$ by RoIAlign. A frozen LayoutLM model generates pseudo-target features $\mathbf{e}^\ast\_{l\_i}$. The objective aligns segment features with their pseudo-targets: $$p\_{\theta}(\mathbf{e}\_{l\_i}, \mathbf{e}^{\ast}\_{l\_i}) = \frac{\exp(\mathbf{e}\_{l\_i} \cdot \mathbf{e}^{\ast}\_{l\_i} / \tau)}{\exp(\mathbf{e}\_{l\_i} \cdot \mathbf{e}^{\ast}\_{l\_i} / \tau) + \sum\_{k \in \mathcal{N}\_{l\_i}} \exp(\mathbf{e}\_{l\_i} \cdot \mathbf{e}^{\ast}\_{k} / \tau)}$$ $$\mathcal{L}\_{\text{SLM}}(\theta) = -\frac{1}{N\_S} \sum\_{i=1}^{N\_S} \log p\_{\theta}(\mathbf{e}\_{l\_i}, \mathbf{e}^{\ast}\_{l\_i})$$ where $\cdot$ denotes cosine similarity, $\tau = 0.01$ is a temperature parameter, $\mathcal{N}\_{l\_i}$ is the set of negative samples for segment $l\_i$, and $N\_S$ is the number of sampled segments per page. Both losses are combined equally: $\mathcal{L} = \mathcal{L}\_{\text{MGLM}} + \mathcal{L}\_{\text{SLM}}$. ### D4LA Dataset D4LA (Diverse and Detailed Dataset for Document Layout Analysis) addresses the scientific-paper bias in existing DLA benchmarks. Key properties: - **Images:** 11,092 pages from RVL-CDIP across 12 document types (Budget, Email, Form, Invoice, Letter, Memo, News Article, Presentation, Resume, Scientific Publication, Scientific Report, Specification). Noisy, handwritten, and low-text images are filtered. - **Annotation:** Manual, with OCR from IIT-CDIP as a textual resource. - **Categories:** 27 classes designed for real-world DLA: DocTitle, ListText, LetterHead, Question, RegionList, TableName, FigureName, Footer, Number, ParaTitle, RegionTitle, LetterDear, OtherText, Abstract, Table, Equation, PageHeader, Catalog, ParaText, Date, LetterSign, RegionKV, Author, Figure, Reference, PageFooter, PageNumber. Notably, RegionKV (key-value regions in invoices) and RegionList (wireless form lists) are absent from academic-paper-focused datasets. - **Splits:** 8,868 train / 2,224 validation. --- ## What experiments were performed? ### Benchmarks - **PubLayNet:** 360K scientific PDFs, 5 classes (Text, Title, List, Table, Figure). Trained on 335,703, evaluated on the 11,245-image validation split. - **DocBank:** 500K document pages with 13 region-level categories. Trained on 400K, evaluated on the 5K validation split. - **D4LA:** 8,868 train / 2,224 validation, 27 categories across 12 document types. For ablation efficiency, two sub-datasets PubLayNet2K and DocBank2K (2K train / 2K validation each) were used for early module comparisons. ### Baselines - **ResNeXt-101:** CNN-based Cascade R-CNN, no document-specific pre-training. - **DiT-Base:** ViT-based DLA model with BEiT-style document image pre-training. Visual only. - **LayoutLMv3-Base:** Multi-modal pre-training with unified text and image masking, but only image tokens are used at DLA fine-tuning time (no text embeddings passed to the detector head). - **VSR:** Two-stream CNN model using char-level and sentence-level text grids. Multi-modal but no pre-training for the text stream. ### Evaluation - Primary metric: mAP at IoU[0.50:0.95] (COCO-style bounding box AP). - Detection framework: Cascade R-CNN with FPN, implemented in Detectron2. ### Ablation Studies **Table 5** (on PubLayNet2K / DocBank2K) isolates contributions: - ViT alone (baseline): 86.92 / 59.61 - GiT alone with only layout (no text): 64.12 / 40.56, showing layout information alone is insufficient - GiT alone with text, no pre-training: 65.88 / 49.15, showing text helps, especially for semantic DocBank categories - GiT alone with pre-training (MGLM+SLM): 74.96 / 55.46, showing pre-training substantially improves GiT - ViT + GiT (LayoutLM embeddings, no pre-training): 87.76 / 64.01, showing multi-modal fusion helps over ViT alone - Full VGT (ViT + GiT + pre-training): 88.44 / 65.94, the best result, verifying all components contribute **Table 6** (DocBank2K) tests pre-training objectives in isolation: - No pre-training: 64.01 - MGLM only: 64.54 - SLM only: 65.11 - MGLM + SLM: 65.17 (SLM alone contributes more than MGLM, and combining both yields the best result) --- ## What are the outcomes/conclusions? ### Results **PubLayNet (Table 7):** | Model | Text | Title | List | Table | Figure | mAP | |---|---|---|---|---|---|---| | ResNeXt-101 | 93.0 | 86.2 | 94.0 | 97.6 | 96.8 | 93.5 | | DiT-Base | 94.4 | 88.9 | 94.8 | 97.6 | 96.9 | 94.5 | | LayoutLMv3-Base | 94.5 | 90.6 | 95.5 | 97.9 | 97.0 | 95.1 | | VSR | 96.7 | 93.1 | 94.7 | 97.4 | 96.4 | 95.7 | | VGT | 95.0 | **93.9** | **96.8** | **98.1** | **97.1** | **96.2** | VGT improves on VSR's prior result (95.7%) by 0.5%, with notable gains in Title and List categories, which the authors attribute to GiT's text-aware features. **DocBank (Table 8):** | Model | mAP | |---|---| | ResNeXt-101 | 77.4 | | DiT-Base | 79.6 | | LayoutLMv3-Base | 78.3 | | VGT | **84.1** | VGT's 4.5-point gain over DiT-Base is most pronounced in text-semantic categories: Caption (+5.7), Date (+5.7), Equation (+8.9), List (+5.9), Paragraph (+8.5). This confirms that GiT pre-training specifically benefits classes that require understanding of text content. **D4LA (Table 9):** | Model | mAP | |---|---| | ResNeXt-101 | 65.1 | | DiT-Base | 67.7 | | LayoutLMv3-Base | 60.5 | | VGT | **68.8** | LayoutLMv3 underperforms even ResNeXt-101 on D4LA, which the authors attribute to the disconnect between multi-modal pre-training and visual-only inference. VGT's gain over DiT-Base (+1.1 overall, +6.6 on Abstract, +8.9 on Question) demonstrates the benefit of using textual features at inference time. ### Limitations - **Model size:** VGT has 243M parameters, significantly larger than DiT-Base (138M) and LayoutLMv3 (138M). The two-stream design adds compute proportionally. - **Inference speed:** VGT takes 460ms per image, compared to 210ms for DiT-Base and 270ms for LayoutLMv3. This makes it unsuitable for latency-sensitive applications. - **OCR dependency:** GiT requires word bounding boxes and tokens as input. For PDFs this comes from a parser (PDFPlumber); for scanned images it requires an external OCR engine. OCR errors propagate to the text grid. - **Image-centric scope:** VGT is primarily an object detection system. The authors note that extending it to text-centric tasks like information extraction is future work. --- ## Reproducibility ### Models - Both ViT and GiT are 12-layer ViT-Base encoders (12-head attention, D=768, 3,072 MLP size, patch size 16). - ViT initialized from published DiT-base weights. GiT initialized from DiT-base weights and then further pre-trained with MGLM+SLM. - GiT text embeddings initialized from LayoutLM with embedding dimension reduced to $C\_G = 64$. - Total model: 243M parameters. - Detection head: Cascade R-CNN with FPN, implemented in Detectron2. All other Cascade R-CNN settings follow DiT. - Code and models stated as publicly available at: https://github.com/AlibabaResearch/AdvancedLiterateMachinery. Licensed under Apache-2.0. ### Algorithms **GiT Pre-training:** - Dataset: ~4 million images from IIT-CDIP. - Optimizer: Adam, batch size 96, 150,000 steps. - Learning rate: 5e-4 with 2% linear warmup. - Image size: 768 x 768. - SLM: 64 segments sampled per page; LayoutLM used as the pseudo-target generator; temperature $\tau = 0.01$. **Fine-tuning:** - Optimizer: AdamW with 1,000 warmup steps, LR=2e-4. - DocBank: 200,000 steps, batch size 24. - PubLayNet: 120,000 steps, batch size 24. - D4LA: 10,000 steps, batch size 12. ### Data - **IIT-CDIP (~4M images):** Used for GiT pre-training. This is a tobacco litigation archive dataset. OCR results from IIT-CDIP are reused for D4LA annotation as well. - **RVL-CDIP:** Source of D4LA images. RVL-CDIP is a subset of IIT-CDIP organized into 16 document categories; D4LA draws 12 of those. No explicit license for RVL-CDIP images is stated in this paper; the underlying IIT-CDIP collection was assembled for litigation and carries no open data grant. Treat the D4LA image license as unknown. - **D4LA:** 8,868 train / 2,224 validation (split from 11,092 total). Manually annotated. Available on ModelScope: https://modelscope.cn/datasets/damo/D4LA/summary. License listed as Unknown (the ModelScope page states the data will be made publicly available but provides no explicit license terms). - **PubLayNet and DocBank:** Standard benchmarks used as-is. ### Evaluation - Metric: mAP at IoU[0.50:0.95] (COCO-style bounding box detection AP). - PubLayNet and DocBank results are on the validation splits. - No error bars or multi-run statistics are reported. - No data contamination analysis is described (PubLayNet/DocBank images come from scientific paper repositories; overlap with IIT-CDIP pre-training data is not discussed). ### Hardware - Training hardware: not explicitly stated in the paper. The pre-training scale (~4M images, 150K steps, batch 96) and fine-tuning scale suggest multi-GPU setup. - Inference speed: VGT 460ms, DiT-Base 210ms, LayoutLMv3 270ms (comparison hardware not stated). - Total GPU-hours and cloud cost are not reported. - CPU-only or consumer GPU feasibility is not discussed. --- ## BibTeX ```bibtex @InProceedings{Da_2023_ICCV, author={Da, Cheng and Luo, Chuwei and Zheng, Qi and Yao, Cong}, title={Vision Grid Transformer for Document Layout Analysis}, booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)}, month={October}, year={2023}, pages={19462--19472}, doi={10.1109/ICCV51070.2023.01783} } ```