# DocFormerv2: Local Features for Document Understanding > Research notes on DocFormerv2, a multimodal encoder-decoder transformer for visual document understanding with asymmetric pre-training tasks that encourage local feature alignment. - Authors: Hunter Heidenreich - Published: 2026-03-13 - Updated: 2026-03-15 - Paper: Preprint (arXiv) - Paper: Published (AAAI 2024) - Source: Bevaya Labs, --- ## TL;DR DocFormerv2 is a multimodal encoder-decoder transformer for Visual Document Understanding (VDU) that introduces two encoder pre-training tasks (Token-to-Line and Token-to-Grid) designed to encourage local semantic alignment between text, vision, and spatial modalities. Pre-trained on 64M document pages from the Industrial Document Library (IDL), it reports competitive results across eight benchmarks spanning table VQA, document VQA, entity extraction, sequence labeling, and scene-text VQA. Published at AAAI 2024. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Method}}$ The headline contribution is the architecture (encoder-decoder multimodal transformer with simplified visual branch) and the asymmetric pre-training recipe (two novel encoder tasks + decoder language modeling). The paper devotes most of its pages to ablations proving these design choices drive gains. **Secondary:** None. The paper does not release datasets, models, or benchmarks; the ablations serve the method rather than constituting an independent measurement contribution. ## What is the motivation? Visual Document Understanding requires reasoning over three modalities simultaneously: visual appearance, text content, and spatial layout. Prior multimodal document models were predominantly encoder-only (LayoutLM, LayoutLMv2, BROS) or used encoder-decoder architectures with only a single language modeling pre-training task (TILT). The authors argue that: 1. Most VDU tasks require **local, layout-relative** understanding (e.g., a "1" at the top-right is a page number, while in a table it is a quantity). 2. Prior approaches relied on heavyweight visual encoders (ResNeXt, Faster R-CNN, Swin) that add complexity without necessarily helping on document images, which are visually simpler than natural scenes. 3. Existing pre-training objectives do not explicitly teach local multi-modal feature alignment. ## What is the novelty? ### Asymmetric Pre-training DocFormerv2 uses three pre-training tasks applied asymmetrically: two on the encoder and one on the decoder. **Token-to-Line (encoder):** Given two randomly selected text tokens, predict the quantized line distance between them. Labels are $\\{0, 1, 2\\}$, where 2 captures all pairs more than 2 lines apart. For tokens on lines $a, b, c, d$: $$F(a, d) = 0;\; F(a, b) = 1;\; F(b, c) = 2$$ This teaches the model relative positional relationships between tokens. Ablations show +2.2% on DocVQA. **Token-to-Grid (encoder):** The document is virtually divided into an $m \times n$ grid. Each OCR token is assigned to a grid cell based on its top-left coordinate: $$g\_i = \lfloor \frac{x\_i}{\Delta\_x} \rfloor + \lfloor \frac{y\_i}{\Delta\_y} \rfloor \cdot m$$ where $\Delta\_x$ and $\Delta\_y$ are cell dimensions. This pairs language semantics with spatial location. A 4$\times$4 grid was found optimal. **Decoder Language Modeling:** Standard T5-style denoising masked language modeling, with the additional constraint that spatial features of masked tokens are also masked, forcing the model to infer position from context. The final pre-training loss combines all three: $$L\_{\text{final}} = k \cdot L\_{\text{tol}} + l \cdot L\_{\text{tog}} + m \cdot L\_{\text{dlm}}$$ ### Simplified Visual Branch Instead of using a pretrained CNN or object detector for visual features, DocFormerv2 uses a single convolutional layer followed by a linear projection: $$V = \text{linear}(\text{conv}\_{2 \times 2}(v))$$ with randomly initialized weights. Ablations show this outperforms Swin Transformer by +4.3% on DocVQA, which the authors attribute to documents having extensive white space that a linear downsampling layer can efficiently compress. ### Architecture Variants | Variant | dim | ff | Heads | Layers (E, D) | Params | | :--- | :---: | :---: | :---: | :---: | :---: | | Small | 512 | 2048 | 8 | 6, 6 | 66M | | Base | 768 | 3072 | 12 | 12, 12 | 232M | | Large | 1024 | 4096 | 16 | 24, 24 | 750M | ## What experiments were performed? DocFormerv2 is evaluated on eight datasets across four task categories: ### Table VQA - **TabFact:** 83.2% accuracy (+4.3% over UDOP) ### Document VQA - **DocVQA:** 87.84 ANLS (+0.79% over TILT, with extra VQA data) - **InfographicsVQA:** 48.8 ANLS (+1.4% over UDOP) ### Sequence Labeling / Entity Extraction - **FUNSD:** 88.89 F1 (competitive; models using entity-box priors like LayoutLMv3 score higher but are not directly comparable) - **CORD:** 97.70 F1 (competitive with UDOP which uses entity-box priors) ### Scene-Text VQA (Generalization) Fine-tuned directly from document pre-trained weights with no image-text pre-training: - **OCR-VQA:** 71.5% accuracy (+3.4% over GIT) - **TextVQA:** 64.0% accuracy (+2.4% over LaTr) - **ST-VQA:** 71.8 ANLS (+2.2% over LaTr) On the TextVQA validation set, DocFormerv2 (750M parameters) outperforms Flamingo (80B, +9.9%), PaLi-3B (+6.8%), and PaLi-15B (+1.5%), though those models were trained on substantially larger and more diverse data. ### Ablations - **Pre-training tasks:** Each task contributes cumulatively. Baseline + Vision + Line + Grid yields +4.0% on DocVQA over the baseline (small model, 1M pre-training docs). - **Pre-training data:** DFv2 base outperforms LayoutLMv2 base when both use 11M pre-training documents, and improves further with 64M. - **OCR noise robustness:** 20% character error rate causes only -1.68% F1 drop on FUNSD (vs. -9.84% for LayoutLMv2), demonstrating the benefit of the generative decoder. - **Vision tokens:** 128 image tokens is optimal. - **Grid size:** 4$\times$4 is optimal for Token-to-Grid; too fine (8$\times$8, 12$\times$12) or too coarse (4$\times$1) hurts. - **Image encoder:** Linear projection outperforms Swin Transformer v2 by +4.3% on DocVQA. ## What are the outcomes/conclusions? The results suggest that local feature alignment through carefully designed pre-training tasks may matter more for VDU than complex visual encoders. The two encoder tasks (Token-to-Line, Token-to-Grid) provide complementary benefits: line-level relative positioning for nearby token reasoning, and grid-level spatial awareness for document region understanding. **Limitations:** - No code or model weights have been released (AWS AI Labs), so results cannot be independently reproduced. - The IDL pre-training corpus (64M pages) is substantially larger than what most baselines used (11M for LayoutLMv2/v3), making direct comparisons imperfect despite the ablation controlling for data size. - FUNSD and CORD comparisons are complicated by differing use of entity-box priors across methods. - The paper does not report training cost, GPU hours, or inference latency, making practical deployment assessment difficult. - Scene-text VQA results, while notable as a generalization test, compare against models not specifically designed for that domain. ## Reproducibility ### Models - Three variants: small (66M), base (232M), large (750M) - Encoder-decoder transformer with sentence-piece tokenizer - Visual branch: single $2 \times 2$ convolution + linear projection (randomly initialized) - Spatial features: four learnable embedding layers for $x$, $y$, height, width - Modality embeddings for text and vision - **No weights released.** No model cards or checkpoints available. ### Algorithms - Pre-training: three concurrent losses (Token-to-Line, Token-to-Grid, denoising LM) with empirically determined weights $k, l, m$ (values not specified in the paper) - Fine-tuning: no dataset-specific hyperparameter tuning reported (authors note this may leave performance on the table) - Optimizer, learning rate, batch size, and training schedule: not specified in the main paper or supplemental - Maximum sequence length $s$ is used but the value is not reported - Implementation: PyTorch + HuggingFace Transformers (no official code released; a [third-party reimplementation](https://github.com/uakarsh/docformerv2) exists under Apache-2.0 but is incomplete/WIP) ### Data - **Pre-training:** [Industrial Document Library (IDL)](https://www.industrydocuments.ucsf.edu/) from UCSF Industry Documents. 13M documents (70M pages) collected, filtered to 64M pages after cleaning and pruning ~6M documents. Data distribution referenced in supplemental (not in main paper). The IDL is publicly accessible, but the specific filtering pipeline and OCR extraction process are not described in reproducible detail. - **Fine-tuning:** Standard public benchmarks (DocVQA, InfoVQA, FUNSD, CORD, TabFact, OCR-VQA, TextVQA, ST-VQA). Some experiments use an additional 850K document VQA question-answer pairs (source not specified). ### Evaluation - Metrics: ANLS (DocVQA, InfoVQA, ST-VQA), Accuracy (TabFact, OCR-VQA, TextVQA), F1 (FUNSD, CORD) - Baselines: LayoutLMv1/v2/v3, TILT, UDOP, BROS, DocFormerv1, GIT/GIT2, PaLi, Flamingo, LaTr, and others - FUNSD/CORD comparisons note that some baselines use entity-box priors (line-level bounding boxes during fine-tuning), which is not directly comparable - No error bars, significance tests, or multi-run statistics reported ### Hardware - Not specified. No GPU type, count, training time, or cost information provided. ## BibTeX ```bibtex @inproceedings{appalaraju2024docformerv2, title={DocFormerv2: Local Features for Document Understanding}, author={Appalaraju, Srikar and Tang, Peng and Dong, Qi and Sankaran, Nishant and Zhou, Yichu and Manmatha, R.}, booktitle={Proceedings of the AAAI Conference on Artificial Intelligence}, volume={38}, number={2}, pages={709--718}, year={2024}, doi={10.1609/aaai.v38i2.27828} } ```