# VILA: Improving Structured Content Extraction from Scientific PDFs Using Visual Layout Groups > Research note on VILA, which introduces I-VILA and H-VILA methods for injecting visual layout group structure into BERT-based models for scientific document parsing, plus the S2-VL dataset of 1,337 human-annotated pages across 19 disciplines. - Authors: Hunter Heidenreich - Published: 2026-03-16 - Paper: Preprint (arXiv) - Paper: Published (TACL 2022) - Code: VILA Code & Models - Dataset: S2-VL Dataset - Benchmark: S2-VLUE Benchmark Suite - Source: Bevaya Labs, --- ## TL;DR VILA introduces two methods for incorporating visual layout group structure (text lines and blocks) into BERT-based models for scientific PDF parsing. I-VILA inserts boundary indicator tokens and improves Macro F1 by up to 1.9%; H-VILA uses hierarchical encoding to reduce inference time by 47% with minimal accuracy loss. Both require only fine-tuning, cutting training cost by up to 95% compared to layout-aware pretraining. The paper also introduces S2-VL, a human-annotated dataset of 1,337 pages across 19 scientific disciplines. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Method}}$. The headline contribution is two novel methods (I-VILA and H-VILA) for incorporating visual layout group information into existing language models without expensive pretraining. Ablations, efficiency comparisons, and accuracy improvements are the paper's center of gravity. **Secondary:** $\Psi\_{\text{Resource}}$. The paper introduces the S2-VL dataset (1,337 human-annotated pages across 19 disciplines) and the S2-VLUE benchmark suite unifying three evaluation datasets. **Secondary:** $\Psi\_{\text{Evaluation}}$. The paper introduces the group category inconsistency metric $H(G)$ and systematically evaluates how different VILA group detectors (ground-truth, vision model, PDF parsing) affect downstream performance. ## What is the motivation? Extracting structured content from scientific PDFs (titles, abstracts, references, body text, etc.) is a critical first step for NLP over scientific papers. Layout-aware language models like LayoutLM improve accuracy by encoding each token's 2D position, but they do not explicitly model higher-level layout structures: the grouping of tokens into text lines and blocks. These models also require expensive pretraining (1,000+ GPU-hours for LayoutLM, several thousand for LayoutLMv2). Existing evaluation datasets (GROTOAP2, DocBank) rely on automatic labeling from source files, have limited domain coverage (life sciences or math/physics/CS only), and contain systematic annotation errors. No manually annotated multi-discipline evaluation set existed. ## What is the novelty? The core insight is the **group uniformity assumption**: tokens within a visual layout group (text line or text block) generally share the same semantic category. This is exploited in two ways: ### I-VILA: Layout Indicator Injection A special `[BLK]` token is inserted at each group boundary in the linearized token sequence. The resulting input has the form: $$[CLS], T\_1^{(1)}, \ldots, T\_{n\_1}^{(1)}, [BLK], T\_1^{(j+1)}, \ldots, T\_{n\_m}^{(m)}, [SEP]$$ These boundary tokens signal possible category changes between groups. When used with a layout-aware base model like LayoutLM, the `[BLK]` token inherits the 2D positional embedding of its group's bounding box. No pretraining is needed; the model learns to use these indicators during fine-tuning. ### H-VILA: Hierarchical Encoding A two-level transformer architecture: 1. **Group encoder** ($l\_g = 1$ layer): encodes tokens within each group into a single vector $\tilde{\mathbf{h}}\_j$, combined with 2D positional embeddings: $$\mathbf{h}\_j = \tilde{\mathbf{h}}\_j + p(b\_j)$$ 2. **Page encoder** ($l\_p = 12$ layers): models inter-group relationships across the full page, producing group-level predictions via an MLP classifier. Both encoders are initialized from pretrained BERT/LayoutLM weights (layer 1 for the group encoder, full model for the page encoder), so no additional pretraining is required. ### S2-VLUE Benchmark Suite Three datasets unified with VILA structure annotations: - **GROTOAP2:** 119K pages, 22 categories, life sciences, automatic labels from CERMINE PDF parsing. - **DocBank:** 498K pages, 12 categories, math/physics/CS, automatic labels from LaTeX source. VILA structures regenerated via Mask R-CNN. - **S2-VL (new):** 1,337 pages from 87 papers across 19 disciplines, 15 categories, human-annotated using the PAWLS tool. Inter-annotator agreement: 0.95 token-level accuracy. ### Group Category Inconsistency Metric A new diagnostic metric $H(G)$ measuring the entropy of predicted token categories within each group: $$H(G) = \frac{1}{m} \sum\_i^m H(g\_i), \quad H(g) = -\sum\_c p\_c \log p\_c$$ Lower values indicate more consistent predictions within groups. ## What experiments were performed? Experiments are conducted on all three S2-VLUE datasets. For S2-VL, 5-fold cross-validation is used (split by paper, not page). **I-VILA results (Table 2):** | Model | GROTOAP2 F1 | DocBank F1 | S2-VL F1 | | :--- | :--- | :--- | :--- | | LayoutLM (baseline) | 92.34 | 91.06 | 82.69 | | + I-VILA (Text Block) | **93.38** | 92.00 | 83.44 | | + I-VILA (Text Line) | 92.37 | **92.79** | **83.77** | I-VILA with text blocks also reduces group inconsistency $H(G)$ by 32.1% on GROTOAP2. **H-VILA results (Table 3):** | Model | S2-VL F1 | Inference Time (ms) | | :--- | :--- | :--- | | LayoutLM (baseline) | 82.69 | 52.56 | | H-VILA (Text Line) | 83.69 | 28.07 (47% faster) | | H-VILA (Text Block) | 82.09 | 16.37 (69% faster) | **Generalization across base models (Table 4, GROTOAP2):** I-VILA improves all tested base models: DistilBERT (+1.60), BERT (+1.53), RoBERTa (+0.88), LayoutLM (+1.04 Macro F1). **Training cost comparison (Table 5):** LayoutLM pretraining requires 1,000+ GPU-hours. VILA methods require only fine-tuning: 4.7 GPU-hours for I-VILA and 3.5 GPU-hours for H-VILA on GROTOAP2, representing a 95%+ cost reduction. **VILA group quality ablation (Table 6, S2-VL):** Using ground-truth text blocks yields the best I-VILA performance (86.50 F1) compared to vision model predictions (83.44 F1) or PDF parsing (83.95 F1). H-VILA is more sensitive to group detection errors since it cannot assign different categories within a group. ## What are the outcomes/conclusions? I-VILA provides consistent accuracy improvements (+1-2% Macro F1) and better prediction consistency across all three benchmarks, without any pretraining. H-VILA trades modest accuracy for large efficiency gains (up to 69% inference time reduction). Both methods generalize across different BERT variants. The S2-VL dataset fills a gap as the first human-annotated, multi-discipline evaluation set for scientific document parsing, covering 19 disciplines with 15 token categories. Key limitations: - VILA methods are evaluated only on scientific documents; generalization to other document types is untested. - H-VILA is sensitive to group detection quality: incorrect block boundaries propagate errors since all tokens in a group receive the same label. - S2-VL is relatively small (1,337 pages, 87 papers), limiting its use as a training set. - The group uniformity assumption does not always hold, particularly at block boundaries or when block detectors make errors. ## Reproducibility ### Models - **I-VILA:** Fine-tuned LayoutLM-BASE with `[BLK]` indicator tokens. 110M params (same as LayoutLM-BASE). - **H-VILA:** Group encoder ($l\_g = 1$, initialized from LayoutLM layer 1) + Page encoder ($l\_p = 12$, initialized from full LayoutLM). Positional embeddings from group bounding boxes. - Eight fine-tuned model checkpoints released on HuggingFace (baseline, I-VILA, H-VILA variants). ### Algorithms - Optimizer: AdamW, lr $5 \times 10^{-5}$ ($2 \times 10^{-5}$ for S2-VL), $\beta = (0.9, 0.999)$. - Linear warmup over 5% steps, then linear decay. - Batch sizes: 40 (GROTOAP2), 40 (DocBank), 12 (S2-VL). - Epochs: 24 (GROTOAP2), 6 (DocBank), 10/20 (S2-VL). - Mixed precision training. - S2-VL uses 5-fold cross-validation, split by paper. ### Data - **GROTOAP2:** 119K pages from PubMed Central. Automatic annotations. Available via the VILA repo. - **DocBank:** 498K pages from arXiv. Automatic annotations. VILA structures regenerated via Mask R-CNN. - **S2-VL:** 1,337 pages from 87 papers across 19 disciplines. Human-annotated with PAWLS. 15 categories. Inter-annotator agreement 0.95. Available via the VILA repo. ### Evaluation - Primary metric: Macro F1 (token-level). - Diagnostic metric: Group category inconsistency $H(G)$ (entropy-based). - Inference time: average over 3 runs on 1,000 GROTOAP2 test pages, single V100 GPU. - S2-VL: 5-fold cross-validation with standard deviations reported. ### Hardware - Training: 4-GPU RTX 8000 or A100 machines. - Training cost: 4.7 GPU-hours (I-VILA) to 3.5 GPU-hours (H-VILA) on GROTOAP2, compared to 1,000+ GPU-hours for LayoutLM pretraining. - Inference: single V100 GPU. I-VILA: ~53 ms/page. H-VILA (text line): ~28 ms/page. H-VILA (text block): ~16 ms/page. ## BibTeX ``` ```bibtex @article{shen2022vila, title={{VILA}: Improving Structured Content Extraction from Scientific {PDFs} Using Visual Layout Groups}, author={Shen, Zejiang and Lo, Kyle and Wang, Lucy Lu and Kuehl, Bailey and Weld, Daniel S. and Downey, Doug}, journal={Transactions of the Association for Computational Linguistics}, volume={10}, pages={376--392}, year={2022}, publisher={MIT Press}, doi={10.1162/tacl_a_00466} } ``` ```