# LiLT: A Simple yet Effective Language-Independent Layout Transformer > LiLT disentangles text and layout into parallel Transformer streams, enabling pre-training on English-only documents and fine-tuning with any monolingual or multilingual text encoder. - Authors: Bevaya AI Team - Published: 2026-03-15 - Paper: Preprint (arXiv) - Paper: Published (ACL 2022) - Code: LiLT (GitHub) - Model: LiLT Base (RoBERTa-EN) - Model: LiLT Base (InfoXLM) - Source: Bevaya Labs, --- ## TL;DR LiLT (Language-independent Layout Transformer) is a dual-stream Transformer that separates text and layout processing into parallel flows connected by a Bi-directional Attention Complementation Mechanism (BiACM). The key insight: pre-train the layout stream on English-only structured documents (IIT-CDIP), then swap in any off-the-shelf text encoder (monolingual or multilingual) at fine-tuning time. This avoids the costly multilingual pre-training corpus that LayoutXLM requires while achieving competitive or superior results on 8 languages. --- ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Method}}$. The core contribution is the disentangled dual-stream architecture and BiACM mechanism that enables language-independent layout pre-training. **Secondary:** $\Psi\_{\text{Evaluation}}$. Comprehensive benchmarking on FUNSD, CORD, EPHOIE, RVL-CDIP, and the multilingual XFUND benchmark across 8 languages. ## What is the motivation? LayoutXLM demonstrated that multilingual document understanding requires both layout and text signals. However, it requires scraping 22M multilingual PDFs, cleaning them, and pre-training a 369M+ parameter model from scratch. LiLT asks: can we learn layout structure from English-only documents and transfer that knowledge to other languages by simply swapping the text encoder? The key observation is that document layout structure is largely language-independent. The spatial arrangement of fields on a form looks similar whether the text is in English, Chinese, or Arabic. If the layout knowledge can be separated from the language knowledge, each can be learned independently. ## What is the novelty? ### Dual-Stream Architecture LiLT uses two parallel Transformer streams: - **Text flow:** Processes token embeddings + 1D positional embeddings. Standard Transformer architecture. - **Layout flow:** Processes 2D positional (bounding box) embeddings only. Reduced hidden size for efficiency (adds only 6.1M parameters to the text flow). ### BiACM (Bi-directional Attention Complementation Mechanism) At each layer, the attention weights from each stream are added to the other stream's attention computation before the softmax: $$A\_T' = \text{Softmax}(\frac{Q\_T K\_T^T}{\sqrt{d\_T}} + \text{DETACH}(\frac{Q\_L K\_L^T}{\sqrt{d\_L}}))$$ $$A\_L' = \text{Softmax}(\frac{Q\_L K\_L^T}{\sqrt{d\_L}} + \text{DETACH}(\frac{Q\_T K\_T^T}{\sqrt{d\_T}}))$$ The DETACH operation (gradient stopping during pre-training) prevents the layout flow from corrupting the text encoder's representations. At fine-tuning time, DETACH is removed to allow full end-to-end optimization. ### Pre-training Objectives Three objectives, all operating on text tokens: 1. **MVLM:** Standard masked visual-language modeling. 2. **KPL (Key Point Location):** Predict the top-left and bottom-right corner coordinates of masked bounding boxes. 3. **CAI (Cross-modal Alignment Identification):** Binary classification of whether each text-layout pair is correctly aligned or randomly shifted. ### Asynchronous Optimization The text flow's learning rate is slowed down by a factor of 1000x relative to the layout flow during pre-training. This prevents the text encoder from overfitting to English while allowing the layout encoder to fully learn spatial structure. ## What experiments were performed? ### Monolingual Benchmarks | Dataset | LiLT[EN-R] BASE | LayoutLMv2 BASE | LayoutXLM BASE | | :--- | ---: | ---: | ---: | | FUNSD (F1) | **0.8841** | 0.8276 | 0.8034 | | CORD (F1) | 0.9607 | 0.9495 | 0.9481 | | RVL-CDIP (Acc) | 95.68% | 95.25% | 95.21% | LiLT with English RoBERTa reports the highest F1 among the compared baselines on FUNSD and competitive results on CORD and RVL-CDIP, outperforming both LayoutLMv2 and LayoutXLM at BASE size. Note that on CORD, DocFormer BASE (0.9633 F1) outperforms LiLT; the table above shows only the most relevant layout-aware baselines. ### Multilingual (XFUND) The paper evaluates three settings on XFUND: (1) language-specific fine-tuning (train and test on the same language), (2) zero-shot transfer (train on English, test on target language), and (3) multitask fine-tuning (train on all languages jointly). On zero-shot transfer, LiLT with InfoXLM achieves competitive performance with LayoutXLM despite using only English pre-training data. On language-specific fine-tuning, LiLT often matches or exceeds LayoutXLM. ## What are the outcomes/conclusions? LiLT demonstrates that layout structure is sufficiently language-independent that it can be learned from a single language and transferred to others. The disentangled architecture enables a practical workflow: pre-train layout once on English, then combine with any text encoder for the target language. The BiACM mechanism is critical. Without it (plain concatenation), performance drops sharply. Replacing it with co-attention (a common dual-stream approach) also hurts performance, because deeper cross-modal interaction damages the text flow's consistency during pre-training. ### Limitations - No visual features (no CNN backbone). LiLT uses only text + layout, unlike LayoutLMv2/LayoutXLM which also use document images. - Only BASE size evaluated; no LARGE variant. - Pre-trained on IIT-CDIP only (English); adding multilingual layout data could further help. ## Reproducibility ### Models - **LiLT BASE:** 12-layer layout encoder with 192 hidden size, 768 FFN size, 12 attention heads (6.1M parameters). Combined with a text encoder at fine-tuning time. - **Released checkpoints:** lilt-roberta-en-base (293 MB, English), lilt-infoxlm-base (846 MB, multilingual), and lilt-only-base (21 MB, layout flow only). Available on both GitHub (OneDrive links) and HuggingFace under MIT license. - The text flow is initialized from an existing pre-trained model (RoBERTa BASE or InfoXLM BASE) and can be swapped at fine-tuning time. ### Algorithms - **Pre-training:** Adam optimizer ($\beta\_1 = 0.9$, $\beta\_2 = 0.999$), learning rate $2 \times 10^{-5}$ with linear warmup over the first 10% of steps followed by linear decay, weight decay $1 \times 10^{-2}$, batch size 96, 5 epochs on IIT-CDIP. - **Text flow slow-down ratio:** 1000x (text learning rate = base rate / 1000). - **Max sequence length:** 512. ### Data - **Pre-training:** IIT-CDIP Test Collection 1.0 (English only): 6M+ documents, 11M+ scanned images. OCR via TextIn API. - **Ablation pre-training:** 2M randomly sampled documents from IIT-CDIP (5 epochs). - **Fine-tuning benchmarks:** FUNSD (149/50 train/test forms), CORD (800/100/100), EPHOIE (1183/311 Chinese exam papers), RVL-CDIP (320K/40K/40K), XFUND (149/50 per language, 7 languages + English FUNSD). ### Evaluation - **Metrics:** Entity-level F1 for SER (FUNSD, CORD, EPHOIE, XFUND); overall accuracy for document classification (RVL-CDIP); F1 for relation extraction (XFUND RE). - **Baselines:** LayoutLM, LayoutLMv2, LayoutXLM, BROS, DocFormer, and text-only models (BERT, RoBERTa, InfoXLM). - **Statistical rigor:** No error bars, confidence intervals, or multi-run statistics reported. ### Hardware - 4x NVIDIA A40 (48 GB) GPUs for pre-training. - Training time not reported. ## BibTeX ```bibtex @inproceedings{wang2022lilt, title={LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understanding}, author={Wang, Jiapeng and Jin, Lianwen and Ding, Kai}, booktitle={Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, pages={7747--7757}, year={2022}, doi={10.18653/v1/2022.acl-long.534} } ```