# VTLayout: Fusion of Visual and Text Features for Document Layout Analysis > Research note on VTLayout, a two-stage DLA model that fuses deep visual, shallow visual, and text features to improve category classification on PubLayNet. - Authors: Bevaya AI Team - Published: 2026-03-14 - Paper: Preprint (arXiv) - Paper: Published (PRICAI 2021) - Source: Bevaya Labs, --- ## TL;DR VTLayout is a two-stage document layout analysis model. The first stage uses Cascade Mask R-CNN for region localization, and the second stage re-classifies those regions by fusing three feature types: deep visual features (MobileNetV2), shallow visual features (pixel histogram statistics), and text features (TF-IDF over OCR output). On PubLayNet, the model reports an overall F1 of 0.9599, with the largest gains on List and Title categories where pure detection models struggle. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Method}}$. The headline contribution is a two-stage architecture that decouples localization from classification and introduces a multimodal fusion strategy combining three distinct feature extractors (DVFE, SVFE, TFE) with a Squeeze-and-Excitation weighting mechanism. **Secondary:** None. The paper does not release code, data, or models. ## What is the motivation? Standard object detection models (Faster R-CNN, Mask R-CNN) applied to DLA perform well at localizing regions but often misclassify visually similar categories, particularly List vs. Text and Title vs. Text. The authors observe that: 1. Text-format categories (List, Text, Title) share similar deep visual features, making them hard to distinguish with a detection backbone alone. 2. Shallow statistical features (pixel value histograms) exhibit distinct distributions across categories. 3. Text content (e.g., short vs. long runs of text) carries discriminative signal that vision-only models ignore. The paper argues that fusing these three complementary feature types can substantially improve classification accuracy without changing the underlying detector. ## What is the novelty? The core contribution is a two-stage pipeline: **Stage 1 (Localization):** Cascade Mask R-CNN with ResNeXt-101-64x4d backbone detects and localizes all category blocks. This stage handles bounding box regression and initial classification. **Stage 2 (Re-classification):** Three parallel feature extractors process each detected region: 1. **Deep Visual Feature Extractor (DVFE):** MobileNetV2 extracts CNN features from each cropped region (resized to $128 \times 128$). A Squeeze-and-Excitation (SE) block re-calibrates channel-wise feature responses. 2. **Shallow Visual Feature Extractor (SVFE):** Converts each region to grayscale and computes a 256-dimensional pixel histogram (count of pixels at each intensity value 0-255). This captures intuitive differences: Title blocks tend to have bold/dark pixel distributions distinct from body Text. 3. **Text Feature Extractor (TFE):** PaddleOCR extracts text from each region (with 8$\times$ enlargement for small Title blocks), then TF-IDF produces a feature vector capturing word-level statistics. The SVFE and TFE vectors are concatenated and passed through a 4-layer fully connected network (512, 256, 128, 64 neurons). The output is concatenated with the SE-weighted DVFE features, then classified via softmax. The loss function for the classification stage is cross-entropy: $$\mathcal{L} = -\sum\_{i=1}^{C} y\_i \log(\hat{y}\_i)$$ where $C = 5$ is the number of PubLayNet categories. ## What experiments were performed? All experiments use the **PubLayNet** dataset (335,703 training images, 11,245 validation images; 5 categories: Text, Title, List, Table, Figure). The test set was not publicly available, so the authors use the validation set for evaluation. **Baselines:** Faster R-CNN, Mask R-CNN, and Cascade Mask R-CNN, all reproduced by the authors. **Metrics:** Precision, Recall, and F1 score (classification-focused, not mAP). **Key experiments:** 1. **Full comparison** (Table 2): VTLayout vs. baselines on full PubLayNet. 2. **Per-category F1** (Table 3): Breakdown showing where VTLayout improves. 3. **Small-dataset stability** (Table 4): Training on a reduced subset (25k Text/Title, 10k Figure/List/Table). Five-fold cross-validation to assess stability. 4. **Ablation study** (Table 5): Seven configurations removing individual or pairs of extractors (DVFE, SVFE, TFE) to measure each component's contribution. ## What are the outcomes/conclusions? **Main results:** | Model | Precision | Recall | F1 | | :--- | :--- | :--- | :--- | | Faster R-CNN | 0.9319 | 0.9130 | 0.9224 | | Mask R-CNN | 0.9379 | 0.9410 | 0.9385 | | Cascade Mask R-CNN | 0.9515 | 0.9506 | 0.9510 | | **VTLayout** | **0.9584** | **0.9618** | **0.9599** | **Per-category highlights:** - List F1 improves from 0.9055 (Cascade Mask R-CNN) to 0.9177 - Title F1 improves from 0.9166 (Cascade Mask R-CNN) to 0.9411, though Faster R-CNN still edges it out at 0.9425, suggesting the fusion approach does not uniformly dominate all baselines on every category - Text F1 improves from 0.9688 to 0.9751 - Table F1 improves from 0.9794 to 0.9833 - Figure F1 drops slightly from 0.9846 to 0.9824 **Ablation findings:** - DVFE alone achieves 0.9272 F1, confirming it as the most important component - Removing SVFE from the full model drops F1 from 0.9599 to 0.8956 (large degradation, especially for List: 0.9177 to 0.7635) - Removing TFE drops F1 from 0.9599 to 0.9440; the effect is moderate but consistent across categories - SVFE alone achieves 0.8209, showing pixel histograms carry meaningful discriminative signal **Limitations:** - Evaluated only on PubLayNet, a single-domain (biomedical) dataset with 5 categories. Generalization to other domains or finer-grained taxonomies is untested. - The two-stage pipeline is sequential and likely slower than end-to-end detectors, though no latency measurements are reported. - Relies on PaddleOCR for text extraction, introducing a dependency on OCR quality. The authors use `chinese_ocr_db_crnn_mobile`, a model name suggesting Chinese-language optimization, on English biomedical text. While PaddleOCR supports English, this choice may limit TFE effectiveness. The authors also note that enlarging small Title blocks before OCR helps but blurring from upscaling still limits recognition. - No code, model weights, or trained artifacts released. - The evaluation uses precision/recall/F1 on classification rather than the standard mAP@IoU metric, making direct comparison with other PubLayNet results difficult. Because F1 is measured only on regions that Stage 1 successfully detected, entirely missed regions are not penalized; the reported scores reflect classification accuracy on localized regions, not end-to-end system quality. - The "shallow visual feature" (pixel histogram) is sensitive to document style. Its effectiveness on diverse document types (e.g., handwritten, colored, degraded) is unknown. - The five-fold cross-validation results in Table 4 contain a suspicious discrepancy: per-fold List F1 values range from 0.9648 to 0.9810 (averaging approximately 0.9730), yet the paper's reported column average is 0.9286. This appears to be a typographical error in the original paper. - The pipeline spans two frameworks (PyTorch for Stage 1, TensorFlow for Stage 2), adding integration complexity for anyone attempting to reproduce the work. ## Reproducibility ### Models - **Stage 1:** Cascade Mask R-CNN with ResNeXt-101-64x4d backbone, pre-trained on ImageNet. Implemented in PyTorch. - **Stage 2 DVFE:** MobileNetV2 pre-trained on ImageNet (without top FC layer), implemented in TensorFlow. Input: $128 \times 128 \times 3$. Global average pooling applied to last conv block output. SE block for channel re-calibration. - **Stage 2 classifier:** 4-layer FC network with 512, 256, 128, 64 neurons. Softmax output over 5 classes. - No weights released. ### Algorithms - **Stage 1 training:** SGD optimizer, initial LR 0.02, momentum 0.9, weight decay 0.0001. Batch size 8 (1 sample/GPU). 30 epochs. - **Stage 2 training:** Adam optimizer, initial LR 0.001. Cross-entropy loss. - **SVFE:** Color to grayscale conversion, then 256-bin pixel histogram per region. - **TFE:** PaddleOCR (`chinese_ocr_db_crnn_mobile` model) for text extraction. Title blocks enlarged 8$\times$ before OCR. TF-IDF via scikit-learn. - No data augmentation details reported for either stage. - Several Stage 2 details are unspecified: number of training epochs, FC layer activation functions, SE block reduction ratio, and TF-IDF vector dimensionality (max features/vocabulary size). ### Data - **Training:** PubLayNet training set (335,703 images, 3.3M annotations across 5 categories). PubLayNet annotations are licensed under [CDLA-Permissive-1.0](https://github.com/ibm-aur-nlp/PubLayNet); source images come from PubMed Central Open Access. - **Evaluation:** PubLayNet validation set (11,245 images, 120,761 annotations). Test set not publicly available. - **Small-dataset experiment:** 25k Text/Title images, 10k Figure/List/Table images randomly sampled from Cascade Mask R-CNN stage 1 output. ### Evaluation - **Metrics:** Precision, Recall, F1 score (classification-focused). The authors explicitly chose not to use mAP@IoU to isolate classification performance from localization quality. - **Baselines:** Faster R-CNN and Mask R-CNN reproduced by the authors; Cascade Mask R-CNN is VTLayout's own first stage. - **Ablation:** 7 configurations testing all subsets of {DVFE, SVFE, TFE}. - **Stability:** 5-fold cross-validation on the small subset. - No error bars, significance tests, or multi-seed runs reported. ### Hardware - 8 GPUs mentioned (batch size 8, 1 sample per GPU) for Cascade Mask R-CNN training. - GPU type, total training time, and inference speed not reported. ## BibTeX ```bibtex @inproceedings{li2021vtlayout, title={VTLayout: Fusion of Visual and Text Features for Document Layout Analysis}, author={Li, Shoubin and Ma, Xuyan and Pan, Shuaiqun and Hu, Jun and Shi, Lin and Wang, Qing}, booktitle={PRICAI 2021: Trends in Artificial Intelligence}, pages={304--319}, year={2021}, publisher={Springer}, doi={10.1007/978-3-030-89188-6_23} } ```