# Document AI: Comparing Transformers, GNNs, and CNNs for Layout Analysis

> A comparative study evaluating LayoutLMv3, Paragraph2Graph, and YOLOv5 across image-centric and text-centric document layout analysis tasks, plus cross-lingual experiments with LiLT and machine translation.

- Authors: Hunter Heidenreich
- Published: 2026-03-14
- Paper: Preprint (arXiv) <https://arxiv.org/abs/2308.15517>
- Code: Experiment Notebooks <https://github.com/samakos/Document-AI->
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## TL;DR

A comparative benchmarking study that evaluates LayoutLMv3 (transformer), Paragraph2Graph (GNN), and YOLOv5 (CNN) on document layout analysis across DocLayNet and GROTOAP2 datasets, framing the task as both image-centric (object detection) and text-centric (token classification). The study also tests whether machine translation can improve LiLT's cross-lingual zero-shot transfer on FUNSD/XFUND forms; the answer is no.

## What kind of paper is this?

**Dominant:** $\Psi\_{\text{Evaluation}}$. The paper proposes no new model or method. Its contribution is a head-to-head comparison of three existing architectures (LayoutLMv3, Paragraph2Graph, YOLOv5) across two benchmarks using standard metrics (mAP, F1). The cross-lingual machine translation experiment is also evaluative in nature.

**Secondary:** $\Psi\_{\text{Systematic}}$. The paper organizes the DLA landscape along two axes: architecture type (transformer vs. graph vs. CNN) and task framing (image-centric vs. text-centric), providing a structured overview of related work.

## What is the motivation?

The authors identify three gaps in document layout analysis research:

1. **No cross-architecture comparison.** Prior work developed transformer-based, graph-based, and CNN-based models independently, but no study directly compared their effectiveness on the same benchmarks.
2. **Limited dataset coverage.** Existing comparisons focused on narrow datasets (often PubLayNet or DocBank). The authors target DocLayNet (diverse, multi-domain) and GROTOAP2 (scientific articles, 22 classes) for broader coverage.
3. **Unexplored machine translation for cross-lingual DLA.** Language-independent models like LiLT showed promise for zero-shot cross-lingual transfer, but no prior work tested whether translating documents to the training language (English) could improve performance.

## What is the novelty?

The novelty is primarily in the experimental design rather than in any algorithmic contribution:

- First direct comparison of LayoutLMv3, Paragraph2Graph, and YOLOv5 on DocLayNet and GROTOAP2 under both image-centric and text-centric framings.
- First use of machine translation (M2M100) as a preprocessing step for cross-lingual document layout analysis with LiLT.

The evaluation metrics are standard:

$$
\text{mAP} = \frac{1}{N} \sum\_{i=1}^{N} AP\_i
$$

for image-centric tasks (COCO-style, IoU thresholds $[0.5, 0.95]$ with step 0.05), and F1-score for text-centric token classification.

## What experiments were performed?

### Experiment 1: Image-Centric DLA (Object Detection)

**GROTOAP2** (first pages only, 22 classes):

| Model | mAP | Training Time | Hardware |
| :--- | ---: | :--- | :--- |
| LayoutLMv3 | **0.751** | ~3 days (7.5 epochs) | 1 GPU |
| YOLOv5s | 0.725 | ~2.5 hours (75 epochs) | 1 GPU |

YOLOv5 outperformed LayoutLMv3 on 14 of 22 individual classes. LayoutLMv3 was stronger on glossary, tables, figures, unknown elements, and page numbers.

**DocLayNet** (11 classes): The authors compared LayoutLMv3 against previously reported numbers for YOLOv5 and Paragraph2Graph from prior work. Exact per-class numbers are reported in the paper. The discussion concludes YOLOv5 is most practical for industry use due to speed.

### Experiment 2: Text-Centric DLA (Token Classification)

**GROTOAP2** (first pages):

| Model | F1 | Training Time |
| :--- | ---: | :--- |
| LayoutLMv3 | **0.866** | ~2 days (6 epochs) |
| Paragraph2Graph | 0.698 | ~7 hours (9 epochs) |

LayoutLMv3 outperformed Paragraph2Graph by a large margin, though at substantially higher computational cost.

### Experiment 3: Cross-Lingual Transfer with Machine Translation

LiLT (with XLM-RoBERTa tokenizer) was fine-tuned on English FUNSD forms (4 classes), then evaluated zero-shot on XFUND forms in 7 languages. The M2M100 translation model was used to translate non-English forms to English before inference.

| Method | EN | DE | IT | ZH | JA | ES | FR | PT |
| :--- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| LiLT (direct) | 0.48 | **0.49** | **0.38** | **0.35** | **0.37** | **0.39** | **0.50** | **0.44** |
| LiLT + MT | - | 0.43 | 0.33 | 0.34 | 0.30 | 0.33 | 0.43 | 0.35 |

Machine translation consistently degraded performance across all languages. The authors attribute this to token-level translation errors disrupting layout alignment.

## What are the outcomes/conclusions?

1. **YOLOv5 is the most practical choice for image-centric DLA.** While LayoutLMv3 achieves marginally higher mAP on GROTOAP2, YOLOv5 trains in hours (vs. days) on a single GPU and runs inference on 1,300 pages in under a minute (vs. ~7 minutes for LayoutLMv3).
2. **LayoutLMv3 substantially outperforms Paragraph2Graph in text-centric DLA.** With an F1 of 0.866 vs. 0.698 for Paragraph2Graph, the gap is substantial, though the authors note LayoutLMv3's CC-BY-NC-SA license makes it unsuitable for commercial use. They suggest LiLT as a viable alternative.
3. **Machine translation does not help cross-lingual DLA.** Translating documents to English before applying LiLT performed worse than direct zero-shot transfer in every language tested. Token-level translation introduces alignment errors that outweigh any linguistic benefit.

**Limitations the authors acknowledge:**
- Paragraph2Graph was not evaluated on GROTOAP2 for image-centric tasks (left as future work).
- No hyperparameter tuning for YOLOv5; only YOLOv5s (smallest variant) was tested.
- The GROTOAP2 experiments used only first pages of documents, which biases toward front-matter elements.
- Machine translation was only tested at token level; sequence-level classification might yield different results.

**Limitations not acknowledged:**
- All comparisons use models "as-is" from prior work with minimal tuning, making it unclear whether performance gaps reflect fundamental architectural differences or just different optimization effort.
- The FUNSD/XFUND experiments report F1 scores below 0.50 even in English, suggesting the 4-class setup or fine-tuning protocol may be suboptimal, limiting the interpretability of the cross-lingual comparison.
- DocLayNet results partially rely on numbers from prior work rather than consistent re-evaluation.

## Reproducibility

### Models

- **LayoutLMv3**: Pre-trained on IIT-CDIP (11M images). Uses RoBERTa for text, DiT for vision. 133M params (base). CC-BY-NC-SA-4.0.
- **Paragraph2Graph**: Language-independent GNN. Exact parameter count not reported.
- **YOLOv5s**: Smallest YOLOv5 variant. Pre-trained on COCO. Open-source (GPL-3.0 via Ultralytics).
- **LiLT**: Language-independent layout transformer with XLM-RoBERTa tokenizer. MIT license.
- **M2M100**: 1.2B-parameter many-to-many translation model from Meta.

### Algorithms

- **Text-centric LayoutLMv3**: LR 1e-5 (from original paper), trained until convergence (~6 epochs on GROTOAP2).
- **Text-centric Paragraph2Graph**: LR 0.001, weight decay 0.005 (from original paper), trained until convergence (~9 epochs).
- **Image-centric LayoutLMv3**: LR 2e-4, batch size 2, 100 epochs max with early stopping.
- **Image-centric YOLOv5s**: Default configuration, no hyperparameter tuning, trained until convergence (~75 epochs).
- Convergence determined by monitoring training/validation loss and mAP on validation data.
- The released code consists of Jupyter notebooks for individual experiments, not a unified training framework.

### Data

- **DocLayNet**: 80,863 pages (69,103 train / 6,480 val / 4,994 test), 11 classes. Diverse domains (financial, tenders, laws, manuals, patents, scientific). 95% English.
- **GROTOAP2**: Originally 13,210 documents / 119,334 pages. Authors used only first pages: 10,492 train / 1,310 val / 1,317 test. 22 classes. Scientific articles only. Converted from XML to COCO format for image-centric tasks and preprocessed for token classification.
- **FUNSD**: 149 train + 50 test English forms, 4 classes.
- **XFUND**: Multilingual forms in 7 languages, 4 classes.

### Evaluation

- **Image-centric**: mAP @ IoU [0.5:0.05:0.95] (COCO-style).
- **Text-centric**: F1-score.
- No error bars, confidence intervals, or multi-run statistics reported.
- GROTOAP2 results are reported on the validation set, not the held-out test set.
- DocLayNet image-centric results for YOLOv5 and Paragraph2Graph taken from prior publications, not re-run.

### Hardware

- All experiments on a single GPU (type not specified).
- YOLOv5s: ~2.5 hours for 75 epochs on GROTOAP2.
- LayoutLMv3: ~3 days for 7.5 epochs (image-centric) or ~2 days for 6 epochs (text-centric) on GROTOAP2.
- YOLOv5 inference: <1 min for 1,300 images. LayoutLMv3 inference: ~7 min for 1,300 images.

## BibTeX

```bibtex
@article{kastanas2023document,
  title={Document AI: A Comparative Study of Transformer-Based, Graph-Based Models, and Convolutional Neural Networks For Document Layout Analysis},
  author={Kastanas, Sotirios and Tan, Shaomu and He, Yi},
  journal={arXiv preprint arXiv:2308.15517},
  year={2023}
}
```