# ÉCLAIR: Jointly Extracting Text, Bounding Boxes, and Semantic Classes in Reading Order

> ÉCLAIR is a 937M-parameter multimodal encoder-decoder that simultaneously extracts formatted text in reading order, bounding boxes, and semantic classes from document images, trained on a custom 5M-page arXiv dataset with a modified LaTeX compiler.

- Authors: Hunter Heidenreich
- Published: 2026-04-02
- Paper: Preprint (arXiv) <https://arxiv.org/abs/2502.04223>
- Benchmark: DROBS Benchmark <https://arxiv.org/abs/2502.04223>
- Dataset: arXiv-5M Dataset <https://arxiv.org/abs/2502.04223>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## TL;DR

ÉCLAIR is a 937M-parameter multimodal encoder-decoder that jointly extracts formatted text in reading order, block-level bounding boxes, and semantic class labels from a single document image pass. The authors also introduce DROBS, a 789-page human-annotated benchmark covering visually diverse layouts, and report competitive or leading performance on reading-order OCR and document object detection tasks.

## What kind of paper is this?

**Dominant:** $\Psi\_{\text{Method}}$: The central contribution is a new model architecture and training recipe for end-to-end document text extraction with simultaneous reading-order, bounding box, and semantic class prediction. The paper includes ablations, SOTA-style comparison tables, and an inference-speed optimization (multi-token decoding) that together characterize a methodological paper.

**Secondary:** $\Psi\_{\text{Resource}}$: The authors release the arXiv-5M synthetic training dataset (generated via a modified LaTeX compiler) and the DROBS human-annotated benchmark.

**Secondary:** $\Psi\_{\text{Evaluation}}$: DROBS is presented as a new evaluation protocol to address gaps in existing OCR benchmarks, which lack simultaneous reading-order and semantic class annotations.

## What is the motivation?

Extracting useful text from complex documents requires more than recognizing characters. A downstream system needs to know which text block comes first, what role each block plays (body text, footnote, table, formula), and where each block sits on the page. Traditional OCR systems work at the word or line level and cannot resolve the spatial and semantic relationships that govern reading order. More capable pipelines chain multiple specialized models together, creating brittle systems where errors in one stage propagate through the rest.

Recent end-to-end models address parts of this problem but leave gaps. Nougat produces structured markdown but predicts no spatial information. GOT similarly omits bounding boxes. Kosmos-2.5 supports either formatted text or bounding boxes through two mutually exclusive prompts, so a caller cannot obtain both simultaneously. None of these models predict semantic classes alongside text and boxes.

The authors identify two further limitations in existing benchmarks: GOT's benchmark captures document-level reading order but lacks block-level bounding boxes and semantic labels, while DocLayNet provides bounding boxes and classes but not reading order. DROBS is designed to fill both gaps at once.

## What is the novelty?

ÉCLAIR combines a ViT-H/16 vision encoder (657M parameters, initialized from RADIO) with a lightweight mBART-based autoregressive decoder (279M parameters, trained from scratch). A horizontal convolutional neck reduces the 5120-token image sequence to 1280 tokens before feeding the decoder.

The decoder conditions on encoder features to predict tokens autoregressively:

$$P(t\_i \mid \mathcal{N}(Z),\, t\_{<i}), \quad Z = \mathcal{E}(I)$$

where $\mathcal{E}$ is the vision encoder, $\mathcal{N}$ is the neck, $I \in \mathbb{R}^{3 \times H \times W}$ is the input image, and $\\{t\_1, \dots, t\_P\\}$ are prompt tokens.

The output for each bounding box, in the maximal-information prompt (MIP) setting, follows:

```
<x_(\d+)><y_(\d+)>(text content)<x_(\d+)><y_(\d+)><class_([^>]+)>
```

where the first coordinate pair gives the top-left corner, the second gives the bottom-right corner, and the class token identifies the semantic category.

The prompt interface is a tuple drawn from three independent binary choices:

- text mode: `<structured_text>`, `<plain_text>`, or `<no_text>`
- bounding boxes: `<bbox>` or `<no_bbox>`
- classes: `<classes>` or `<no_classes>`

This yields eight valid combinations. The MIP (`<structured_text><bbox><classes>`) is used as the canonical training target. During fine-tuning, information is selectively dropped per dataset to accommodate partially annotated sources, allowing visually diverse data with incomplete labels to contribute to training.

A second technical contribution is multi-token decoding. Standard autoregressive decoding requires one step per output token, which is expensive for text-dense documents. The authors add $n-1$ linear heads on top of the decoder's final hidden state, each predicting the next offset token:

$$\hat{t}\_{i+k} = \text{Head}\_k(h\_i), \quad k \in \\{1, \dots, n-1\\}$$

With $n = 2$ or $n = 3$, multi-token ÉCLAIR matches or slightly improves text accuracy while reducing decoding steps by a factor of $n$.

The arXiv-5M dataset underpins pre-training. Rather than post-processing compiled PDFs (as Nougat does via LaTeXML), the authors embed Python hooks directly into the TeX Live compiler to intercept node allocations, character placements, and hbox/vbox events as they occur. This allows character-level bounding boxes and semantic class assignments to be recorded at compile time, producing a tightly aligned image/annotation dataset at roughly 5 million pages.

## What experiments were performed?

**DROBS reading-order benchmark.** The primary evaluation uses 789 human-annotated pages sampled from magazines, books, and Common Crawl documents. Annotations follow DocLayNet's labeling schema with reading order added. ÉCLAIR in MIP mode is compared against Kosmos-2.5 and GOT, each run in both OCR and markdown output modes. Evaluation metrics include word error rate (WER), character-level edit distance, token-level F1/precision/recall, BLEU, METEOR, and a Counting F1 score designed to penalize missing repeated words:

$$\text{Counting F1} = \frac{2 \cdot P\_{\text{count}} \cdot R\_{\text{count}}}{P\_{\text{count}} + R\_{\text{count}}}$$

Tables and equations are masked from images prior to inference because DROBS does not yet include ground-truth labels for those element types.

**GOT benchmark.** ÉCLAIR is also evaluated on the English-language split of the GOT benchmark, comparing edit distance, F1, BLEU, and METEOR against Nougat, TextMonkey, DocOwl1.5, Vary, Fox, GOT, and Qwen-VL variants.

**Nougat validation set.** ÉCLAIR pre-trained on arXiv-5M is evaluated on text, math, and table extraction on a 10,000-sample arXiv validation set. The authors are careful to note that direct comparison to Nougat's published numbers is not valid due to differences in output format and evaluation protocol.

**DocLayNet object detection.** ÉCLAIR is fine-tuned on DocLayNet alone for 50K steps and evaluated with COCO-mAP (IoU 0.5:0.95) against Mask R-CNN and SwinDocSegmenter. Because ÉCLAIR does not over-predict boxes, the authors also use sequence augmentation and top-k class sampling from Pix2Seq to produce multiple candidates for AP computation.

**LLM training quality.** Nemotron-8B is trained from scratch on 300B tokens using text extracted by ÉCLAIR versus PyMuPDF4LLM from a common PDF corpus. The resulting models are compared on MMLU and eight additional benchmarks (ARC-Easy, ARC-Challenge, HellaSwag, OpenBookQA, PIQA, RACE, WinoGrande, TriviaQA).

**Inference speed and multi-token decoding.** Average seconds per image and seconds per 100 tokens are measured on DROBS using an H100 GPU in bfloat16, comparing Nougat, GOT, and ÉCLAIR variants with $n \in \\{1, 2, 3, 4\\}$ tokens per step.

**Training and inference ablations.** Table 5 in the paper isolates the contributions of pre-training, fine-tuning, and the repetition penalty on DROBS Counting F1 and WER.

## What are the outcomes/conclusions?

On DROBS, ÉCLAIR-MIP achieves WER of 0.142, F1 of 0.942, BLEU of 0.886, and METEOR of 0.930, outperforming Kosmos-2.5 (OCR mode: WER 0.195) and GOT (OCR mode: WER 0.302) on most metrics. Kosmos-2.5 in OCR mode achieves higher recall (0.950 vs. 0.942), and the authors attribute score differences between OCR and markdown modes to differences in training data blending rather than any architectural effect.

On the GOT English benchmark, ÉCLAIR achieves the best edit distance (0.032) among models with under 1B parameters, outperforming Fox (1.8B) and Qwen-VL-Max (72B+) on edit distance and several other metrics, though GOT (580M) achieves marginally higher F1 (0.972 vs. 0.968).

On DocLayNet mAP, ÉCLAIR reaches 73.9 overall, between Mask R-CNN (73.5) and SwinDocSegmenter (75.2). The authors note that mAP is structurally unfavorable to autoregressive detectors because there is no confidence score to sweep for a precision-recall curve. In the supplementary point-to-point precision/recall comparison (Table S1), ÉCLAIR achieves higher mean precision and mean recall than SwinDocSegmenter at matched operating points for most classes.

LLM quality improves when training on ÉCLAIR-extracted text: Nemotron-8B scores 39.1 MMLU (versus 37.2 with PyMuPDF4LLM) and 56.7 on the multi-benchmark average (versus 55.72). ÉCLAIR also extracts more tokens from the same PDFs (55.1B versus 43.6B), likely because it handles formatting and multi-column layouts that PyMuPDF4LLM misses.

Multi-token decoding at $n = 2$ reduces inference time from 3.8 to 2.5 seconds per image with a slight accuracy improvement (WER 0.13, F1 0.94). At $n = 3$, speed is 1.77 s/img with a small accuracy cost. The $n = 4$ variant degrades noticeably. Using a multi-token-trained model at next-token inference ($\frac{tkn}{step} = 1$) provides accuracy gains at no speed cost.

Limitations acknowledged by the authors include: absence of multilingual training (no Chinese or other non-English data beyond what appears incidentally), challenges with repetition-loop hallucinations mitigated by post-processing rather than eliminated, and the fact that DROBS does not yet include ground-truth table and equation labels.

## Reproducibility

### Models

- Total parameters: 937M (reported as 936M or 963M at different points in the paper; the supplementary states 937M as the canonical figure).
- Vision encoder: RADIO ViT-H/16, 657M parameters, input resolution 1280x1024, producing 80x64 patches. Initialized from a pretrained RADIO checkpoint.
- Vision neck: horizontal 1x4 convolutional compression (stride 1x4), reducing 5120 patches to 1280 tokens.
- Decoder: mBART architecture, 10 layers, 279M parameters, trained from scratch. Maximum sequence length: 3584 tokens.
- Tokenizer: Galactica tokenizer extended with $H + W + C + 7$ special tokens for coordinates, semantic classes, and prompt components.
- Pretrained model weights are not publicly released as of the arXiv submission.

### Algorithms

- Two-stage training: pre-training on arXiv-5M, then fine-tuning on the full multi-source dataset.
- Both stages: AdamW optimizer, 130,000 iterations, effective batch size 128.
- Pre-training learning rate: $2 \times 10^{-5}$ constant with 5,000 linear warmup steps.
- Fine-tuning learning rate: $8 \times 10^{-6}$ constant with 500 linear warmup steps.
- Inference: greedy decoding with repetition penalty of 1.1.
- DocLayNet fine-tuning: 50,000 additional steps for the detection evaluation.
- Multi-token variants: $n - 1$ additional linear heads (1024 nodes each) added on top of the decoder hidden state; trained with teacher forcing on next-$n$ ground-truth tokens.
- Post-processing: MIP inference followed by strict regex syntax validation, bounding box spatial validity checks, and class schema validation to filter hallucinated outputs.

### Data

- arXiv-5M: approximately 5 million pages of arXiv paper images with character-level bounding boxes, semantic classes, and formatted text. Generated via a modified TeX Live compiler with Python hooks. License not specified in the paper.
- SynthTabNet: 480K synthetic table images with HTML-to-LaTeX converted annotations.
- README: 302K pages rendered from Stack source-code README files via Pandoc and wkhtmltopdf, with Nougat's data pipeline for alignment.
- DocLayNet: 56K pages, plain text with boxes and classes.
- G1000: 324K pages with plain text obtained via Tesseract OCR.
- Human-annotated Common Crawl: 14K pages with plain text, boxes, and classes.
- Total training data: approximately 6.176M pages.
- DROBS benchmark: 789 pages drawn from magazines, books, and Common Crawl, annotated by human labelers following DocLayNet guidelines with reading order added. The authors state they will release it to the research community; a formal URL or repository was not provided in the arXiv version.

### Evaluation

- DROBS metrics: WER, character edit distance, token-level F1/precision/recall, BLEU, METEOR, Counting F1. String normalization removes non-alphanumeric characters and collapses whitespace before scoring. Tables and equations are masked from images for all methods because DROBS lacks ground-truth for those elements.
- GOT benchmark: same text metrics, evaluated on English split only; ÉCLAIR does not train on Chinese data.
- arXiv-5M validation set: 10,000 samples evaluated for text, math, and table extraction separately.
- DocLayNet: COCO-mAP (IoU 0.5:0.95, maxDets=100). Sequence augmentation and top-k class sampling are used specifically to produce enough candidate boxes for the AP curve. The authors note this metric is structurally unfavorable to autoregressive models and provide a point-to-point precision/recall comparison as an alternative in the supplementary.
- LLM benchmark: Nemotron-8B trained on 300B tokens total; 3.3 epochs of PDF-extracted tokens from a shared PDF corpus; remaining tokens from CommonCrawl, StackExchange, OpenWebMath, PubMed, bioRxiv, SEC filings, Wikipedia, and arXiv.
- The paper does not report error bars, statistical significance tests, or multiple runs.

### Hardware

- Inference speed measured on a single NVIDIA H100 GPU in bfloat16.
- Single-token ÉCLAIR: 3.8 s/img, 0.42 s/100 tokens on DROBS.
- ÉCLAIR-2tkn (2 tokens per step): 2.5 s/img, 0.31 s/100 tokens.
- ÉCLAIR-3tkn: 1.77 s/img, 0.23 s/100 tokens.
- ÉCLAIR-4tkn: 1.32 s/img, 0.20 s/100 tokens.
- Training hardware (GPU count, total GPU-hours, memory requirements) is not reported in the paper or supplementary.

## BibTeX

```bibtex
@article{karmanov2025eclair,
  title={{\'E}CLAIR: Extracting Content and Layout with Integrated Reading Order for Documents},
  author={Karmanov, Ilia and Deshmukh, Amala Sanjay and V{\"o}gtle, Lukas and Fischer, Philipp and Chumachenko, Kateryna and Roman, Timo and Sepp{\"a}nen, Jarno and Parmar, Jupinder and Jennings, Joseph and Tao, Andrew and Sapra, Karan},
  journal={arXiv preprint arXiv:2502.04223},
  year={2025}
}
```