# FocalOrder: Focal Preference Optimization for Reading Order Detection

> FocalOrder introduces Focal Preference Optimization (FPO), a training framework that replaces uniform cross-entropy supervision with EMA-based difficulty discovery and difficulty-calibrated pairwise ranking to address the Positional Disparity failure mode in reading order detection.

- Authors: Hunter Heidenreich
- Published: 2026-04-02
- Paper: Preprint (arXiv) <https://arxiv.org/abs/2601.07483>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## TL;DR

FocalOrder identifies a systematic failure mode called Positional Disparity, in which all existing reading order models achieve near-perfect accuracy at document start and end positions but collapse in the middle 20%--80% of the document. The authors attribute this to uniform cross-entropy supervision drowning out learning signals from structurally ambiguous layout transitions. Their proposed framework, Focal Preference Optimization (FPO), addresses this by pairing an EMA-based difficulty tracker with a difficulty-calibrated pairwise ranking objective. The authors report the lowest edit distance on OmniDocBench v1.0 and the highest REDS on Comp-HRDoc in their comparison, at 0.4B parameters.

## What kind of paper is this?

**Dominant:** $\Psi\_{\text{Method}}$: The headline contribution is FPO, a new training objective and curriculum strategy for reading order models. The paper includes a progressive ablation study, comparison tables against pipeline tools, general VLMs, and expert document models, and an inference latency comparison showing that FPO adds no overhead at test time.

**Secondary:** $\Psi\_{\text{Evaluation}}$: The Positional Disparity analysis (Section 3) is a careful empirical measurement across multiple architectures and two benchmarks. The Spatial-Logical Mismatch metric introduced there quantifies the structural cause of the observed disparity and stands as a diagnostic contribution independent of the proposed fix.

## What is the motivation?

Reading order detection is a pre-requisite for reliable downstream document understanding. A misordered sequence fed into a RAG pipeline or a reasoning system propagates structural errors throughout the downstream task.

The dominant training recipe across all recent methods, from LayoutReader through MinerU 2.5, is standard cross-entropy over the predicted order sequence. This recipe treats every transition in a document as equally important. The authors conduct an empirical analysis across LayoutReader, PaddleOCR-VL, and MinerU 2.5 on OmniDocBench and Comp-HRDoc, measuring prediction error rate binned by relative document position $p = t/T$. All models show a consistent "Inverted-U" error curve: error is low in the first and last 10--20% of the document, peaks between positions 20% and 80%, and drops back at the end.

To diagnose the cause, the authors introduce the Spatial-Logical Mismatch rate: the proportion of ground-truth transitions where the next correct region is not the geometrically nearest neighbor (measured by Euclidean distance between bounding box centers). This mismatch distribution peaks in the same 20%--80% interval, suggesting that the intermediate sections contain the most complex layout decisions, precisely those where simple spatial heuristics fail and where uniform supervision provides the weakest signal.

Formally, all existing methods optimize a standard cross-entropy objective that assigns equal weight to every predicted token:

$$\mathcal{L}\_{\text{CE}} = -\frac{1}{N} \sum\_{t=1}^{N} \log P(y\_t \mid Y\_{<t}, O, I) \quad (1)$$

The Positional Disparity analysis is correlated evidence rather than a controlled causal proof: the paper evaluates only three models (LayoutReader, PaddleOCR-VL, MinerU 2.5), all trained with standard CE on similar data distributions. Whether the Inverted-U pattern arises from the loss function specifically or from the training data distribution is not fully disentangled.

## What is the novelty?

FocalOrder adds two training-time components on top of a standard generative reading order backbone (LayoutLMv3-large):

**Adaptive Difficulty Discovery** tracks per-bin loss using exponential moving average (EMA). The document is partitioned into $K$ bins by relative position. A global difficulty vector $\mathbf{D} \in \mathbb{R}^K$ is updated after each iteration:

$$\tilde{\mathcal{L}}\_k^{(\text{iter})} = \gamma \cdot \tilde{\mathcal{L}}\_k^{(\text{iter}-1)} + (1-\gamma) \cdot \mathcal{L}\_{\text{batch}}^{(k)} \quad (2)$$

The momentum $\gamma = 0.99$ acts as a low-pass filter. A per-token difficulty weight $w\_t$ is then derived by normalizing bin loss against the mean of $\mathbf{D}$ and clipping to $[1 - \delta, 1 + \delta]$ with $\delta = 0.8$:

$$w\_t = \text{Clip}\!\left(\frac{\tilde{\mathcal{L}}\_k}{\mu\_{\mathbf{D}}},\; 1-\delta,\; 1+\delta\right) \quad (3)$$

These weights modulate the standard cross-entropy loss at the token level, amplifying gradients from the ambiguous document body while attenuating them at the deterministic start and end.

**Difficulty-Calibrated Pairwise Ranking (DCPR)** adds a sequence-level contrastive objective. For each generated candidate $\hat{Y}\_i$, a reward is computed as the inverted normalized Levenshtein distance against the ground truth:

$$R(\hat{Y}, Y^{\ast}) = 1 - \frac{\text{Lev}(\hat{Y}, Y^{\ast})}{\max(|\hat{Y}|, |Y^{\ast}|)} \quad (4)$$

A difficulty bonus is added to form the advantage score:

$$A\_i = R(\hat{Y}\_i, Y\_i^{\ast}) + \beta \cdot \tilde{\mathcal{L}}\_{\text{CE}}^{(i)} \quad (5)$$

where $\tilde{\mathcal{L}}\_{\text{CE}}^{(i)}$ is the length-normalized per-sample cross-entropy (further normalized by a running mean for scale consistency). Setting $\beta = 0.05$ ensures the reward dominates while hard-sample bonuses act as tie-breakers. Training pairs $(i, j)$ are drawn from the top and bottom $\rho = 20\%$ of samples ranked by $A\_i$. The ranking loss is a hinge over sequence log-probability scores:

$$\mathcal{L}\_{\text{Rank}} = \frac{1}{|\mathcal{P}|} \sum\_{(i,j) \in \mathcal{P}} \left[S(\hat{Y}\_j) - S(\hat{Y}\_i) + m\_{ij}\right]\_+ \quad (6)$$

The adaptive margin scales with the structural complexity of the harder sequence in the pair, forcing larger probability gaps for complex layout transitions:

$$m\_{ij} = \alpha \cdot \max(w^{(i)}, w^{(j)}) \quad (7)$$

where $\alpha$ is a base scaling factor and $w^{(i)}$ is the mean token-level difficulty weight for sequence $i$.

The total objective combines both components:

$$\mathcal{L}\_{\text{total}} = \sum\_{t=1}^{N} w\_t \cdot \mathcal{L}\_{\text{CE}}^{(t)} + \lambda\_{\text{Rank}} \cdot \mathcal{L}\_{\text{Rank}} \quad (8)$$

## What experiments were performed?

**Benchmarks.** All main experiments use OmniDocBench v1.0 (981 pages) and v1.5 (1,355 pages), reporting normalized edit distance (lower is better), and Comp-HRDoc (1,500 documents), reporting Reading Edit Distance Score, REDS (higher is better), separately for text and graphical regions.

**Backbone.** LayoutLMv3-large is used as the unified backbone encoder across all ablation conditions and the final model, enabling controlled component comparisons. The model is trained for 50 epochs with batch size 24.

**Comparison baselines.** On OmniDocBench v1.0, FocalOrder is compared against a wide range of systems across three categories: pipeline tools (MinerU, Marker, Mathpix, Docling, Pix2Text, Unstructured, OpenParse, PP-StructureV3), general VLMs (GPT-4o, Qwen2-VL-72B, Qwen2.5-VL-72B, Gemini-1.5 Pro, Doubao-1.5-pro, InternVL2-76B, InternVL3-78B, GOT-OCR, Nougat, Mistral OCR, OLMOCR-sglang, SmolDocling-256M), and expert VLMs (Dolphin, MinerU 2.0, OCRFlux, MonkeyOCR-pro-3B, dots.ocr, PaddleOCR-VL, MinerU 2.5).

**Ablation study.** A six-row progressive ablation on OmniDocBench v1.0 isolates contributions from: (1) base LayoutReader model; (2) fine-tuning; (3) category token embeddings; (4) standard preference optimization; (5) EMA fine-grained loss alone; and (6) the full FocalOrder combining group contrastive ranking with EMA. All ablation variants use the same 0.4B backbone; inference latency increases from 12.1 ms to 12.3 ms between the base model and the full FocalOrder framework (the EMA-only intermediate variant reaches 12.4 ms).

**Sensitivity analysis.** The number of bins $K$ is swept over $\\{1, 5, 10, 20, 50\\}$. The advantage weight $\beta$ is swept over $\\{0.0, 0.01, 0.05, 0.1, 0.2\\}$. The pair selection ratio $\rho$ is swept over $\\{10\%, 20\%, 30\%\\}$.

**Comparison with alternative weighting strategies.** FocalOrder is benchmarked against uniform supervision, a static Inverted-U Gaussian weighting baseline, and a token-level EMA weighting variant (without spatial binning), all at 0.4B parameters on OmniDocBench v1.0.

## What are the outcomes/conclusions?

The authors report that FocalOrder achieves an edit distance of 0.038 (EN) and 0.055 (ZH) on OmniDocBench v1.0, the lowest in their comparison across all categories, including much larger general VLMs such as Gemini-1.5 Pro (0.049 EN) and models with dedicated document pipelines such as PaddleOCR-VL (0.045 EN) and MinerU 2.5 (0.045 EN). On OmniDocBench v1.5, the model reaches 0.044 edit distance, tying MinerU 2.5 (1.2B) and outperforming general VLMs up to 241B parameters, though PaddleOCR-VL (0.9B) achieves 0.043 in their comparison.

On Comp-HRDoc, FocalOrder achieves 97.1% REDS on text regions and 91.1% on graphical regions, improving 0.1% over UniHDSA-R50 (91.0%) on graphical regions.

The Positional Disparity analysis confirms the mechanism: in the intermediate document body (20%--80%), FocalOrder reduces the average error rate from 25.99% to 10.28%, a 60.4% relative reduction. The learned difficulty weights form an Inverted-U distribution with dual peaks at 1.61 in the 40%--50% and 60%--70% bins, dropping to 0.32 at the document start, directly mirroring the ground-truth mismatch distribution.

The sensitivity analysis shows robustness: performance peaks at $K = 10$ bins but remains strong across $\\{5, 10, 20\\}$; $\beta = 0.05$ is optimal but the method is stable in $[0.01, 0.1]$; $\rho = 20\%$ pair selection is best. Static Inverted-U weighting (0.042) and token-level EMA (0.043) both underperform bin-level EMA within FocalOrder (0.038), confirming that the data-driven spatial binning is necessary.

Acknowledged limitations:
- FocalOrder is a downstream serialization module; errors in the upstream layout detection stage (missed or mislabeled regions) propagate without correction.
- Category embeddings are aligned to the training benchmark ontology (English and Chinese). Zero-shot transfer to documents with different semantic schemas or scripts may require re-alignment.
- The definition of "correct" reading order is subjective for artistic or highly unstructured layouts; the difficulty-aware formulation may not cover all ambiguous cases.
- Training incurs marginal overhead from pairwise ranking, though inference latency is unaffected.

## Reproducibility

### Models

- Backbone: LayoutLMv3-large, approximately 0.4B parameters. A standard pre-trained checkpoint is used; FocalOrder adds no new parameters to the architecture.
- Input representation: bounding box coordinates and semantic category labels encoded as a unified token sequence. Category labels are sourced from an upstream layout analysis model, not ground truth, during inference.
- No model weights are released with the paper.

### Algorithms

- Optimizer: not explicitly named; initial learning rate $3 \times 10^{-5}$, linear warmup for the first 5% of steps, followed by cosine decay.
- Epochs: 50; batch size: 24.
- EMA momentum: $\gamma = 0.99$.
- Number of difficulty bins: $K = 10$.
- Clipping parameter: $\delta = 0.8$, yielding weight range $[0.2, 1.8]$.
- Advantage difficulty weight: $\beta = 0.05$.
- Pair selection ratio: $\rho = 20\%$ (top and bottom 20% by advantage).
- Margin scaling factor: $\alpha$ (value not stated in the main paper).
- Ranking loss weight: $\lambda\_{\text{Rank}}$ (value not stated in the main paper).
- Training pseudocode is provided in Algorithm 1 (Appendix B.1).

### Data

- Training data: a subset is included in submission supplementary materials; the full training data is not publicly released. The authors cite upload size limitations as the reason (Appendix B.3).
- No code repository is released with this paper.
- Evaluation: OmniDocBench v1.0 (981 pages, Apache-2.0), OmniDocBench v1.5 (1,355 pages), and Comp-HRDoc (1,500 documents, 1,000 train / 500 test, MIT license).
- Baselines are either reproduced from official codebases or cited from their respective papers.
- The paper was submitted to ACL 2026 (per Appendix E AI Usage Declaration). As of the arXiv submission date, it has not yet appeared in a published venue.

### Evaluation

- OmniDocBench metric: normalized edit distance (Levenshtein distance divided by max sequence length); lower is better. Evaluated separately for English (EN) and Chinese (ZH) documents.
- Comp-HRDoc metric: REDS (Reading Edit Distance Score); higher is better. Evaluated separately for text regions and graphical regions.
- Positional Disparity metric: position-wise error rate derived from Levenshtein backtrace alignment, aggregated into $K = 10$ bins by relative sequence position. Defined formally in Appendix A.5.
- The paper does not report standard deviations, confidence intervals, or multiple runs.

### Hardware

- Training hardware: NVIDIA RTX 4090 GPUs. Number of GPUs not specified.
- Inference latency: 12.1--12.4 ms per document on the same hardware class (not GPU-hours or throughput reported).
- No cloud cost or energy consumption estimates are provided.

## BibTeX

```bibtex
@article{liu2026focalorder,
  title={FocalOrder: Focal Preference Optimization for Reading Order Detection},
  author={Liu, Fuyuan and Yu, Dianyu and Ren, He and Liu, Nayu and Kang, Xiaomian and Qiu, Delai and Zhang, Fa and Zhen, Genpeng and Liu, Shengping and Liang, Jiaen and Huang, Wei and Wang, Yining and Zhu, Junnan},
  journal={arXiv preprint arXiv:2601.07483},
  year={2026}
}
```