# ROPE: Reading Order Equivariant Positional Encoding for Graph-based Document Information Extraction > ROPE injects relative reading order context into graph convolutional networks for document information extraction by generating equivariant positional codes for each word's neighborhood, improving entity extraction F1 by up to 8.4 points on form-like documents. - Authors: Bevaya AI Team - Published: 2026-04-02 - Paper: Preprint (arXiv) - Paper: Published (ACL-IJCNLP 2021) - Source: Bevaya Labs, --- ## TL;DR ROPE (Reading Order Equivariant Positional Encoding) addresses a specific limitation in graph-based document information extraction: GCN aggregation functions discard word ordering during message passing, just as average pooling discards position in CNNs. ROPE fixes this by assigning relative reading order codes to each word's neighbors before message passing, adding ordering context without requiring a serialized document. On FUNSD and a large proprietary payment dataset, ROPE improves node and edge classification F1 by up to 8.4 points, and proves at least as important as geometric edge features in the GCN baseline. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Method}}$: The primary contribution is the ROPE encoding mechanism itself: a lightweight modification to existing GCN pipelines that injects relative reading order into graph message passing. The paper centers on ablation tables, architecture description, and performance comparisons across encoding variants. ## What is the motivation? Key information extraction from form-like documents is difficult because layout varies enormously: multi-column layouts, tables, scattered text blocks, and non-aligned sections mean there is no reliable linear serialization of words. Sequence models applied to OCR output suffer when the left-to-right, top-to-bottom order produced by an OCR engine does not reflect actual reading order. GCNs avoid serialization by operating directly on word-level graphs where nodes are tokens and edges encode spatial relationships. However, the aggregation step in GCNs (typically a pooling or attention function applied over all incoming neighbor messages) is order-invariant. Any sequential context latent in the OCR reading order is lost at this step. The gap between sequence models (which capture order but struggle with complex 2D layout) and graph models (which handle 2D layout but drop order) motivates ROPE. ## What is the novelty? The core idea is as follows: before each GCN message passing step, iterate through the neighbors of a target word in OCR reading order and assign consecutive integer codes $p \in \mathbb{N}$ starting from zero. These ROPE codes are concatenated to the corresponding neighbor messages before aggregation. Formally, for a target token $t\_i$ with neighbor set $\mathcal{N}(i)$ in the graph sorted by OCR reading order, the $k$-th neighbor receives ROPE code $p = k$. The augmented message from neighbor $v\_j$ at position $k$ becomes: $$\tilde{m}\_{ij} = [m\_{ij} \,\|\, \phi(k)]$$ where $m\_{ij}$ is the raw neighbor message, $\phi(k)$ is the positional encoding function, and $[\cdot \| \cdot]$ denotes concatenation. The self-attention aggregation then operates over these augmented messages, becoming sensitive to the relative reading order of each neighbor. A key property of this formulation is equivariance: if the entire local neighborhood (target word plus all its neighbors) shifts by the same offset in the document while preserving relative order, the assigned codes remain unchanged. This makes ROPE robust to global position variation while remaining sensitive to local sequential structure. The authors explore two choices for $\phi(k)$: 1. **Index encoding**: $\phi(k) = k$, used directly as a scalar or embedded. 2. **Sinusoidal encoding**: following Vaswani et al. (2017) with 3 base frequencies $\omega \in \\{1, 10, 100\\}$: $$\phi\_{\sin}(k) = \left[\sin\!\left(\frac{k}{\omega}\right),\, \cos\!\left(\frac{k}{\omega}\right)\right]\_{\omega \in \\{1,\,10,\,100\\}}$$ Ablations show that sinusoidal encoding marginally outperforms index encoding on word labeling, while their combination is strongest on word grouping. Since each target word in a $\beta$-skeleton graph typically has fewer than 8 neighbors, index encoding alone already provides most of the benefit. ROPE is plugged into a GCN with multi-head self-attention aggregation (inspired by Graph Attention Networks), which allows the model to weight incoming messages by both content and their ROPE-encoded sequential position. ## What experiments were performed? ### Tasks Two entity extraction tasks are evaluated, both from Jaume et al. (2019): - **Word labeling**: multi-class node classification assigning each token a semantic entity type. Metric: macro F1. - **Word grouping**: binary edge classification deciding whether two tokens belong to the same entity. Metric: F1 with precision and recall. These tasks deliberately avoid CRF-style decoders and do not rely on pre-annotated entity groupings, which isolates the quality of node and edge embeddings. ### Datasets - **Payment**: a proprietary large-scale collection of roughly 18K single-page payment documents from multiple vendors. Human-annotated with 13 semantic entity types, producing over 3M word-level annotations. 80/20 train/test split. OCR performed with Google Cloud Vision. - **FUNSD**: a public benchmark of 199 noisy scanned forms with 4 entity types (header, question, answer, other), 9,707 entities, and 31,485 word-level annotations. Official 75/25 train/test split. ### Architecture and training details - Node features: BERT word embeddings + normalized bounding box coordinates (height, width, four corners). - Edge features: spatial embeddings (relative distances, aspect ratios) and MobileNetV3 visual embeddings from the union bounding box of connected token pairs. - Graph construction: $\beta$-skeleton with $\beta = 1$ (ball-of-sight strategy), providing high connectivity while remaining sparser than fully-connected graphs. - Aggregation: 3-layer multi-head self-attention pooling (4 heads, 32-dimensional head size). - Node update: 2-layer MLP with 128 hidden units. - GCN depth: 7 hops for payment (due to scale), 2 hops for FUNSD. - Optimizer: Adam, learning rate $10^{-4}$, warm-up proportion 0.01, batch size 1. - Hardware: 8 Tesla P100 GPUs; approximately 1 day of training on the largest corpus. ### Ablations **Table 1** crosses EdgeGeo (geometric edge features) and ROPE independently and in combination across both datasets: | Dataset | EdgeGeo | ROPE | WL F1 | WG F1 | |:--------|:-------:|:----:|------:|------:| | Payment | | | 74.55 | 86.64 | | Payment | ✓ | | 60.80 | 83.80 | | Payment | | ✓ | 66.09 | 84.94 | | Payment | ✓ | ✓ | 68.17 | 85.88 | | FUNSD | | | 57.22 | 89.33 | | FUNSD | ✓ | | 50.86 | 86.86 | | FUNSD | | ✓ | 53.16 | 87.37 | | FUNSD | ✓ | ✓ | 51.78 | 89.28 | (WL = Word Labeling; WG = Word Grouping. Full precision/recall numbers are in the paper.) The paper frames these results relative to a fully-featured GCN reference that includes both EdgeGeo and ROPE. Without any explicit positional encoding (the bottom row per dataset), word labeling F1 drops 13.75 points and word grouping drops 2.84 points on payment. Adding ROPE alone recovers 6.38 of those word labeling points; adding EdgeGeo alone recovers less. The authors conclude that reading order information is at least as important as geometric edge features, and that the two provide orthogonal improvements when combined. Note that the "no encoding" row in Table 1 is the internal GCN baseline from Qian et al. (2019), which already incorporates BERT node embeddings and spatial bounding box features; what is being ablated is only the additional positional encoding on top of that. **Table 2** compares encoding functions for ROPE: | Dataset | Index | Sine | WL F1 | WG F1 | |:--------|:-----:|:----:|------:|------:| | Payment | ✓ | | 66.09 | 84.94 | | Payment | | ✓ | 72.41 | 86.53 | | Payment | ✓ | ✓ | 70.94 | 85.66 | | FUNSD | ✓ | | 53.16 | 87.37 | | FUNSD | | ✓ | 55.48 | 88.94 | | FUNSD | ✓ | ✓ | 54.14 | 89.12 | Results are compared against the no-encoding GCN baseline (74.55 WL F1 on payment). Both encoding functions improve over the no-ROPE variants. For word grouping, the combined Index+Sine setting is best on both datasets. For word labeling, sinusoidal encoding alone edges out the combined variant on payment (72.41 vs. 70.94), suggesting slight interference between the two functions at this task. Notably, all ROPE variants trail the full no-encoding baseline on WL F1, meaning ROPE's gains appear primarily through reducing the performance gap rather than exceeding the baseline when EdgeGeo is absent. **Sensitivity to OCR order quality**: the authors shuffle a fraction of words in the input reading order before feeding into ROPE. Performance is robust up to roughly 30% shuffling; beyond that, degradation is more pronounced for word grouping than word labeling, suggesting that grouping relies more heavily on sequential consistency. ## What are the outcomes/conclusions? ROPE consistently improves GCN-based document entity extraction, with gains of up to 8.4 F1 points on the payment dataset (word labeling) and around 2.5 F1 points on FUNSD (word labeling). Reading order encoding and geometric edge features provide orthogonal improvements, suggesting they capture different aspects of document structure. The equivariance property is a meaningful design choice: by defining codes relative to the target word, ROPE avoids dependence on absolute document position and remains consistent when local neighborhoods span column or section boundaries. The paper is narrow in scope: it applies ROPE only to a GCN with self-attention aggregation, and only to two entity extraction tasks (node and edge classification). The payment dataset is proprietary and not released, limiting reproducibility of the larger-corpus results. Results on FUNSD are modest. The authors acknowledge that ROPE is compatible with other GCN-based document tasks but do not demonstrate this directly. One limitation not explicitly acknowledged by the authors: the paper predates the dominant transformer-based document models (LayoutLM family, Donut, etc.), and ROPE's benefit in an architecture where positional encoding is already handled by cross-attention or 2D sinusoidal embeddings is untested. The $\beta$-skeleton graph construction also requires pre-computed bounding box information, so ROPE is not applicable to end-to-end pixel-level models. ## Reproducibility ### Models - The GCN architecture is fully described: 2-layer MLP node updates (128 hidden), 3-layer multi-head self-attention aggregation (4 heads, 32 head size), 7 or 2 GCN hops depending on dataset. - Node input: BERT embeddings (exact variant unspecified in the paper; likely BERT-base given the 2021 era and compute budget) plus 2D bounding box spatial embeddings. - Edge input: spatial geometric features (relative distances, aspect ratios) plus MobileNetV3 visual embeddings. - No model weights or code repository are released with the paper. ### Algorithms - Training: Adam optimizer, $lr = 10^{-4}$, warm-up proportion 0.01, batch size 1, cross-entropy loss. - Graph construction: $\beta$-skeleton with $\beta = 1$; ball-of-sight strategy. - ROPE: integer code assignment by OCR order; optionally mapped through sinusoidal encoding with 3 base frequencies. - No gradient clipping or mixed precision details are reported. ### Data - **FUNSD**: publicly available at [guillaumejaume.github.io/FUNSD](https://guillaumejaume.github.io/FUNSD/). 199 annotated scanned forms, 4 entity types, official 75/25 split. License: not formally specified by the authors, but widely used for academic research. - **Payment**: proprietary, approximately 18K single-page payment documents with 13 entity types, annotated by human labelers. Not released. OCR from Google Cloud Vision. ### Evaluation - Word labeling metric: multi-class node classification F1. - Word grouping metric: binary edge classification F1 (precision + recall also reported). - No error bars, confidence intervals, or multiple-run variance reported. - Comparison is against an internal GCN baseline; no comparison to externally published methods on FUNSD. ### Hardware - Training: 8 Tesla P100 GPUs. - Training time: approximately 1 day on the payment dataset (largest corpus). - No inference latency or memory consumption reported. ## BibTeX ```bibtex @inproceedings{lee-etal-2021-rope, title = {{ROPE}: Reading Order Equivariant Positional Encoding for Graph-based Document Information Extraction}, author = {Lee, Chen-Yu and Li, Chun-Liang and Wang, Chu and Wang, Renshen and Fujii, Yasuhisa and Qin, Siyang and Popat, Ashok and Pfister, Tomas}, booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)}, year = {2021}, publisher = {Association for Computational Linguistics}, address = {Online}, doi = {10.18653/v1/2021.acl-short.41}, url = {https://aclanthology.org/2021.acl-short.41}, } ```