# DocTrack: Eye-Tracking Ground Truth for Visually-Rich Document Reading Order > DocTrack is the first benchmark dataset for reading order in visually-rich documents grounded in human eye-tracking data, covering weak, structured, and infographic document types. - Authors: Hunter Heidenreich - Published: 2023-10-23 - Updated: 2026-04-02 - Dataset: DocTrack Dataset - Paper: Preprint (arXiv) - Paper: Published (Findings of EMNLP 2023) - Source: Bevaya Labs, --- ## TL;DR DocTrack is a benchmark dataset of 539 visually-rich documents with reading order ground truth derived from human eye-tracking experiments using Tobii hardware. The authors use it to study whether human reading order actually helps document AI models, finding that simple heuristic orderings often outperform both OCR default order and true eye-tracking order on downstream understanding tasks. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Resource}}$: The paper's primary contribution is the DocTrack dataset itself: a collection of visually-rich document images paired with reading order labels derived from eye-tracking experiments. The dataset construction methodology, participant protocols, and annotation voting procedure are the core of the paper. **Secondary:** $\Psi\_{\text{Evaluation}}$: The paper also conducts a systematic evaluation study measuring how different reading order strategies (OCR default, Z-order heuristics, XYLayout rules, and learned multimodal models) affect downstream task performance. This benchmarking analysis is substantial but serves to validate the dataset's utility rather than propose a new model. **Secondary:** $\Psi\_{\text{Method}}$: A "preordering pipeline" is introduced alongside atomic comparison models (Box, Text, Text+Box, Text+Box+Image) for generating human-like reading orders. ## What is the motivation? Visually-rich documents (VRDs) (forms, tables, infographics, and other document types mixing text with graphical elements) present a challenge for document AI models because those models require a linearized token sequence as input. The standard approach is to either follow OCR engine output order or apply simple left-to-right, top-to-bottom rules. Both approaches diverge significantly from how humans actually read such documents. Prior work on reading order, notably ReadingBank (Wang et al., 2021), derived ground truth from the XML structure of Word source files. The authors argue this proxy measure may not reflect actual human behavior, particularly for complex layouts. No dataset existed that directly measured human reading order via eye tracking for real-world document images. DocTrack fills that gap. The motivation is also evaluative: the authors want to know whether bringing machine reading order closer to human reading order actually improves document understanding performance, a question that had not been rigorously tested. ## What is the novelty? The core novelty is the use of eye-tracking hardware to collect reading order ground truth directly from human participants reading real document images. This avoids the assumptions embedded in proxy measures like XML source order. **Dataset construction.** Documents are drawn from three existing sources: FUNSD (form understanding), SeaBill (structured tabular shipping documents), and InfographicVQA. These form three subsets within DocTrack: - **WEAK**: 149 train / 50 test documents from FUNSD, characterized by the normal-Z reading pattern - **STRUCTURED**: 160 train / 50 test documents from SeaBill, with local-priority reading patterns - **INFOGRAPH**: 100 train / 30 test documents from InfographicVQA, exhibiting cross-modal interaction and visual instruction patterns The full dataset contains 539 documents with 39,671 semantic entities and 86,054 tokens across train and test splits. **Eye-tracking methodology.** Five participants (graduate and undergraduate students) were recruited and divided into five groups, each assigned a disjoint subset of the data. Participants read documents on an HP 24-inch 1080p display using Tobii TX300 and Tobii Studio to record fixation points, fixation time, fixation frequency, saccade distance, and pupil size. Two of the five participants labeled an overlapping subset for annotation agreement; final labels were selected by majority vote across all five participants. **Gaze-to-OCR alignment.** Raw gaze trajectories contain noise. The authors apply three corrections: (1) when a gaze point lands on the periphery of a known OCR bounding box within a Euclidean distance threshold, the ordinal position of that peripheral gaze point is used as the reading sequence index for that box; (2) when gaze points are missing between two known points, ordinal positions are filled from the surrounding adjacent readings; (3) gaze points that represent repeated return fixations are removed, keeping only the first occurrence. This alignment maps continuous eye movement traces to a discrete reading order over OCR bounding boxes. **Reading pattern taxonomy.** The authors identify four qualitative patterns observed in the data: 1. **Normal-Z**: left-to-right, line-by-line zigzag, typical of plain or weakly structured text 2. **Local priority**: attention to cell content before surrounding structure, characteristic of tables and forms 3. **Cross-modal interaction**: radial back-and-forth between graphical elements and their text labels, seen in pie charts, bar charts, and line graphs 4. **Visual instruction**: backtracking over hierarchical diagrams such as flowcharts, where readers re-read earlier nodes for contextual reference **Preordering pipeline.** The authors propose a preprocessing step that reorders OCR bounding boxes before feeding them into downstream document AI models. Four atomic comparison models each predict which of two bounding boxes comes first in reading order, framed as a binary classification: $$p = f(b\_i : b\_j) = \begin{cases} 0, & \text{if } r[b\_i] < r[b\_j] \\ 1, & \text{if } r[b\_i] > r[b\_j] \end{cases}$$ The four model variants are: - **Box**: centroid coordinates fed into a multi-layer Transformer - **Text**: BERT encoding of token content - **Text+Box**: LayoutLM joint encoding of text and 2D position - **Text+Box+Image**: LayoutLMv2 joint encoding of text, 2D position, and image ROI A bubble-sort variant uses pairwise comparison outputs to produce a final sorted sequence. ## What experiments were performed? The paper conducts both intrinsic and extrinsic evaluations. **Intrinsic evaluation.** The four comparison models are evaluated by measuring rank correlation between their predicted order and the eye-tracking ground truth, using Kendall's $\tau$ and Spearman's $\rho$. Results across the three subsets (Table 2 in the paper) show: - Box alone performs worst overall ($\tau = 0.5992$, $\rho = 0.6366$) - Text+Box+Image achieves the best overall performance ($\tau = 0.8052$, $\rho = 0.8665$) - For WEAK documents, visual features matter: Text+Box+Image beats Text+Box - For STRUCTURED and INFOGRAPH documents, Text+Box is nearly as strong or stronger, suggesting visual features provide diminishing returns when layout structure is clear A notable finding is that 38.16% of WEAK documents have missing gaze points (gaze fell outside OCR boxes), compared to 12.98% for STRUCTURED and 9.55% for INFOGRAPH. **Extrinsic evaluation.** The impact of reading order on downstream tasks is measured using: - Semantic Entity Recognition (SER) on WEAK and STRUCTURED subsets, reported as precision, recall, and F1 - Visual Question Answering on INFOGRAPH, reported as Average Normalized Levenshtein Similarity (ANLS) Three backbone models are tested: BERT (text only), LayoutLMv2 (text + position + image), and LayoutLMv3 (text + position + image with unified masking pre-training). Conditions compared include: EYE (raw human eye-tracking order), EYE++ (eye order with position features added), DEFAULT-OCR, Z-ORDER, XYLayout rule-based sorting, and the four machine-generated model orders. ## What are the outcomes/conclusions? The results suggest a counterintuitive finding: true human eye-tracking order (EYE) does not consistently produce better downstream performance than simpler alternatives. Across BERT, LayoutLMv2, and LayoutLMv3, the Z-ORDER heuristic (top-to-bottom, left-to-right coordinate sorting) frequently achieves the highest or near-highest F1 on SER tasks and ANLS on VQA. For example, under LayoutLMv3 on WEAK, Z-ORDER reaches F1 of 93.73%, while EYE achieves only 90.97% and EYE++ 91.33%. The model-generated orders (MODEL-T+B+I) reach 93.56%, close to Z-ORDER but trained on eye data. The pattern holds on STRUCTURED and INFOGRAPH as well. The authors interpret this as evidence that document AI models have learned implicit reading order assumptions aligned with simple coordinate-based rules rather than natural human gaze patterns. The eye-tracking order captures regressions, local backtracking, and cross-modal jumps that do not benefit sequence models. Key conclusions: - Document AI models still cannot read VRDs as flexibly as humans - Human reading order is nuanced (four distinct patterns) and encodes more cognitive strategy than current models can exploit - Simple rule-based sorting often suffices for current architectures - The DocTrack dataset provides a resource for studying the human-machine reading order gap more deeply **Limitations acknowledged by the authors.** Due to high annotation cost, most documents were labeled by only one participant (with a small overlapping subset assigned to two participants). The authors state that "the inner-agreement rate is not available for the current dataset" because full multi-annotator labeling was not performed. The study also ignores fixation duration, back-gaze frequency, and fixation count, which may carry additional signal. ## Reproducibility ### Models The atomic comparison models use standard pretrained weights: BERT-base for the Text model, LayoutLM for Text+Box, and LayoutLMv2 for Text+Box+Image. No custom architecture is introduced. Exact model sizes and layer configurations are not specified in the paper beyond the pretrained model names. ### Algorithms The preordering algorithm is a pairwise comparison wrapped in bubble sort (Algorithm 1 in the paper). The atomic comparison takes two bounding box representations and predicts binary precedence. No optimizer, learning rate, batch size, or training duration details are provided in the paper. ### Data - DocTrack is hosted at https://github.com/hint-lab/doctrack - The dataset builds on document images from FUNSD (CC-BY-4.0), SeaBill (license not stated in this paper), and InfographicVQA (license not stated in this paper) - Eye-tracking annotations were collected under institutional ethics approval with informed participant consent; participants approved data release for research purposes - The repository carries an Apache-2.0 license, but the README includes a non-commercial restriction note: "The DocTrack dataset should only be used for non-commercial research purposes. For any person/institution/company working on this direction, please contact us for a commercial license." This conflicts with the Apache-2.0 header; treat as research-only until clarified with the authors. - Five participants labeled the data; the dataset was divided into five disjoint parts, with each participant assigned one part. Two of the five participants were additionally assigned an overlapping subset; final labels were selected by majority vote across all five participants. The paper's Limitations section notes that "due to the high annotation cost, the annotation has not been done by multiple annotators. Therefore, the inner-agreement rate is not available for the current dataset." - Missing gaze rates vary substantially by document type (9.55% for INFOGRAPH to 38.16% for WEAK) ### Evaluation - Intrinsic: Kendall's $\tau$ and Spearman's $\rho$ correlation between predicted and human reading order - Extrinsic SER: precision, recall, F1 on semantic entity recognition - Extrinsic VQA: ANLS on InfographicVQA subset - Baselines include DEFAULT-OCR, Z-ORDER, XYLayout, and human eye-tracking conditions - No error bars or significance tests are reported - The paper does not specify the number of training runs or random seeds ### Hardware No training hardware, GPU specifications, training time, or inference costs are reported in the paper. ## BibTeX ```bibtex @inproceedings{wang-etal-2023-doctrack, title = "{D}oc{T}rack: A Visually-Rich Document Dataset Really Aligned with Human Eye Movement for Machine Reading", author = "Wang, Hao and Wang, Qingxuan and Li, Yue and Wang, Changqing and Chu, Chenhui and Wang, Rui", editor = "Bouamor, Houda and Pino, Juan and Bali, Kalika", booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2023", month = dec, year = "2023", address = "Singapore", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2023.findings-emnlp.344/", doi = "10.18653/v1/2023.findings-emnlp.344", pages = "5176--5189" } ```