# DocGenome: An Open Large-scale Scientific Document Benchmark

> A 500k-document scientific dataset that annotates 13 entity types and 6 logical relationships per document, derived automatically from arXiv LaTeX source.

- Authors: Bevaya AI Team
- Published: 2024-06-17
- Updated: 2026-03-12
- Paper: Preprint (arXiv) <https://arxiv.org/abs/2406.11633>
- Dataset: DocGenome <https://huggingface.co/datasets/U4R/DocGenome>
- Dataset: DocGenome (Test QA) <https://huggingface.co/datasets/U4R/DocGenome-Testset-DocQA>
- Code: DocParser <https://github.com/UniModal4Reasoning/DocGenome>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## TL;DR

DocGenome is a 500K-document scientific benchmark built from arXiv LaTeX source, annotating 13 entity types and 6 logical relationships across 6.8M pages spanning 153 disciplines. It enables a 7-task benchmark suite covering layout detection, document transformation, and MLLM reasoning. The dataset and pipeline (DocParser) are fully open-source; ground truth is recovered directly from LaTeX compilation rather than PDF-based extraction, which largely eliminates OCR noise.

## What kind of paper is this?

**Dominant:** $\Psi\_{\text{Resource}}$: the primary contribution is the DocGenome dataset and its automated construction pipeline (DocParser). The paper's center of gravity is dataset curation, annotation quality control, and the resulting benchmark suite.

**Secondary:** $\Psi\_{\text{Method}}$: the paper also trains two baseline transformation models (EqVLM-B, TableVLM-B) and fine-tunes YOLOv8 for layout detection to demonstrate the dataset's utility.

The benchmark covers **7 tasks**: document classification, visual grounding, layout detection, equation-to-LaTeX, table-to-LaTeX, single-page QA, and multi-page QA.

## What is the motivation?

Scientific documents are unusually demanding for Multimodal Large Language Models (MLLMs). A typical arXiv paper combines structured prose, mathematical notation, figures, tables, cross-references, and multi-column layouts. Understanding these documents requires knowing not just what each element looks like, but how elements relate to one another logically.

Existing document datasets fall short in two ways. Purely visual datasets like PubLayNet and DocLayNet annotate bounding boxes and category labels, but not the logical structure connecting them: which caption belongs to which figure, which paragraph belongs under which section heading, which equation is referenced by which sentence. Datasets that do capture richer structure are either small (requiring expensive human annotation) or domain-narrow.

The paper's argument is that the field lacks a large-scale, open resource that aligns visual layout with logical structure across a broad range of scientific disciplines. DocGenome is that resource, covering 500K documents from **153 disciplines across 8 primary scientific fields**.

## What is the novelty?

The core innovation is the **logical graph annotation** derived from the LaTeX compilation process itself. Rather than inferring structure from the rendered PDF (which discards authorial intent), the DocParser pipeline intercepts the LaTeX source before compilation, recovering the document's logical hierarchy directly.

### The "Genome" Graph

Each document is represented as a graph: nodes are semantic units (one of 13 entity types), and edges are one of 6 logical relationship types.

**Entity types (13):** `Algorithm`, `Caption`, `Equation`, `Figure`, `Footnote`, `List`, `Table`, `Text`, `Text-EQ` (text with inline math), `Title` (section titles), `PaperTitle`, `Code`, `Abstract`.

**Relationship types (6):**

| Relation | Description |
| :--- | :--- |
| **Identical** | Two units share the same source code (e.g., text split across columns or pages). |
| **Title adjacent** | Two adjacent section titles. |
| **Subordinate** | One unit is a subclass of another (e.g., `\section{Introduction}` $\rightarrow$ a paragraph within it). |
| **Non-title adjacent** | Two adjacent text or equation units. |
| **Explicitly-referred** | One unit references another via `\ref` or a footnote (e.g., "As shown in Figure 5" $\rightarrow$ Figure 5). |
| **Implicitly-referred** | A caption unit refers to its corresponding float (e.g., Table Caption 1 $\rightarrow$ Table 1). |

Reading order is not stored as an explicit relationship; it is the implicit ordering produced by the TexSoup segmentation in Stage 2 of the pipeline.

## What experiments were performed?

### The DocParser Pipeline

DocParser is a 4-stage automated annotation pipeline:

1. **Preprocessing**: Cleaning LaTeX source, expanding imports (`\input`, `\include`), and normalizing figure formats to PNG.
2. **Unit Segmentation**: Using `TexSoup` to parse the LaTeX AST into a linear sequence of semantic units, establishing ground-truth reading order.
3. **Attribute and Relation Retrieval**: Entities and relations are derived structurally from the LaTeX parse tree. Hierarchical relations (subordinate, adjacent) apply to fixed-form units; reference relations (explicit via `\ref`/`\label`, implicit for caption-float pairs) apply to floating elements.
4. **Color Rendering for Bounding Box Extraction**: To avoid fuzzy text matching when locating units in the rendered PDF, the pipeline renders each target unit $u\_i$ in isolation: $u\_i$ is rendered black on white while all other content is rendered white. Differencing against a blank page yields a pixel-perfect bounding box for $u\_i$.

### Quality Control

Two metrics filter the automatic annotations:

- **Intra-consistency ($IoU\_{\text{intra}}$)**: Checks for illegitimate bounding-box overlap within a paper. For $N$ bounding boxes $B\_1, \dots, B\_N$ in a document, with $J(\cdot, \cdot)$ denoting Intersection-over-Union:

$$IoU\_{\text{intra}} = \frac{1}{N(N-1)} \sum\_{i=1}^{N} \sum\_{j=1,\, j \neq i}^{N} J(B\_i, B\_j)$$

- **Alignment ($IoU\_{\text{align}}$)**: Compares generated boxes against DocXChain (a document parsing toolkit from Alibaba DAMO Academy) to flag anomalies where the automatic annotation diverges from an independent detector. Given reference boxes $G\_1, \dots, G\_N$:

$$IoU\_{\text{align}} = \frac{1}{N} \sum\_{i=1}^{N} J(B\_i, G\_i)$$

Only documents in the top quality tier (Tier-1, requiring $IoU\_{\text{intra}}$ < 0.05% and $IoU\_{\text{align}}$ > 60%) are eligible for the test set. From this pool, 1,004 documents were sampled to form DocGenome-test. QA pairs for the test set were generated by GPT-4V (2 single-page + 2 multi-page per paper) and validated through a 3-step process: (1) automated review by Kimi, (2) two independent faculty reviews with 0-3 confidence scoring, and (3) cross-verification against the original document text.

### Benchmark Tasks and Baselines

The authors construct a 7-task benchmark using DocGenome-test:

- **Layout Detection**: YOLOv8 fine-tuned on increasing amounts of DocGenome training data, compared against DocXChain and models trained on smaller datasets.
- **Document Transformation**: EqVLM-B (equation-to-LaTeX) and TableVLM-B (table-to-LaTeX) fine-tuned from Pix2Struct-B, evaluated against Mathpix (commercial API) using normalized edit distance.
- **MLLM Reasoning**: Zero-shot evaluation of GPT-4V, GPT-4o, Qwen-VL, InternVL, and others on document classification, single-page QA, and multi-page QA.

## What are the outcomes/conclusions?

### Layout Detection

Fine-tuning YOLOv8 on DocGenome substantially outperforms baselines trained on smaller datasets. The mAP for Footnotes improves from 48.43% to 86.52% when scaling to 700K pages of training data. On out-of-distribution (OOD) human-annotated layout data, YOLOv8 trained on DocGenome (mAP 50.15) also outperforms DocXChain (mAP 37.99), suggesting the layout annotations generalize reasonably beyond the arXiv distribution.

### Document Transformation

On the in-distribution DocGenome-test set, both trained models outperform Mathpix:

| Task | Model | Edit Distance |
| :--- | :--- | :--- |
| Eq-to-LaTeX | Mathpix | 0.4738 |
| Eq-to-LaTeX | EqVLM-B (1M samples) | **0.2111** |
| Table-to-LaTeX | Mathpix | 0.4436 |
| Table-to-LaTeX | TableVLM-B (500K samples) | **0.2223** |

On OOD Sci-Hub data, however, the picture reverses for the equation task. EqVLM-B's edit distance rises to 0.6627, while Mathpix holds at 0.4873. The trained models appear to overfit to the arXiv/LaTeX distribution and do not generalize as cleanly as the in-distribution numbers suggest.

### MLLM Reasoning

Neither GPT-4V nor GPT-4o dominates across all tasks:

| Model | Classification | Single-page QA | Multi-page QA |
| :--- | :--- | :--- | :--- |
| GPT-4V | **0.9821** | 0.6101 | 0.6501 |
| GPT-4o | 0.9761 | **0.7183** | **0.6762** |

GPT-4V leads on document classification; GPT-4o leads on both QA tasks. Open-source models (Qwen-VL, InternVL) trail significantly on the structural reasoning tasks, suggesting that scale and instruction tuning still matter for document understanding.

The paper concludes that large-scale structural annotation is a meaningful lever for improving document understanding models, but the OOD gaps for transformation tasks indicate that distribution breadth matters as much as scale.

## Bevaya Analysis: Mapping to Matter vs. Meaning

The following is internal team analysis, not the paper's own framing. We map all 13 DocGenome entity types and all 6 relation types against the [Matter vs. Meaning framework](./matter-vs-meaning/) to assess coverage, identify gaps, and understand where the dataset fits into our annotation and pipeline design.

### Entity Type Mapping

| DocGenome Entity | Visual Primitive | Logical Role | Notes |
| :--- | :--- | :--- | :--- |
| `PaperTitle` | `Text` | `Title` | The document root headline. Clean 1-to-1 match. |
| `Title` (section) | `Text` | `SectionHeader` | Maps to H1-H6 depth; DocGenome does not record heading level. |
| `Abstract` | `Text` | `Abstract` | Clean 1-to-1 match. |
| `Text` | `Text` | `Body` | Standard prose. Clean 1-to-1 match. |
| `Text-EQ` | `Text` + `Formula` span | `Body` + `InlineEquation` child | DocGenome treats this as a distinct top-level category. Our framework handles it as a `Text` block (`Body`) with one or more `InlineEquation` enrichment spans inside, per the Span vs. Block model. **Do not train a separate detector for `Text-EQ`**: detect the block as `Text`, then run a formula spotter on the crop. |
| `Equation` | `Formula` | `DisplayEquation` | Standalone math block. Clean 1-to-1 match. |
| `Figure` | `Image` | `Figure` | Visual content. Clean 1-to-1 match. |
| `Table` | `Table` | *(implicit)* | DocGenome annotates the outer bounding box. TSR is a separate downstream step. |
| `Caption` | `Text` | `Caption` | Linked to its float via the `Implicitly-referred` relation, which maps to our `group_with` edge. |
| `List` | `Text` | `ListItem` | **Granularity mismatch:** DocGenome annotates the container block, not individual items. Per the framework's "Why There Is No ListContainer" rationale, we map this to `ListItem` and note the mismatch. Individual bullet boundaries must be recovered by a separate step. |
| `Footnote` | `Text` | `Footnote` | Clean 1-to-1 match. |
| `Code` | `Text` | `Code` | Computer source code. Could alternatively map to `Image` if rendered as a screenshot, but in LaTeX-sourced papers it is always machine-readable `Text`. |
| `Algorithm` | `Text` or `Image` | `Code` | Ambiguous primitive. Algorithm pseudocode in LaTeX is typeset text (`Text` → `Code`), but some papers render it as a figure. DocGenome does not distinguish these cases. Treat as `Text` / `Code` unless visual inspection confirms otherwise. |

### Relation Type Mapping

| DocGenome Relation | Framework Equivalent | Notes |
| :--- | :--- | :--- |
| `Implicitly-referred` | `group_with(Figure/Table, Caption)` | Direct match. This is the canonical caption-to-float binding. |
| `Subordinate` | `header_of(SectionHeader, Body)` | Direct match for section-to-content hierarchy. Also covers nested sections (SectionHeader-to-SectionHeader at different depths). |
| `Explicitly-referred` | *(unmapped)* | In-text `\ref` and footnote pointer relations. The framework's current relation set has no general citation or cross-reference edge. This is a gap worth addressing if we build reading-order or citation graphs. |
| `Identical` | *(unmapped)* | Marks units whose source code is shared but renders in two locations (e.g., text split across columns or pages). The framework has no cross-page or column-continuation relation. Important for multi-column layout reconstruction. |
| `Title adjacent` | *(reading order)* | Two consecutive section titles. In the framework this is implicit reading order, not an explicit relation. No structural equivalent needed unless building explicit TOC graphs. |
| `Non-title adjacent` | *(reading order)* | Two consecutive text or equation units. Same as above: implicit reading order, not a structural edge the framework needs to represent explicitly. |

### Coverage Gaps

**Present in DocGenome, absent from the framework's detection targets:**

- The `Identical` relation (cross-column/cross-page text continuation) has no equivalent in the current framework relation set. This matters for multi-column scientific papers where a paragraph spans two columns: without this relation, a downstream reader has no signal to stitch the halves together. Worth adding as a `continues_from` edge in a future revision.
- The `Explicitly-referred` relation (in-text `\ref` citations) is similarly absent. It would be useful for building document knowledge graphs but is not needed for the core layout-to-text pipeline.

**Present in the framework, not annotated by DocGenome:**

DocGenome is scoped to arXiv scientific papers and annotates only what is needed for that domain. The following roles from the framework are therefore absent:

| Missing Role | Why It Matters | Where It Appears |
| :--- | :--- | :--- |
| `Author` / `Affiliation` | Needed for paper metadata extraction | Paper header / byline |
| `Keywords` | Indexing and classification | After abstract |
| `BibEntry` | Reference list parsing | Back matter |
| `FormulaNumber` | Linking equations to in-text references | Equation label (e.g., "(1)") |
| `PageHeader` / `PageFooter` | Running navigation strips | Top/bottom of page |
| `PageNumber` | Pagination | Corner of each page |

These are not DocGenome failures; they reflect a deliberate scope decision. Any pipeline that ingests DocGenome-trained models and processes full scientific papers will need to supplement them with detectors for the above roles.

### Summary Assessment

DocGenome's entity taxonomy covers the **core content primitives** of the framework well: the 13 types map cleanly except for `Text-EQ` (which should be decomposed rather than treated as a separate class) and `Algorithm` (which requires visual disambiguation). Its relation set covers the two most structurally important edges (`group_with` and `header_of`) but leaves cross-page continuation and citation links unaddressed. The dataset is a strong foundation for scientific document layout, but will require supplementation for full-document metadata extraction and multi-column reconstruction.

## Reproducibility

### Models

EqVLM-B and TableVLM-B are both fine-tuned from **Pix2Struct-B (0.2B parameters)**. The paper does not release pretrained weights for these baseline models; researchers would need to retrain from scratch using the released dataset and pipeline code. The same team later released [StructTable-InternVL2-1B](https://huggingface.co/U4R/StructTable-InternVL2-1B), a successor model for table-to-LaTeX trained on DocGenome data, but this is a separate effort not described in the paper.

### Algorithms

- **Layout detection (YOLOv8):** AdamW, learning rate 0.01; 30 epochs
- **Document transformation (EqVLM-B, TableVLM-B):** AdamW, learning rate 5e-5, weight decay 0.01; 30 epochs
- **Hardware:** 64x NVIDIA A100 (80G) GPUs

No additional training tricks (warmup schedule, gradient clipping, mixed precision) are reported in the paper.

### Data

- **Training set:** ~500K arXiv documents (6.8M pages), spanning 153 disciplines across 8 primary scientific fields, derived from LaTeX source via DocParser
- **Test set:** 1,004 documents (9K pages, DocGenome-test), sampled from the Tier-1 quality pool ($IoU\_{\text{intra}}$ < 0.05%, $IoU\_{\text{align}}$ > 60%). QA pairs were generated by GPT-4V (7,028 questions across 1,757 papers), then filtered through a 3-step quality process by 20 PhD/Master's-level annotators. 2,498 QA pairs survived, with 66.93% modified by reviewers
- **Availability:** Training data is publicly available on Hugging Face under CC-BY-4.0. The test QA set is also on Hugging Face but has no license explicitly declared in its metadata (the paper states CC-BY-4.0 for the project overall)
- **Bias:** The dataset is entirely derived from arXiv submissions, meaning it is heavily skewed toward scientific papers with LaTeX source. Applicability to scanned documents, business forms, or non-LaTeX documents is not addressed.

### Evaluation

- **Layout detection:** `mAP@0.5:0.95` (primary COCO metric, averaged over IoU thresholds 0.5 to 0.95)
- **Document transformation:** Normalized edit distance (lower is better)
- **Reasoning tasks:** Top-1 accuracy (classification) and GPT-acc (QA, where GPT-4 judges predicted answers against ground truth as True/False)
- **OOD evaluation:** Sci-Hub data (transformation) and human-annotated layout data; both are reported in Table 6 of the paper
- **Baselines:** DocXChain (layout), Mathpix (transformation), GPT-4V, GPT-4o, Qwen-VL, InternVL (reasoning)
- **Evaluation scripts:** The test set repository includes evaluation scripts (`eval_open_docqa_gpt.py`, `eval_normal_docqa.py`), but reproducing QA evaluation requires OpenAI API access for GPT-acc scoring.
- The paper does not report error bars, significance tests, or multi-seed results for the trained baselines.

### Hardware

- **Training:** 64x NVIDIA A100 (80G); total GPU-hours not reported
- **Inference:** No latency, throughput, or VRAM requirements reported for inference
- **Cost:** No cloud cost or energy consumption estimates reported

## Peer Review History

DocGenome was submitted to ICLR 2025 (Datasets and Benchmarks track) and rejected. The [OpenReview discussion](https://openreview.net/forum?id=CI9JMBAsPg) provides useful context for understanding the paper's reception.

### Reviewer Scores

| Reviewer | Rating | Confidence |
| :--- | :--- | :--- |
| i9VG | 8/10 | 4/5 |
| c3jL | 6/10 | 4/5 |
| 4Uws | 6/10 | 4/5 |
| TmKL | 5/10 | 5/5 |

### Key Criticisms

**Limited novelty in pipeline methodology.** Multiple reviewers (4Uws, TmKL) noted that parsing LaTeX source to derive layout annotations is not new; prior work (GROTOAP, PubLayNet, DocBank) uses similar approaches. The area chair's meta-review summarized this as: "its technical contributions and novelty appear limited."

**Logical relationships are underexploited.** Reviewer 4Uws flagged that while the paper introduces 6 relationship types as a key differentiator, the experiments do not actually leverage them. The benchmarked tasks operate on individual components, not the graph structure. In response, the authors showed that providing layout bounding boxes as visual prompts improves InternVL-1.5 QA accuracy (0.4529 $\rightarrow$ 0.4922 on single-page QA), but this addresses layout detection, not the relationship graph itself.

**GPT-4 dependence in QA generation.** Reviewers i9VG and 4Uws raised concerns about bias introduced by using GPT-4V to generate QA pairs. The authors revealed that from 7,028 initial GPT-4-generated questions across 1,757 papers, only 2,498 QA pairs survived quality filtering, and **66.93% of retained pairs were modified** by the 20 PhD/Master's-level quality checkers. This high editing rate somewhat mitigates the bias concern but also raises questions about the efficiency of the generation process.

**Non-text modality coverage.** Reviewer c3jL noted that QA pairs focus heavily on textual understanding, with limited coverage of chart/plot or table-specific questions, which is a gap for a benchmark targeting multimodal document understanding.

**LaTeX distribution bias.** Reviewer TmKL questioned whether LaTeX-formatted arXiv papers are representative of scientific documents broadly. The authors conducted OOD experiments on 966 Sci-Hub papers (covering medicine, chemistry, biology, humanities, and other fields outside arXiv's core), where their DocGenome-trained YOLOv8 layout detector outperformed DocXChain (mAP 50.15 vs. 37.99), partially addressing this concern for layout detection. However, the equation-to-LaTeX task showed the opposite pattern on OOD data (edit distance 0.6627 vs. Mathpix's 0.4873).

### Noteworthy Author Revelations

- The open-source tool [MinerU](https://github.com/opendatalab/MinerU) reportedly uses DocGenome for training its equation and table parsing models.
- The full QA pipeline details: 7,028 questions generated $\rightarrow$ quality-checked by 20 annotators $\rightarrow$ 2,498 retained (editing rate 66.93%).
- Evaluation metric limitations were acknowledged: edit distance and BLEU do not capture semantic equivalence of different LaTeX expressions (e.g., `\frac{a}{b}` vs. `a/b`). The authors mention ongoing work on a rendering-based metric that compares visual similarity of compiled LaTeX output.

## BibTeX

```bibtex
@misc{xia2024docgenome,
  title={DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Language Models},
  author={Xia, Renqiu and Mao, Song and Yan, Xiangchao and Zhou, Hongbin and Zhang, Bo and Peng, Haoyang and Pi, Jiahao and Fu, Daocheng and Wu, Wenjie and Ye, Hancheng and Feng, Shiyang and Wang, Bin and Xu, Chao and He, Conghui and Cai, Pinlong and Dou, Min and Shi, Botian and Zhou, Sheng and Wang, Yongwei and Wang, Bin and Yan, Junchi and Wu, Fei and Qiao, Yu},
  year={2024},
  eprint={2406.11633},
  archivePrefix={arXiv},
  primaryClass={cs.CV}
}
```