# WordScape: Multilingual Layout Annotations from Web Crawl Data > A pipeline for extracting millions of multilingual document pages with layout annotations from Common Crawl Word files, producing 40M+ pages across 136 languages. - Authors: Bevaya AI Team - Published: 2023-12-15 - Updated: 2026-03-15 - Paper: Preprint (arXiv) - Paper: Published (NeurIPS 2023) - Code: WordScape Pipeline - Source: Bevaya Labs, --- ## TL;DR WordScape is an open-source pipeline that extracts Word documents from Common Crawl, parses their Open XML structure, and produces layout-annotated page images with text. The authors release 9.5M URLs to Word files that yield 40M+ pages across 136 languages and 28 topic categories. Pre-training on WordScape annotations consistently improves layout detection over random initialization and PubLayNet pre-training, particularly in low-resource regimes. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Resource}}$: the headline contribution is the WordScape pipeline and its associated URL corpus. The paper's center of gravity is the data creation methodology, dataset statistics (language distribution, topic modeling, entity breakdowns), and quality analysis. The pipeline itself is the released artifact, not a fixed dataset. **Secondary:** $\Psi\_{\text{Method}}$: the bounding-box annotation algorithm (colorization of Open XML elements, rendering via LibreOffice, color detection with OpenCV) is a distinct technical contribution, though it serves the primary goal of dataset creation. ## What is the motivation? Existing document layout datasets are limited along two axes: 1. **Scale vs. quality tradeoff:** Human-annotated datasets (DocLayNet, FUNSD) top out at ~80K pages. Auto-annotated alternatives (PubLayNet, DocBank) reach hundreds of thousands but are confined to English scientific papers. 2. **Domain and language diversity:** Nearly all large-scale layout datasets draw from PubMed Central or arXiv. Real-world document understanding requires coverage across industries, cultures, and languages, especially low-resource ones. Word documents are ubiquitous on the web, used in formal and professional contexts, and carry structured Open XML metadata that enables automatic annotation. Common Crawl provides a scalable source of such documents without manual curation. ## What is the novelty? The core insight is that Word's Open XML format already encodes the semantic structure of a document (headings, tables, lists, figures) in its XML tags and built-in styles. Rather than training a model to predict layout, WordScape *reads* the structure directly from the source format. The annotation pipeline has three stages: 1. **URL Extraction:** Parse Common Crawl `.wat` metadata files for URLs ending in `.doc`/`.docx`. Deduplicate per-snapshot and globally. 2. **Document Download:** Send HTTP requests, filter by content-type, apply OLE-based malware checks (VBA macros, encryption, flash embeds), hash-based deduplication. 3. **Processing & Annotation:** - **Category identification:** Parse Open XML using `python-docx`. Identify elements via (a) built-in Word styles (headings, titles), (b) native XML tags (tables, headers, footers, figures), or (c) heuristic fallbacks (font-size ranking, bullet detection). - **Colorization:** Edit XML tags to color each element category uniquely. Render via LibreOffice to JPEG. - **Bounding box extraction:** Detect colors on rendered images with OpenCV to produce per-category bounding boxes. - **Text extraction:** Document-level text from `python-docx` (in reading order); page-level with word bounding boxes from `PDFPlumber`. - **Language ID:** FastText classifier (176 languages) at both document and page level. The pipeline annotates **30 semantic entity categories**: Title, Heading Levels 1-9, Plain Text, List Item, Header, Footer, Table (plus Table Header, Table Header Cell, Table Cell, Table Row, Table Column), Table of Contents, Bibliography, Quote, Equation, Figure, Table Caption, Footnote, Annotation, Form Field, and Form Tag. An annotation reliability score $R$ captures the proportion of entities identified via built-in/XML methods vs. heuristics, weighted by character count: $$R = \sum\_{i=1}^{N} \gamma\_i r\_i, \quad \gamma\_i = \frac{c\_i}{\sum\_{j=1}^{N} c\_j}, \quad r\_i = \frac{b\_i}{b\_i + h\_i}$$ where $c\_i$ is the character count of entity $i$, $b\_i$ counts built-in/XML-tagged annotations, and $h\_i$ counts heuristic-based annotations. ## What experiments were performed? The authors validate WordScape annotations on three downstream tasks, each using a "pre-train on WordScape, then fine-tune on target" protocol: **1. Text Detection on FUNSD** (199 scanned forms, word-level bounding boxes) - Model: Faster R-CNN - Pre-training on 10K WordScape pages + fine-tuning on just 25 FUNSD samples exceeds full-FUNSD performance (F1 0.840 vs. 0.772), a 6$\times$ reduction in manual labeling cost. **2. Table Detection on ICDAR 2019 cTDaR** (600 training / 240 test images) - Model: YOLOv8m, 640$\times$640 resolution, SGD then AdamW fine-tuning - Pre-training on 5K WordScape table pages improves mAP@[0.50:0.95] by ~5.5 points in the 75-sample regime (0.924 vs. 0.869). **3. Layout Analysis on DocLayNet** (80K+ pages, 11 classes) - Model: YOLOv5, 200K pre-training images - WordScape pre-training (mAP 0.508 at 1K fine-tune) outperforms PubLayNet pre-training (0.467) and random initialization (0.299). The gap narrows but persists at full 69K fine-tuning (0.755 vs. 0.745 vs. 0.753). **4. Handcrafted Scientific Dataset** (2,500 pages, 31 categories, 8 scientific domains) - Models: DETR and YOLOv8 - Pre-training on WordScape reduces the need for downstream labeled data; improvements are most pronounced at smaller fine-tuning budgets. ## What are the outcomes/conclusions? **Key findings:** - From a single Common Crawl snapshot (Nov/Dec 2022), the pipeline produced 1M annotated documents (5.5M pages) after filtering. Across 6 snapshots, 9.5M unique URLs were collected, projecting to 40M+ pages. - The language distribution is heavily skewed: Russian (2M pages) and English (1M pages) dominate, with a long tail down to ~1K pages for Tajik and Urdu. 136 languages total are represented. - Entity distribution is imbalanced: table cells account for ~50% of all 173M bounding boxes. Excluding table sub-elements, the distribution flattens to a more even split across headings, plain text, and list items. - Topic modeling (Google Cloud NLP API on 25K documents) reveals the top domains are Law & Government (15.3%), Forms/Templates (9.0%), and Education (5.6%). There are significant cross-language differences: Law & Government dominates Russian (~37%) but is rare in other languages. - Pre-training on WordScape consistently outperforms PubLayNet pre-training and random initialization across all benchmarks, with the largest gains in low-data regimes. **Limitations the authors acknowledge:** - Bounding box annotation quality depends on the assumption that document formatting (built-in styles, font sizes) correlates with user intent. Heuristic-based annotations are less reliable. - The pipeline does not address toxic or offensive content filtering. - URL-based distribution means the dataset cannot be released as a fixed, downloadable corpus; documents must be re-crawled, and availability degrades over time (only 12.5% of 2013 URLs were still live). - Per-snapshot deduplication removes 60-80% of URLs, and ~20% of downloaded documents fail processing (mainly invalid zip files). ## Reproducibility ### Models No pre-trained model weights are released. The experiments use standard architectures (Faster R-CNN, YOLOv5, YOLOv8m, DETR) with publicly available implementations. Training details are provided for each experiment (learning rates, optimizers, batch sizes, epochs, early stopping). ### Algorithms The annotation algorithm is fully described and the code is open-sourced at [DS3Lab/WordScape](https://github.com/DS3Lab/WordScape) under Apache-2.0. The pipeline depends on `python-docx`, LibreOffice, OpenCV, PDFPlumber, and FastText. **Repo status (as of March 2026):** The GitHub repository has 39 stars and 6 forks. The last commit was 2023-11-20, roughly a month before publication. The entire commit history spans 8 commits. One open issue (Dec 2023) has no response. The repo is effectively a paper code dump: sufficient to document the approach but not actively maintained. A reimplementation from the paper's description using the same underlying libraries would likely be more practical than reviving this codebase. ### Data - **URL corpus:** 9.5M URLs to be released (not the documents themselves). Availability degrades over time as URLs go stale. - **No fixed dataset download:** Users must run the pipeline to produce their own dataset. This is a fundamental design choice (pipeline, not dataset), but it means results depend on which URLs are still live at download time. - Quality filters are provided: perplexity-based (Wikipedia KenLM models), annotation reliability score, character count thresholds, language confidence, and file size limits. ### Evaluation - FUNSD: F1@IoU 0.5. Standard split (149 train / 50 test). - ICDAR 2019 cTDaR: mAP@[0.50:0.95]. Modern tables subset (600 train / 240 test). 4 models per setting with error bars reported. - DocLayNet: mAP@[0.50:0.95]. Standard splits with varying fine-tuning sizes. - Handcrafted scientific dataset: 2,500 pages, 31 categories. Quartile-based confidence bands reported. ### Hardware The paper's appendix provides a detailed breakdown for processing one Common Crawl snapshot: - **Common Crawl parsing:** 64 CPU cores, 512GB RAM, 49 hours (3,087 CPU hours) - **Document download:** 64 CPU cores, 256GB RAM, 22.5 hours (1,440 CPU hours) - **Document processing:** 24 nodes with 24 CPU cores and 96GB RAM each, ~22 hours (12,672 CPU hours) - **Total per snapshot:** ~17K CPU hours (CPU-only; no GPUs required for the pipeline itself) Topic modeling uses the Google Cloud NLP API (paid external service). GPU types, total GPU-hours, and training costs for the downstream object detection experiments are not reported. ## Practical Assessment **The paper's lasting value is the idea, not the artifacts.** The Common Crawl URL corpus is ephemeral (URLs go stale), no fixed dataset is downloadable, and the reference implementation is unmaintained. However, the core technique is straightforward and well-documented: parse Open XML for semantic structure, colorize, render, detect bounding boxes. The underlying libraries (`python-docx`, LibreOffice, OpenCV) are all mature and actively maintained. **The strongest use case is private `.docx` collections.** The paper frames WordScape around Common Crawl, but the technique applies to any `.docx` source. Organizations that generate or receive Word documents as part of normal operations can use this approach to produce domain-specific layout training data at zero annotation cost. Each element identified via built-in Word styles or XML tags (headings, tables, lists, figures) becomes a free bounding-box label. The annotation reliability score lets you filter out heuristic-based labels when you need higher confidence. The result is pre-training data that matches your actual document distribution, not the scientific-paper distribution that dominates public datasets. ## BibTeX ``` ```bibtex @inproceedings{weber2023wordscape, title={WordScape: a Pipeline to extract multilingual, visually rich Documents with Layout Annotations from Web Crawl Data}, author={Weber, Maurice and Siebenschuh, Carlo and Butler, Rory M. and Alexandrov, Anton and Thanner, Valdemar R. and Tsolakis, Georgios and Jabbar, Haris and Foster, Ian and Li, Bo and Stevens, Rick and Zhang, Ce}, booktitle={Advances in Neural Information Processing Systems}, volume={36}, year={2023} } ``` ```