# The Newspaper Navigator Dataset > A dataset of 16M+ pages of visual content extracted from historic newspapers via Faster R-CNN, featuring a complete pipeline and search application. - Authors: Hunter Heidenreich - Published: 2020-05-04 - Updated: 2026-03-12 - Paper: Preprint (arXiv) - Paper: Published (ACM CIKM 2020) - Code: Codebase - Dataset: Full Dataset (GitHub Releases) - Dataset: Partial Dataset (Hugging Face) - Model: Model Weights - Source: Bevaya Labs, --- --- ## TL;DR Newspaper Navigator processes all 16.3 million pages from the [*Chronicling America*](https://chroniclingamerica.loc.gov/) archive using a fine-tuned Faster R-CNN (R50-FPN) to extract 7 classes of visual content. The resulting dataset combines a 3.5k-page human-annotated ground truth with roughly 102 million silver-label extractions (at a 0.9 confidence threshold; up to 198 million at 0.5), all placed in the public domain. The paper was recognized as **Best Resource Paper Runner-up at CIKM 2020**. ## What kind of paper is this? **Dominant Basis:** $\Psi\_{\text{Resource}}$ - The primary contribution is the dataset, codebase, fine-tuned model weights, and search application released for the Digital Humanities community. **Secondary Basis:** $\Psi\_{\text{Method}}$ - The pipeline applies a modern object detection architecture to historic document images across 16.3 million pages. This paper presents **Newspaper Navigator**, a massive dataset created by running a visual content extraction pipeline on the *Chronicling America* newspaper archive. It provides the codebase, fine-tuned models, and a search interface, making it a significant infrastructural contribution for Digital Humanities and Document Layout Analysis. ## What is the motivation? The *Chronicling America* collection contains over 16 million digitized newspaper pages with OCR text, but searching for specific *visual* content (like photos, maps, cartoons) was historically difficult because the metadata was primarily text-based and page-level. Unlocking this visual content requires automated methods to detect, classify, and extract visual regions from the scanned pages at scale, surpassing the limitations of keyword-only search. ## What is the novelty? 1. **Headline extraction as a visual task:** Prior work (e.g., Google Newspaper Search) identified headlines via font-size heuristics applied to OCR text. This paper instead detects headlines as image-level bounding boxes, treating them like any other visual element. This shift allows the detector to leverage the visual distinctiveness of headline typography rather than relying on imperfect OCR and metadata. 2. **Scale:** Processing 16.3 million pages to extract visual content is a massive undertaking in the Digital Humanities space. 3. **Dataset:** The resulting extracted dataset is the largest of its kind for historic newspapers. 4. **Pipeline:** A complete pipeline utilizing a fine-tuned Faster R-CNN (R50-FPN) trained on the *Beyond Words* dataset (augmented with headlines/ads) to detect 7 classes of visual content. 5. **Application:** A search interface utilizing image embeddings (ResNet-18, 512-dim; ResNet-50, 2,048-dim) for visual similarity search, enabling users to train "AI navigators" on the fly to find similar content. ## What experiments were performed? ### Model Architecture The core extraction utilizes a **Faster R-CNN** architecture with a ResNet-50 backbone and Feature Pyramid Network (FPN). The model enables region proposal and classification by optimizing the standard Faster R-CNN multi-task loss function (from [Ren et al. 2015](https://arxiv.org/abs/1506.01497)): $$ \begin{aligned} L(\\{p\_i\\}, \\{t\_i\\}) &= \frac{1}{N\_{cls}} \sum\_i L\_{cls}(p\_i, p\_i^\ast) \\\\ &+ \lambda \frac{1}{N\_{reg}} \sum\_i p\_i^\ast L\_{reg}(t\_i, t\_i^\ast) \end{aligned} $$ Where $L\_{cls}$ is the classification loss (log loss) and $L\_{reg}$ is the regression loss (smooth L1 loss) for the bounding boxes. ### Training Details - **Base Dataset:** *Beyond Words* (3.5k pages), augmented with semi-automated annotations for headlines and advertisements. - **Hyperparameters:** Learning rate of 0.00025, batch size of 8, 64 proposals per image. Augmentation was limited to `RESIZE_SHORTEST_EDGE` and `RANDOM_FLIP` (the only two augmentations supported by Detectron2 at the time). Training ran for **77 epochs** (approximately 17 hours) on a single NVIDIA T4 with early stopping. - **Inference threshold:** Detectron2 default confidence threshold of 0.05 is used during corpus processing. All predictions above this floor are retained in the dataset, allowing downstream users to apply their own cutoff. - **Search Embeddings:** Both **ResNet-18** (512-dim) and **ResNet-50** (2,048-dim) embeddings are generated for all extracted content using ImageNet-pretrained weights. Neither is fine-tuned on newspaper data. ResNet-50 was the primary choice for the search interface; ResNet-18 was retained for its lower dimensionality and faster querying. ## What are the outcomes/conclusions? The model achieves an overall **mAP (COCO) of 63.4%**, computed following the COCO standard: precision averaged over 101 recall values and 10 IoU thresholds from 0.50 to 0.95. Per-class results from Table 2 of the paper: | Class | AP | Count in Val. Set | | :--- | :--- | :--- | | Photograph | 61.6% | 879 | | Illustration | 30.9% | 206 | | Map | 69.5% | 34 | | Comic/Cartoon | 65.6% | 211 | | Editorial Cartoon | 63.0% | 54 | | Headline | 74.3% | 5,689 | | Advertisement | 78.7% | 2,858 | | **mAP (COCO)** | **63.4%** | N/A | | One Class | 75.1% | 9,931 | ### Key Findings **Best and worst performing classes:** Advertisement is the strongest class at 78.7% AP, reflecting the high visual consistency of display advertising. Illustration is the weakest at 30.9%, attributed to annotation heterogeneity in the training data (the boundary between an illustration and a photograph is not always clear, and the *Beyond Words* annotations reflect this ambiguity). **Classification ambiguity vs. detection failure:** The "One Class" AP of 75.1% collapses all 7 classes into a single foreground label. The roughly 12-point gap between One Class AP (75.1%) and 7-class mAP (63.4%) indicates that most errors come from inter-class confusion, not from the detector failing to find regions at all. **CIKM recognition:** The paper was awarded Best Resource Paper Runner-up at CIKM 2020. ### 19th Century Generalization The paper tests on two sets of 500 annotated pages from earlier periods not well-represented in training data. Performance degrades substantially: | Category | AP (1875-1900) | AP (1850-1875) | | :--- | :--- | :--- | | Headline | 51.6% | 21.2% | | Advertisement | 44.7% | 7.3% | | Illustration | 36.4% | N/A | | One Class | 48.1% | 12.1% | Approximately 10.4% of the full 16.3M-page corpus predates 1875. Users working with pre-1875 material should treat model predictions as lower-confidence silver data and consider targeted annotation or fine-tuning for that time period. ### Visual Similarity Search The search application demonstrates that generic ImageNet features (ResNet-18 and ResNet-50, not fine-tuned) are effective for retrieving visually similar historical content (e.g., finding maps of Virginia or cartoons depicting a specific political figure). ## Dataset Details ### 1. The Ground Truth (The "Good" Part) * **Size:** 3,559 full newspaper pages. * **Annotations:** Manual bounding boxes for 7 visual classes. * **Format:** COCO JSON (`beyond_words_data/trainval.json` in the repo). * **Scope warning: partial annotation by design.** The *Beyond Words* crowdsourcing task asked volunteers to find specific visual elements (photographs, illustrations, maps, comics, editorial cartoons) and ignore everything else. Headlines and advertisements were added later by the paper authors. This means **article text columns, bylines, datelines, pull quotes, column rules, and running heads are all unannotated background**. Training a general layout model on this data will teach the model to treat normal text layout as negative examples. It is well-suited for a visual-content detector, not a comprehensive page layout annotator. * **Quality Warning:** Even within the 7 annotated classes, there are significant **recall issues**. Many valid Headlines and Advertisements are unannotated (False Negatives), so training with "background" sampling may suppress valid detections of those classes too. ### 2. The Extracted Dataset (The "Big" Part) * **Size:** Visual content extracted from 16,368,041 pages (99.998% success rate). * **Format:** Metadata (JSON) containing predicted bounding boxes and links to full images (IIIF). * **Nature:** **Silver Data.** These are model predictions, not human annotations. * **Confidence threshold system:** All predictions with confidence ≥ 0.05 are retained, giving users the ability to apply their own threshold. At $\geq 0.9$, the corpus yields approximately 102 million total extractions across all classes (rising to roughly 198 million at $\geq 0.5$). The paper documents counts at three cuts (≥ 0.9, ≥ 0.7, ≥ 0.5); higher thresholds improve precision at the cost of recall. * **Warning:** The [Hugging Face mirror](https://huggingface.co/datasets/biglam/newspaper-navigator) is currently missing the `ads` and `headline` classes (the bulk of the data). Use the original [GitHub Releases](https://github.com/LibraryOfCongress/newspaper-navigator/releases) for the full set. ### 3. Missing Pieces This dataset was designed for **visual content retrieval**, not full page layout understanding. The following are entirely absent from the annotation scheme; not just missing, but actively treated as unlabeled background: * Article text / body copy * Column structure * Bylines, datelines, pull quotes * Tables * Reading order To build a complete newspaper layout model, you would need to combine the **3.5k Visual GT** with raw **OCR XML (ALTO/METS)** from [Chronicling America](https://chroniclingamerica.loc.gov/ocr/), which provides word-level bounding boxes for the text regions this dataset omits. ## Mapping to Unified Taxonomy How does Newspaper Navigator fit into our [Matter vs. Meaning framework](../matter-vs-meaning/)? | NewsNav Class | Visual Primitive (The What) | Logical Role (The Why) | Notes | | :--- | :--- | :--- | :--- | | **Headline** | `Text` | `Title` | Distinct from a "Header" (running page header). This is the article title. | | **Advertisement** | `Image` | `Advertisement` | **Flow Breaking.** Ads interrupt the main article text. Essential to classify separately to *exclude* them from RAG or processing. | | **Photograph** | `Image` | `Figure` | **Includes Caption.** Standard halftone photos. | | **Illustration** | `Image` | `Figure` | **Includes Caption.** Engravings/Drawings. | | **Map** | `Image` | `Diagram` | **Includes Caption.** Geospatial data. | | **Comic** | `Image` | `Figure` | **Includes Caption.** Often includes the strip's title/dialogue. | | **Editorial Cartoon** | `Image` | `Figure` | **Includes Caption.** Single-panel satire. | ## Reproducibility The project is open-source and public domain. While the original `news-navigator.labs.loc.gov` search application was retired in 2025, the code, model weights, and full dataset are preserved in the archived GitHub repository and its releases. ### Models - **Architecture:** Faster R-CNN with ResNet-50-FPN backbone, from Detectron2 Model Zoo. - **Pretraining:** COCO 2017 weights (standard Detectron2 initialization). - **Weights:** Released as `model_final.pth` via GitHub Releases (Unlicense). - **Search embeddings:** ResNet-18 (512-dim) and ResNet-50 (2,048-dim), ImageNet-pretrained, not fine-tuned. ### Algorithms - **Optimizer:** SGD. Base learning rate 0.00025. - **Batch size:** 8. Proposals per image: 64. - **Epochs:** 77 (early stopping). Training time: ~17 hours. - **Augmentation:** `RESIZE_SHORTEST_EDGE` and `RANDOM_FLIP` only (the only two augmentations supported by Detectron2 at the time). - **Image preprocessing:** Input images downsampled by 6x before inference to reduce I/O and memory; Detectron2 handles any further resizing internally. - **Inference confidence threshold:** 0.05 (Detectron2 default). All predictions above this floor are retained in the released dataset; users apply their own cutoff. ### Data - **Training/validation set:** Augmented Beyond Words annotations, 48,409 total bounding boxes across 3,559 pages and 7 classes, split 80/20 train/val (see Table 1 in the paper for per-class breakdown). Headlines and advertisements were added by the paper authors and are not crowdsource-verified; map annotations for 122 pages were added via keyword search. - **Extracted corpus:** 16,368,041 pages (99.998% of Chronicling America) processed and released as silver-label predictions with IIIF image links. - **License:** Unlicense (public domain) for both code and dataset. - **Note:** The Hugging Face mirror is missing the `ads` and `headline` classes. Use GitHub Releases for the complete schema. ### Evaluation - **Metric:** mAP (COCO standard: precision averaged over 101 recall values, IoU thresholds 0.50-0.95). - **Validation set:** World War I-era newspaper pages (held out from the Beyond Words annotations). - **Temporal generalization test:** Two additional sets of 500 annotated pages each from 1875-1900 and 1850-1875, to assess degradation on pre-training-distribution material. ### Hardware - **Training:** Single NVIDIA T4 GPU (AWS `g4dn.2xlarge`), ~17 hours. - **Corpus inference:** Two `g4dn.12xlarge` instances, each with 48 Intel Cascade Lake vCPUs and 4 NVIDIA T4 GPUs. Full 16.3M-page corpus processed in **19 days** (~0.1 seconds/page). - **Dependencies:** Detectron2 (2020-era). Requires older PyTorch/Python versions for exact reproduction. ## BibTeX ```bibtex @inproceedings{lee2020newspaper, title={The Newspaper Navigator Dataset: Extracting And Analyzing Visual Content from 16 Million Historic Newspaper Pages in Chronicling America}, author={Lee, Benjamin Charles Germain and Mears, Jaime and Jakeway, Eileen and Ferriter, Meghan and Adams, Chris and Yarasavage, Nathan and Thomas, Deborah and Zwaard, Kate and Weld, Daniel S.}, booktitle={Proceedings of the 29th ACM International Conference on Information and Knowledge Management}, year={2020}, pages={3055--3062}, publisher={ACM}, doi={10.1145/3340531.3412767} } ```