# HJDataset: Historical Japanese Documents with Complex Layouts > A large-scale layout analysis dataset of 2,271 historical Japanese document pages with 260K annotations across 7 hierarchical element types, including reading order. - Authors: Hunter Heidenreich - Published: 2020-04-17 - Updated: 2026-03-12 - Paper: Preprint (arXiv) - Paper: Published (CVPR 2020 Workshop) - Code: Codebase + Pretrained Models - Dataset: HJDataset Annotations - Source: Bevaya Labs, --- ## TL;DR HJDataset provides 259,616 layout element annotations across 7 hierarchical categories for 2,271 pages of a 1953 Japanese biographical directory. In addition to bounding boxes and masks, the dataset includes parent-child dependency structures and reading orders. A semi-rule-based pipeline generates the annotations, with statistical quality control and human correction achieving an estimated 99.6% accuracy. ## What kind of paper is this? **Dominant Basis:** $\Psi\_{\text{Resource}}$. The headline contribution is a large-scale historical document layout dataset with hierarchical annotations and reading order, filling a gap for Asian-language historical documents. **Secondary Basis:** $\Psi\_{\text{Method}}$. The semi-rule-based annotation pipeline (contour detection, CCL/RLSA segmentation, CNN-based region classification, and statistical error identification) is a meaningful methodological contribution for scalable dataset construction. ## What is the motivation? Deep learning methods for document layout analysis require large annotated datasets, but historical document datasets were small (DIVA-HisDB: 150 images; ENP: 528 images) and almost exclusively Western-language. Asian documents present unique challenges: vertical text orientation, right-to-left reading order within rows, and complex hierarchical structures that off-the-shelf tools cannot handle. The Japanese National Diet Library has millions of digitized scans, but no large-scale layout dataset existed to train models on them. ## What is the novelty? HJDataset is (to the authors' knowledge) the first large-scale layout analysis dataset for historical Japanese documents. Its key contributions: 1. **Hierarchical structure annotations.** Each page is decomposed into a tree: page frame $\rightarrow$ rows $\rightarrow$ title regions / text regions $\rightarrow$ title / subtitle / other. Parent-child relationships are encoded via `parent_id` fields. 2. **Reading order annotations.** Sequential reading order is encoded via `next_id` fields, with $-1$ marking the page end. The dataset captures irregular reading orders caused by section headers disrupting the standard right-to-left flow. 3. **Semi-rule-based annotation pipeline.** Rather than relying on expensive manual annotation, the pipeline combines: - Contour detection for page frames - Connected Component Labeling (CCL) and Run Length Smoothing Algorithm (RLSA) for row and region segmentation - A NASNet Mobile CNN classifier (trained on 1,200 hand-labeled samples, 99% test accuracy) for region type refinement - Statistical outlier detection (percentile filtering on element counts and gap widths) to flag errors for human correction 4. **Quality control.** The statistical approach identified 80% of the 616 corrected errors. After correction, the dataset achieves an estimated 99.6% annotation accuracy. ## What experiments were performed? **Baseline object detection.** Three models were trained on main-page layout elements using Detectron2 with R-50-FPN-3x backbone (pretrained on COCO), on a single NVIDIA RTX 2080Ti: | Model | Page Frame | Row | Title Region | Text Region | Title | Subtitle | Other | mAP | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | Faster R-CNN | 99.0 | 98.8 | 87.6 | 94.5 | 65.9 | 84.1 | 44.0 | 82.0 | | Mask R-CNN | 99.1 | 98.5 | 89.5 | 86.8 | 71.5 | 84.2 | 39.8 | 81.3 | | RetinaNet | 99.0 | 95.1 | 69.6 | 89.5 | 72.6 | 85.9 | 14.4 | 75.2 | All values are mAP @ IoU [0.50:0.95] on the test set. Large-scale elements (Page Frame, Row) are near-perfect; small elements (Title, Other) are harder. **Transfer learning.** Models pretrained on HJDataset main pages were fine-tuned on two targets: index pages and a different 1939 biographical directory. Results are mixed: - **Index pages (all 57 training samples):** HJDataset initialization raised mAP from 34.4 (COCO init) to 47.1, a clear gain. - **Index pages (5-shot):** The improvement was marginal: 10.0 (COCO) to 10.3 (HJDataset), within noise. - **Cross-publication (4 training samples from 12 annotated pages):** HJDataset initialization raised mAP from 69.9 to 81.6, the strongest transfer result. The cross-publication experiment is encouraging but limited: only 12 pages from a single additional source were annotated, making it hard to draw broad generalization conclusions. ## What are the outcomes/conclusions? - HJDataset fills a gap in historical Asian-language document layout data: large scale, hierarchical annotations, and reading order in a single package. - The semi-automatic pipeline achieves an estimated 99.6% annotation accuracy at far lower cost than manual annotation. That estimate is based on a small sampling procedure (20 pages, repeated 3 times), so the true error rate may differ. - Baseline detection models trained on HJDataset achieve mAP above 80% for Faster R-CNN and Mask R-CNN at IoU [0.50:0.95]. - Pretraining on HJDataset provides clear transfer benefits when fine-tuning with sufficient data, though the few-shot gains on index pages are marginal. - The "Other" category (chapter headers and miscellaneous text) has only 148 total annotations, making detection performance on it unreliable (mAP as low as 14.4% for RetinaNet). - **Reading order and hierarchy are not evaluated.** Despite being headline features, neither parent-child structure nor sequential reading order is quantitatively assessed. The experiments focus exclusively on bounding-box detection. - The dataset is limited to a single publication type (1953 biographical directory), with highly uniform visual style and layout. Generalization evidence rests on only 12 manually annotated pages from one other source. ## Mapping to Matter vs. Meaning Framework How does HJDataset's 7-class hierarchical taxonomy map to our [Matter vs. Meaning framework](../matter-vs-meaning/)? | HJDataset Class | Visual Primitive | Logical Role | Notes | | :--- | :--- | :--- | :--- | | **Page Frame** | `Structure` | *(none)* | The outer boundary of the printed area. A structural container, not a content element. | | **Row** | `Text` | `Body` | A horizontal row of entries in the biographical directory. Serves as a spatial grouping container for the elements within it. | | **Title Region** | `Text` | `Title` | A region containing a person's name entry (the "title" of their biographical entry). | | **Text Region** | `Text` | `Body` | A region containing the biographical details of an entry. | | **Title** | `Text` | `Title` | The actual name text within a Title Region. Finer granularity than Title Region. | | **Subtitle** | `Text` | `Subtitle` | Secondary identifying information (birth year, occupation) within a biographical entry. | | **Other** | `Text` | `SectionHeader` | Chapter headers and miscellaneous text that does not fit the biographical entry structure. Only 148 total annotations; underrepresented. | **Structural notes:** HJDataset is unique in providing explicit **parent-child hierarchy** (via `parent_id`) and **reading order** (via `next_id`). The hierarchy maps to: Page Frame $\rightarrow$ Row $\rightarrow$ Title Region / Text Region $\rightarrow$ Title / Subtitle / Other. Reading order is right-to-left within rows (vertical Japanese text), encoded as a linked list. **Coverage gaps:** No explicit classes for `Table`, `Image`, `Formula`, `PageHeader`, `PageFooter`, or any form primitives. The dataset is limited to a single domain (biographical directories) with very uniform layout. ## Reproducibility ### Models - **Baselines:** Faster R-CNN, Mask R-CNN, and RetinaNet with R-50-FPN-3x backbone, initialized from COCO pretrained weights. - **Region classifier:** NASNet Mobile, trained from scratch on 1,200 hand-labeled + 250 synthetic mis-segmentation samples. Input size: 200 $\times$ 522. - Pretrained model configs and weights are released via the GitHub repository (Detectron2 format). ### Algorithms - **Training:** SGD optimizer, base learning rate 0.00025, decay 0.1 every 30k iterations, 60k total iterations, batch size 2. - **Region classifier training:** SGD, converges in 40 epochs. - **Annotation pipeline:** Binarization $\rightarrow$ contour detection (page frames) $\rightarrow$ CCL + horizontal RLSA (rows) $\rightarrow$ CCL + vertical RLSA (text/title regions) $\rightarrow$ NASNet Mobile classification $\rightarrow$ reading order generation $\rightarrow$ statistical error detection $\rightarrow$ human correction. ### Data - **Source:** Japanese Who's Who biographical directory (*Jinji Koshinroku*, Vol. 17, 1953). Biographies of ~50,000 prominent Japanese citizens. - **Scale:** 2,271 page images; 259,616 layout annotations across 7 categories. - **Page types:** Main (2,048), Advertisement (87), Index (82), Other (54). - **Layout element classes (7):** Page Frame, Row, Title Region, Text Region, Title, Subtitle, Other. - **Splits:** 70% train / 15% validation / 15% test, stratified by page type. | Category | Train | Val | Test | Total | | :--- | ---: | ---: | ---: | ---: | | Page Frame | 1,490 | 320 | 320 | 2,130 | | Row | 7,742 | 1,657 | 1,660 | 11,059 | | Title Region | 33,637 | 7,184 | 7,271 | 48,092 | | Text Region | 38,034 | 8,129 | 8,207 | 54,370 | | Title | 66,515 | 14,931 | 14,366 | 95,812 | | Subtitle | 33,576 | 7,173 | 7,256 | 48,005 | | Other | 103 | 16 | 29 | 148 | | **Total** | **181,097** | **39,410** | **39,109** | **259,616** | - **Format:** COCO-style JSON (bounding boxes in XYWH format, plus segmentation masks, `parent_id`, `next_id`, and image-level `category_id` for page type). - **Access:** Annotations available via Dropbox (Apache-2.0). Images require a download request form due to copyright restrictions on the source publication. - **Annotation license:** Apache-2.0. Image license is restricted (copyright held by original publisher). ### Evaluation - **Metric:** mAP @ IoU [0.50:0.95] (COCO-style). - **Baselines:** Faster R-CNN, Mask R-CNN, RetinaNet; all evaluated on the same test split. - Dataset is unbalanced: Title has ~96K annotations while Other has only 148, making rare-class evaluation less reliable. - **Not evaluated:** Reading order accuracy and hierarchical structure correctness. No metrics or baselines are provided for these annotation types despite their prominence in the dataset description. - **Accuracy estimate:** The 99.6% figure is extrapolated from manually checking 20 random pages (repeated 3 times, averaging 0.6% error). No inter-annotator agreement is reported. ### Hardware - **Training:** Single NVIDIA RTX 2080Ti GPU. - **Inference latency:** Not reported. ## BibTeX ``` ```bibtex @inproceedings{shen2020hjdataset, title={A Large Dataset of Historical Japanese Documents with Complex Layouts}, author={Shen, Zejiang and Zhang, Kaixuan and Dell, Melissa}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops}, year={2020}, pages={548--549}, doi={10.1109/CVPRW50498.2020.00282} } ``` ```