# HJDataset: Historical Japanese Documents with Complex Layouts and Reading Order > A large semi-automatically annotated layout dataset of historical Japanese documents providing bounding boxes, hierarchical structure, and reading order for over 250k layout elements across 2,271 page scans. - Authors: Hunter Heidenreich - Published: 2020-04-18 - Updated: 2026-04-02 - Paper: Preprint (arXiv) - Paper: Published (CVPRW 2020) - Dataset: HJDataset (annotations) - Dataset: HJDataset (images) - Source: Bevaya Labs, --- ## TL;DR HJDataset is a semi-automatically constructed layout analysis dataset for historical Japanese documents, containing over 259k annotations of seven element types across 2,271 page scans. Beyond bounding boxes and segmentation masks, it provides two annotations rarely found together in layout datasets: hierarchical dependency structure and reading order for all layout elements. The reading order reflects the real right-to-left, top-to-bottom conventions of Japanese text, including irregular cases where section headers disrupt the standard flow. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Resource}}$: The central contribution is the HJDataset itself, a large annotated corpus accompanied by a semi-rule-based construction pipeline. The paper's organization, tables, and figures all center on the dataset's contents, construction methodology, and statistics. **Secondary:** $\Psi\_{\text{Method}}$: The authors develop a multi-stage pipeline (rule-based detector, CNN classifier, reading order generator, statistical quality control) to construct the dataset with limited manual effort. This pipeline is a reusable contribution in its own right. **Secondary:** $\Psi\_{\text{Evaluation}}$: The paper provides baseline detection results on the dataset using Faster R-CNN, Mask R-CNN, and RetinaNet, and evaluates transfer learning from the main-page subset to index pages and to a different historical Japanese publication. ## What is the motivation? Deep learning approaches for document layout analysis require large labeled datasets for training. Two gaps motivated this work. First, available historical document datasets are small: DIVA-HisDB has 150 instances and the European Newspapers Project has 528. Models trained on these tend to overfit and produce unreliable benchmarks. Second, nearly all historical document datasets cover Western languages, leaving the distinct layout conventions of Asian languages (vertical text, right-to-left column order, dense biographical formats) without dedicated training resources. Japan's National Diet Library has digitized millions of historical scans, providing raw material for a large-scale dataset if an annotation method can be devised that avoids prohibitive manual labeling costs. The Japanese Who's Who biographical directory from 1953 is chosen as the source because it contains roughly 2,000 page scans with consistent structural conventions, making semi-automatic extraction feasible. ## What is the novelty? ### Dataset scope and annotation richness HJDataset contains 2,271 document image scans partitioned into four page categories: main (2,048), advertisement (87), index (82), and other (54). For the main and index pages, 259,616 layout element annotations are provided across seven categories: Page Frame, Row, Title Region, Text Region, Title, Subtitle, and Other. Annotations include both rectangular bounding boxes and quadrilateral segmentation masks (for page frames, which are subject to scan rotation). Each annotation also carries its position in the hierarchical layout tree and its reading order index. The hierarchical structure reflects the physical organization of main pages: five rows are vertically stacked in each page, and within each row, text regions and title regions are arranged horizontally. Title regions are further subdivided into title and subtitle blocks. This nesting gives three levels of granularity, from the full page frame down to individual text blocks. ### Semi-rule-based construction pipeline The construction proceeds in four stages. The *Text Block Detector* extracts bounding boxes hierarchically: page frame by contour detection, rows by horizontal Run Length Smoothing Algorithm (RLSA) followed by Connected Component Labeling (CCL), then text and title regions by applying RLSA vertically within each row. Page frames are characterized as quadrilaterals to handle scan distortion; row and block regions use axis-aligned rectangles after warp affine correction. A CNN classifier (*Text Region Classifier*) is then applied to each cropped region to distinguish text regions, title regions, and mis-segmented regions. The architecture is NASNet Mobile, trained from scratch on 1,200 hand-labeled samples (augmented with 250 synthetic mis-segmentations to address class imbalance) and achieving 0.99 test accuracy on 100 samples. ### Reading order generation Reading order is determined by exploiting Japanese reading conventions: texts inside each region are written in vertical columns read right-to-left across columns; within a row, blocks are also ordered right-to-left. For the majority of pages this yields a deterministic topological ordering. The more interesting case involves irregular reading orders in index pages. Section headers can span multiple rows, breaking the expected right-to-left sequence. The pipeline detects these discontinuities by searching for unusually large horizontal gaps between blocks in a row and assigns a corrected reading order accordingly. These irregular cases are preserved in the dataset rather than smoothed away, making HJDataset useful for evaluating reading order methods that must handle non-trivial flows. Reading order annotations take the form of ordered position indices associated with each annotation entry in the COCO-format JSON files. Combined with the hierarchical parent-child links between layout elements, this gives each annotation both its rank in the reading sequence and its structural role. ### Quality control After automated extraction, statistical screening identifies candidate errors without requiring full manual review. Pages with abnormally many or few layout elements (outside the 5th to 95th percentile range) are flagged for page frame correction (182 pages examined, 18 errors corrected). Blocks with unusually wide gaps (above the 99th percentile) are flagged for missed final text lines (1,011 examined, 487 corrected). Remaining structural noise is handled by human annotators during the inspection pass. In total, over 616 errors are corrected, and the authors estimate the final dataset achieves 99.6% annotation accuracy. ## What experiments were performed? ### Layout element detection benchmark Faster R-CNN, Mask R-CNN, and RetinaNet are trained on main-page training data (181,097 annotations) for 60k iterations using a ResNet-50 FPN backbone pre-trained on COCO. Evaluation uses COCO-style mean Average Precision (mAP) at IOU $[0.50{:}0.95]$. Results on the test set show high accuracy for coarser elements and lower accuracy for fine-grained ones: | Category | Faster R-CNN | Mask R-CNN | RetinaNet | | :--- | :--- | :--- | :--- | | Page Frame | 99.05 | 99.10 | 99.04 | | Row | 98.83 | 98.48 | 95.07 | | Title Region | 87.57 | 89.48 | 69.59 | | Text Region | 94.46 | 86.80 | 89.53 | | Title | 65.91 | 71.52 | 72.57 | | Subtitle | 84.09 | 84.17 | 85.87 | | Other | 44.02 | 39.85 | 14.37 | | **mAP** | **81.99** | **81.34** | **75.22** | Faster R-CNN and Mask R-CNN reach comparable overall mAP (82.0 vs 81.3), both ahead of RetinaNet (75.2). Title detection is the weakest category, likely due to its small physical size and visual similarity to subtitle blocks. The *Other* category (chapter headers and miscellaneous text) has the lowest mAP due to its small sample count (148 total). ### Transfer to index pages Five Faster R-CNN variants are compared on index pages, varying the initialization (COCO or HJDataset weights) and the amount of training data (all 57 training index samples, 5-shot, or zero-shot). Pre-training on HJDataset consistently outperforms COCO initialization: with all index training data, HJDataset initialization reaches 47.1 mAP versus 34.4 for COCO; with 5 samples, the gap is smaller (10.3 vs 10.0) but still favors HJDataset. Zero-shot transfer from main to index reaches only 9.4 mAP, consistent with the structural dissimilarity between page types. ### Transfer to a different publication Twelve pages from a 1939 edition of the same Who's Who directory with a different layout schema are manually annotated. Four pages are used for training, eight for testing. HJDataset-initialized Faster R-CNN reaches 81.6 mAP at 5-shot, versus 69.9 for COCO initialization, a gain of approximately 12 points. ## What are the outcomes/conclusions? HJDataset fills a concrete gap: a large-scale layout analysis dataset for historical Asian language documents. The dataset's combination of bounding boxes, masks, hierarchical structure, and reading order annotations makes it more information-rich than most contemporaneous layout datasets, which typically provide only bounding boxes or masks. The reading order annotations specifically capture right-to-left, top-to-bottom Japanese conventions along with documented irregular cases (section headers disrupting column flow). This makes HJDataset an early example of a dataset that treats reading order as a first-class annotation rather than an afterthought, though the reading order is generated from known rules rather than human annotation. The transfer learning results suggest that pre-training on HJDataset generalizes to related historical Japanese documents, which is useful for real-world digitization workflows where labeled data is scarce. ### Limitations - Reading order annotations are rule-derived rather than human-annotated. For the majority of main pages the rules are deterministic and accurate, but the dataset does not quantify inter-annotator agreement for the reading order specifically. - The source material is a single publication type (Japanese Who's Who, 1953 edition). The layouts are structurally homogeneous (five rows per main page, vertical text only), which limits generalization to documents with less regular or more diverse layout patterns. - No dedicated reading order model is trained or evaluated in the paper. The reading order annotations are included in the dataset but their utility is demonstrated only implicitly through the overall layout task. - The dataset covers one language (Japanese) and one document era. Western-language users and researchers working on other Asian historical materials will not benefit directly from the pre-trained models. - Image access requires submitting a request form due to copyright considerations, which limits immediate reproducibility. ## Reproducibility ### Models - Faster R-CNN, Mask R-CNN, and RetinaNet with R-50-FPN-3x backbone, implemented via Detectron2. Backbone weights initialized from the public COCO pre-trained checkpoint. - The text region classifier uses NASNet Mobile, trained from scratch (no pre-trained weights loaded). - The authors state that training configurations will be open-sourced. Detectron2 is Apache-2.0. ### Algorithms - Text block detection: contour detection for page frames, CCL and RLSA for row and block segmentation. - Text classification: NASNet Mobile trained with SGD, converging in 40 epochs; input images resized to 200 (height) $\times$ 522 (width) pixels. - Reading order: rule-based; right-to-left ordering within rows, with gap-based discontinuity detection for irregular cases. - Detection models: 60k iterations, base learning rate 0.00025, decay factor 0.1 at 30k and 60k iterations, batch size 2, trained on a single NVIDIA RTX 2080Ti. ### Data - Source: 2,271 scans from the 1953 edition of the Japanese Who's Who (Jinji Kshinroku, vol. 17). - 259,616 layout element annotations across seven categories (Page Frame, Row, Title Region, Text Region, Title, Subtitle, Other) for main and index pages. - 70/15/15 train/validation/test split, stratified by page type. - Annotations in COCO JSON format with an additional `category_id` field per image for page-type classification. - Dataset annotations released under Apache-2.0; images subject to copyright and require a download request. - Available at: [https://dell-research-harvard.github.io/HJDataset/](https://dell-research-harvard.github.io/HJDataset/) ### Evaluation - COCO-style mAP at IOU $[0.50{:}0.95]$ as the primary detection metric; AP$\_{50}$ and AP$\_{75}$ also reported for transfer experiments. - No dedicated reading order evaluation metric or evaluation protocol is defined in the paper; the 99.6% annotation accuracy estimate covers layout element detection errors overall, not reading order correctness specifically. Reading order quality has no separate quantitative measure in the paper. - No error bars or significance tests reported. ### Hardware - Single NVIDIA RTX 2080Ti GPU for all detection model training. - No training time or memory requirements reported. - NASNet Mobile inference is lightweight; no GPU requirements stated for the classifier. ## BibTeX ```bibtex @inproceedings{shen2020hjdataset, title={A Large Dataset of Historical Japanese Documents with Complex Layouts}, author={Shen, Zejiang and Zhang, Kaixuan and Dell, Melissa}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops}, year={2020} } ```