# M5HisDoc: A Multi-Style Chinese Historical Document Analysis Benchmark > Analysis of M5HisDoc (NeurIPS 2023 D&B), a 8,000-image Chinese historical document benchmark covering five tasks including reading order prediction, with Regular and Hard subsets testing robustness to rotation, distortion, and resolution reduction. - Authors: Bevaya AI Team - Published: 2026-04-02 - Dataset: M5HisDoc Dataset - Source: Bevaya Labs, --- ## TL;DR M5HisDoc is a 8,000-image Chinese historical document benchmark with five benchmark tasks: text line detection, text line recognition, character detection, character recognition, and reading order prediction. It is the first multi-style benchmark for this domain, spanning 20+ layout types, 20+ document types, and 16,151 character categories. Two subsets are provided: M5HisDoc-R (regular, manual annotation) and M5HisDoc-H (hard, with added rotation, distortion, and resolution reduction). Existing reading order methods designed for modern documents generalize poorly to historical Chinese documents. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Resource}}$: The headline contribution is the M5HisDoc benchmark dataset, covering a previously underserved domain (Chinese historical documents) with multi-style diversity and five distinct task annotations. Most of the paper describes data collection, annotation methodology, and benchmark statistics. **Secondary:** $\Psi\_{\text{Evaluation}}$: Comprehensive baselines are established across all five tasks, exposing systematic weaknesses of modern-document methods when applied to historical Chinese documents, including for reading order prediction. ## What is the motivation? Existing Chinese historical document benchmarks share three limitations: 1. **Single document type.** MTHv2 covers only Buddhist scriptures; ICDAR 2019 HDRC covers only genealogies. Real historical collections span medical texts, anthologies, dictionaries, government records, and more. 2. **Limited layout variety.** Prior datasets cover fewer than 10 layout types; M5HisDoc covers more than 20. 3. **No dedicated reading order study.** Prior reading order work (LayoutReader, Augmented XY Cut) targets modern documents with Western left-to-right reading flows. Chinese historical documents read right-to-left, top-to-bottom in vertical columns, with complex layouts that are poorly handled by these methods. ## What is the novelty? ### Dataset Construction **Data sources:** - 300 images from MTHv2 training set and 700 from SCUT-CAB, selected for dense text or complex layout. - 2,799 images from 131 ancient books sourced from electronic archives (Harvard Library, National Archives of Japan, etc.). - 201 images captured by scanning four physical Chinese ancient books under varied scanning angles and lighting. Total: 4,000 manually annotated images (M5HisDoc-R). **Annotation process:** Character detector and recognizer pre-trained on MTHv2 and CASIA-AHCDB provided preliminary annotations. Three annotation steps followed: bounding box correction, two-round character recognition proofreading (two annotators + one domain expert), text line grouping, and reading order sorting. Three-round quality check. Total annotation effort: ~4,000 person-hours. **Hard subset generation (M5HisDoc-H):** - 834 images with white backgrounds randomly rotated $\pm 5^\circ$ to $\pm 15^\circ$. - 50% of images randomly distorted via DewarpNet to simulate real-world document warping. - 2,041 images with large character boxes scaled down (minimum edge 18 pixels). Both subsets split 2:1:1 (train:val:test) for fair evaluation. **Dataset statistics:** - 8,000 images, 50.48 text lines per image, 545.92 characters per image. - 16,151 character categories (at least 1.6$\times$ more than prior datasets). - Zero-shot recognition scenario: 1,853 (val) and 1,831 (test) character categories unseen in training. - Aspect ratios from under 0.4 to over 1.8; text line lengths from under 6 to over 58 characters. ## What experiments were performed? ### Task 1: Text Line Detection Three method families: regression-based (Mask R-CNN, Cascade R-CNN, OBD), segmentation-based (PSENet, PAN, FCENet, DBNet++), connected component-based (TextSnake). Metrics: precision, recall, F1 at IoU thresholds 0.5, 0.6, 0.7; end-to-end 1-NED. Key findings: segmentation-based methods degrade sharply at higher IoU thresholds due to long texts and dense packing. Rotation and distortion in M5HisDoc-H substantially reduces recall for regression-based methods (RPN proposals remain horizontal). TextSnake achieves the best F1 at IoU 0.5 (90.85/90.94 on H/R). ### Task 2: Text Line Recognition Methods: CRNN, Ma et al., ZCTRN (CTC-based); ASTER, NRTR, Robust Scanner (attention-based); Peng et al. (segmentation-based). Metrics: correct rate (CR) and accurate rate (AR). Key findings: attention-based methods underperform on this task due to the large character category count (16,151) and long text lines causing attention drift. CTC-based methods are more robust. Best overall: Ma et al. at 91.29/92.29 CR (H/R). ### Task 3: Character Detection Faster R-CNN, YOLOv3, YOLOX. Metrics: precision, recall, F1 at IoU 0.5/0.6/0.7; end-to-end top-1 accuracy with a SwinTransformer recognizer. Key findings: character-level methods are substantially more robust to M5HisDoc-H's rotation and distortion than text-line-level methods, because individual character scale variations are smaller. All models achieve F1 above 96% at IoU 0.5. ### Task 4: Character Recognition ResNet50, RegNet, ConvNeXt (CNN-based); ViT, SwinTransformer (ViT-based). Metrics: top-1, top-5, macro accuracy. Key findings: top-1 accuracy is high (94-95%), but macro accuracy is substantially lower (65-75%), reflecting long-tail distribution. Similar-morphology characters are the primary error source. ### Task 5: Reading Order Prediction Three methods evaluated on M5HisDoc using ARD (Average Relative Distance; lower is better): | Type | Method | ARD (H/R) | |:---|:---|:---| | Rule-based | Heuristic (right-to-left, top-to-bottom) | 5.27 / 5.20 | | Rule-based | Augmented XY Cut | 14.63 / 12.60 | | Learning-based | LayoutReader (layout only) | 8.68 / 5.49 | The authors' simple heuristic (center coordinates sorted right-to-left, top-to-bottom, reflecting the dominant Chinese historical reading direction) outperforms both Augmented XY Cut and LayoutReader on M5HisDoc-R, and achieves the best result on M5HisDoc-H. LayoutReader, trained on modern born-digital documents, performs comparably to the heuristic on M5HisDoc-R but degrades on M5HisDoc-H due to distorted bounding boxes (horizontal minimum enclosing rectangles of distorted text overlap significantly, breaking the spatial assumptions the model learned). ### Cross-Validation Models trained on MTHv2 and tested on M5HisDoc show substantial degradation, confirming the style gap between homogeneous (single-type) historical benchmarks and M5HisDoc's multi-style distribution. ## What are the outcomes/conclusions? 1. **Modern reading order methods do not transfer to Chinese historical documents.** The simple domain-aware heuristic (right-to-left, top-to-bottom) outperforms LayoutReader and Augmented XY Cut, both of which were designed for modern left-to-right documents. The benchmark makes this gap quantifiable for the first time in a controlled setting. 2. **M5HisDoc-H is substantially harder than M5HisDoc-R for reading order.** Distortion and rotation cause horizontal bounding boxes to overlap, breaking spatial assumptions. ARD degrades from 5.49 to 8.68 for LayoutReader, from 12.60 to 14.63 for XY Cut. 3. **Character-level analysis is more robust to image degradation than text-line-level.** For detection and recognition tasks, character-level methods degrade less on M5HisDoc-H than text-line-level methods. 4. **Zero-shot character recognition remains an open problem.** Macro accuracy (65-75%) lags far behind top-1 accuracy (94-95%), reflecting the long-tail and similar-morphology challenges. ### Limitations - The dataset requires application and agreement before access (CC-BY-NC-ND-4.0); redistribution and derivatives are restricted. Commercial use is not permitted. - The reading order baselines are limited: only two rule-based methods and LayoutReader. No method specifically designed for vertical right-to-left reading is evaluated beyond the authors' heuristic. - Horizontal minimum enclosing rectangles are used as bounding box inputs to reading order methods; polygon-aware evaluation is not reported. - No cross-lingual or non-Chinese historical document evaluation. - Annotation of reading order uses manual sorting by annotators; no inter-annotator agreement metric is reported for reading order specifically. ## Reproducibility ### Models No new model weights released. All baselines use publicly available architectures and standard training protocols. ### Algorithms - Text line detection: AdamW, lr $10^{-4}$, 160 epochs, batch size 8; LR decays $\times 0.1$ at epochs 80 and 128; input long side 1,333. - Text line recognition: AdamW, lr $4 \times 10^{-4}$ with CosineAnnealing to $4 \times 10^{-6}$, 50 epochs, batch size 32; TPS preprocessing; height normalized to 96. - Character recognition: AdamW, lr $10^{-3}$ with CosineAnnealing to $10^{-6}$, 90 epochs, batch size 1,024; RandAugment; 96$\times$96 input. - Reading order: heuristic described in Appendix; LayoutReader evaluated layout-only using horizontal bounding boxes as input. ### Data - 8,000 images (4,000 regular + 4,000 hard) with character-level and text-line-level bounding boxes, text content, and reading order. - Released at [github.com/HCIILAB/M5HisDoc](https://github.com/HCIILAB/M5HisDoc) under CC-BY-NC-ND-4.0 with application required. - 2:1:1 train/val/test split; zero-shot character categories in val/test. ### Evaluation - Text detection: precision, recall, F1 at IoU 0.5/0.6/0.7. - Text recognition: CR, AR. - Character recognition: top-1, top-5, macro accuracy. - Reading order: ARD (Average Relative Distance). - No error bars or significance tests reported. ### Hardware Not reported. ## BibTeX ```bibtex @inproceedings{shi2023m5hisdoc, title={M$^5$HisDoc: A Large-scale Multi-style Chinese Historical Document Analysis Benchmark}, author={Shi, Yongxin and Liu, Chongyu and Peng, Dezhi and Jian, Cheng and Huang, Jiarong and Jin, Lianwen}, booktitle={37th Conference on Neural Information Processing Systems, Datasets and Benchmarks Track}, year={2023} } ```