# IIIT-AR-13K: Graphical Object Detection in Annual Reports > A manually annotated dataset of 13k annual report pages for detecting tables, figures, logos, and signatures. - Authors: Bevaya AI Team - Published: 2020-07-26 - Updated: 2026-03-12 - Paper: Preprint (arXiv) - Paper: Published (DAS 2020) - Dataset: IIIT-AR-13K - Code: Code - Source: Bevaya Labs, --- ## TL;DR IIIT-AR-13K is a manually annotated dataset of **13,415 pages** from annual reports of **29 companies**, covering five graphical object categories: Table, Figure, Natural Image, Logo, and Signature. At the time of release, it was the largest manually annotated dataset for graphical object detection, explicitly separating data-driven graphics (charts) from natural photographs and adding document-specific elements (logos, signatures) absent from scientific-paper datasets. --- ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Resource}}$ The primary contribution is the dataset itself, addressing the absence of high-quality, human-annotated training data for graphical object detection in business documents. **Secondary:** $\Psi\_{\text{Evaluation}}$ The authors benchmark two standard object detection architectures on the new dataset and run a cross-dataset transfer study to demonstrate that IIIT-AR-13K is more effective as training data than much larger, automatically annotated datasets for most target benchmarks. ## What is the motivation? Prior to this work, document layout analysis datasets were dominated by scientific articles (arXiv, PubMed). Those documents follow rigid column templates and lack graphical elements common in business contexts. The authors identify three gaps: 1. **Domain restriction:** Existing datasets (TableBank, PubLayNet, DeepFigures) contain predominantly scientific papers. Business documents such as annual reports have highly heterogeneous, often artistic layouts that differ substantially from academic typesetting. 2. **Missing classes:** No existing dataset included Logo or Signature classes. These are critical for workflows such as branding authentication and legal document verification. 3. **Figure conflation:** Datasets like PubLayNet use a single `Figure` label for both data-driven charts and natural photographs. IIIT-AR-13K separates these into `Figure` (charts, plots, diagrams, sketches) and `Natural Image` (photographs), enabling more precise encoder selection for downstream tasks. ## What is the novelty? The main novelty is the class taxonomy, which explicitly decomposes visual content into five categories: | Class | Definition | Bevaya Taxonomy Mapping | | :--- | :--- | :--- | | **Table** | Structured row-column data. | `Table` | | **Figure** | Data-driven charts, plots, diagrams, sketches. | `Chart` / `Diagram` | | **Natural Image** | Photographs and raster scenes. | `Figure` | | **Logo** | Company branding elements. | `Logo` | | **Signature** | Handwritten verification zones. | `Signature` | The dataset is also notable for its language diversity. Annual reports were collected in English, French, Japanese, and Russian, introducing script variation absent from prior layout datasets. A secondary claim is that smaller, high-quality human annotations can outperform much larger automatically generated datasets for transfer to new domains, which the cross-dataset study provides partial evidence for. ## What experiments were performed? **Baseline detection on IIIT-AR-13K.** Faster R-CNN (ResNet-101, PyTorch) and Mask R-CNN (ResNet-101, TensorFlow/Keras) were trained on the IIIT-AR-13K training split and evaluated on its validation and test splits. Both models were initialized from MS-COCO pretrained weights. Input images were resized to $800 \times 1024$ preserving aspect ratio. The evaluation metric is mAP (mean average precision) alongside per-class Precision, Recall, and F-measure. **Cross-dataset transfer study.** To establish whether IIIT-AR-13K is effective as a pretraining source, Mask R-CNN was trained separately on: - TableBank (LaTeX only, Word only, LaTeX+Word combined) - PubLayNet - IIIT-AR-13K Each trained model was then evaluated on the test sets of ICDAR-2013, ICDAR-POD-2017, cTDaR, UNLV, Marmot, and PubLayNet, using their common label (Table). This covered three fine-tuning conditions: no fine-tuning, full fine-tuning on the target dataset's training split, and partial fine-tuning with only 1k randomly selected images from the target dataset. ## What are the outcomes/conclusions? ### Baseline Performance on IIIT-AR-13K (Mask R-CNN, Test Set) | Class | Recall | Precision | F-measure | mAP | | :--- | ---: | ---: | ---: | ---: | | Table | 0.9711 | 0.9715 | 0.9713 | 0.9654 | | Figure | 0.8898 | 0.7872 | 0.8385 | 0.8686 | | Natural Image | 0.9179 | 0.8625 | 0.8902 | 0.8945 | | Logo | 0.6330 | 0.3920 | 0.5125 | 0.4699 | | Signature | 0.9175 | 0.7876 | 0.8525 | 0.9115 | | **Average** | **0.8659** | **0.7601** | **0.8130** | **0.8220** | Mask R-CNN outperforms Faster R-CNN across all classes. Table and Signature are the strongest classes. Logo is the weakest: the authors attribute this directly to class imbalance in the training set (approximately 11k table instances vs. 0.3k logo instances), not to visual ambiguity alone. ### Cross-Dataset Transfer (No Fine-tuning) The IIIT-AR-13K-trained model (Experiment-V) achieves the best mAP on ICDAR-2013 (0.9393), cTDaR (0.7478), and UNLV (0.7996) without any target-domain fine-tuning. On Marmot, IIIT-AR-13K (0.8464) is effectively tied with TableBank-LaTeX (0.8465), which the paper acknowledges as the best on that benchmark. On ICDAR-POD-2017, the TableBank (LaTeX+Word) model outperforms it substantially (mAP 0.9035 vs. 0.7509). The authors attribute IIIT-AR-13K's general advantage to its business-document diversity, but the ICDAR-POD-2017 result is a meaningful exception where scientific-domain training data is more appropriate. ### Fine-tuning Findings When any of the larger datasets (TableBank, PubLayNet) are fine-tuned with the full IIIT-AR-13K training set, performance on IIIT-AR-13K's test set rises to match or closely approach training directly on IIIT-AR-13K. More practically: fine-tuning with **only 1k randomly selected IIIT-AR-13K images** is sufficient to bring models pretrained on larger datasets to near-parity with full IIIT-AR-13K training. This suggests IIIT-AR-13K is efficient as a fine-tuning target even when the full training set is not used. ## Reproducibility ### Models - **Faster R-CNN:** ResNet-101 backbone, PyTorch. Five anchor scales (32, 64, 128, 256, 512) and five anchor ratios (1, 2, 3, 4, 5). Pretrained on MS-COCO. - **Mask R-CNN:** ResNet-101 backbone, TensorFlow/Keras. Anchor scales 0.5, 1, 2; anchor box sizes 32, 64, 128, 256, 512. Pretrained on MS-COCO. - No pretrained checkpoints released for either model. ### Algorithms - **Faster R-CNN:** SGD optimizer, initial learning rate 0.001, multiplied by 0.1 every 5 epochs. Batch size 4. - **Mask R-CNN:** 80 total epochs. First 20: all FPN and subsequent layers. Next 20: FPN + last 4 ResNet-101 layers. Final 40: all layers. Learning rate 0.001, momentum 0.9, weight decay 0.0001. Batch size 1. - Input resolution: $800 \times 1024$ (aspect ratio preserved) for both models. ### Data - **Pages:** 13,415 total (9,333 train / 1,955 validation / 2,120 test). Split per company: 70% / 15% / 15% randomly selected pages per company. - **Bounding boxes:** ~23,000 total across five classes: ~16k Table, ~3k Figure, ~3k Natural Image, ~0.5k Logo, ~0.6k Signature. - **Sources:** Publicly available annual reports in English, French, Japanese, and Russian; more than ten years of filings from 29 companies. - **Annotation:** Manual bounding box annotation; axis-aligned rectangles only. No inter-annotator agreement metrics are reported, and the annotation guidelines are not described in detail. - **Format:** Not specified in the paper; available via static download from the project website. - **License:** Project site states "Copyright 2020 All rights reserved." Intended for academic research. Verify terms before commercial use. ### Evaluation - **Metrics:** mAP, Precision, Recall, F-measure (per class and averaged). The paper does not specify the IoU threshold for mAP (likely mAP@0.5 given the era and baselines, but not stated explicitly). - **Transfer benchmarks:** ICDAR-2013 (238 pages, table only), ICDAR-POD-2017 (2,417 pages, table/figure/equation), cTDaR (modern + archival), UNLV (427 pages with tables), Marmot (2,000 Chinese/English pages), PubLayNet (validation split). - **Baseline:** No hyperparameter search; standard detectron / Keras default configs adapted for document resolution. ### Hardware - **NVIDIA Titan X GPU, 12GB memory** (both Faster R-CNN and Mask R-CNN). - Training duration not reported. ## BibTeX ```bibtex @inproceedings{mondal2020iiit, title={IIIT-AR-13K: A New Dataset for Graphical Object Detection in Documents}, author={Mondal, Ajoy and Lipps, Peter and Jawahar, C. V.}, booktitle={Document Analysis Systems (DAS)}, year={2020}, doi={10.1007/978-3-030-57058-3\_16} } ```