# CISOL: An Open and Extensible Dataset for Table Structure Recognition in the Construction Industry

> CISOL is a CC-BY-4.0 dataset of 844 annotated scanned documents from the civil engineering industry, supporting table detection and structure recognition with over 120,000 annotated instances and a public evaluation server.

- Authors: Hunter Heidenreich
- Published: 2025-01-26
- Updated: 2026-03-30
- Dataset: CISOL Dataset (Zenodo) <https://doi.org/10.5281/zenodo.10829550>
- Paper: Preprint (arXiv) <https://arxiv.org/abs/2501.15469>
- Benchmark: Evaluation Server (EvalAI) <https://eval.ai/web/challenges/challenge-page/2257>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## TL;DR

CISOL (Construction Industry Steel Ordering Lists) is a medium-sized, human-annotated dataset of 844 scanned civil engineering documents with over 120,000 annotated instances for table detection (TD) and table structure recognition (TSR). All documents are anonymized real-world steel ordering lists in German, sourced from 10 structural engineering firms. The dataset is released under CC-BY-4.0 on Zenodo and is accompanied by a public evaluation server. Benchmarking with YOLOv8 reaches 67.22 mAP@0.5:0.95:0.05, outperforming the TSR-specific Table Transformer (TATR) on this domain, while both results remain below the annotation-derived convergence threshold.

## What kind of paper is this?

**Dominant:** $\Psi\_{\text{Resource}}$. The primary contribution is the dataset itself: a new, openly licensed, human-annotated collection of real-world civil engineering documents that fills a gap in available table extraction data. The paper documents a thorough creation pipeline, an annotation guideline, a stratified splitting strategy, and a long-term public evaluation server. The dataset release is the headline novelty.

**Secondary:** $\Psi\_{\text{Evaluation}}$. The authors measure annotation consistency using Krippendorff's $\alpha$ and derive a label convergence threshold that defines a theoretical upper bound on model performance. They then benchmark YOLOv8 and TATR against this bound, providing a measurement framework as well as a dataset.

## What is the motivation?

Table extraction from scanned documents is a long-standing problem in document analysis. Most publicly available table extraction datasets come from born-digital government documents, scientific articles, or financial reports. Civil engineering documents, such as steel ordering lists, have a distinctive structure: large tables with spanning cells, embedded tables, technical drawings, and highly domain-specific content. No dataset for this sub-domain existed at the time of writing.

The authors also identify a broader problem in dataset creation: inconsistent annotation conventions, opaque collection pipelines, inadequate licensing, and poor extensibility. Prior human-annotated datasets such as ICDAR 2013, WTW, and TabRecSet lack metadata that would allow others to replicate or extend the annotation process. CISOL is designed from the outset to address these documentation gaps by following the data development lifecycle described by Hutchinson et al., the datasheet framework of Gebru et al., and the FAIR principles (Findability, Accessibility, Interoperability, Reusability).

## What is the novelty?

The novelty is the CISOL dataset itself and the process used to create it. Key properties that distinguish it from prior human-annotated TSR datasets include:

1. **Domain specificity**: all documents come from real construction industry projects, a domain not previously represented in TSR benchmarks.
2. **Anonymized real-world data**: raw documents from 10 structural engineering firms (24 projects, creation dates 2015-2023) were anonymized via an automated pass (replacing names, locations, and personal data with length-preserving substitutions) followed by manual review. This allowed real industry data to be licensed under CC-BY-4.0.
3. **Rich metadata**: each document image is tagged with table size (XS/S/M/L by cell count) and table type (open/closed/mixed by separator line usage), plus provenance metadata. This metadata enables stratified sampling and is available to downstream researchers.
4. **Extensible annotation pipeline**: the annotation guideline and the full pipeline (using the publicly available CVAT tool) are released, making it straightforward for others to add new documents or sub-domains without writing a new guideline from scratch.
5. **Physical structure focus**: the dataset annotates physical structure (column, row, spanning cell, and header bounding boxes) rather than logical structure. This design choice makes downstream heuristic inference of logical structure simpler, supports embedded table recognition extensions, and reduces the need for speculative inference during annotation.
6. **Annotation quality measurement and convergence threshold**: 20 images were re-annotated by all four annotators. Krippendorff's $\alpha$ ($K\text{-}\alpha$) was computed at multiple IoU thresholds and converted to mAP to yield a label convergence threshold, i.e., a theoretical ceiling on model performance given the annotation variance. The authors interpret $K\text{-}\alpha > 0.8$ as reliable agreement and $K\text{-}\alpha > 0.667$ as moderate agreement; CISOL achieves the former at IoU 0.5 and the latter at the aggregate $0.5\text{:}0.95\text{:}0.05$ range. The conversion from $K\text{-}\alpha$ to equivalent mAP follows the formula from Tschirschwitz and Rodehorst (WACV 2025). The convergence threshold sits above the best benchmark result, indicating the problem is not yet solved.

## What experiments were performed?

The paper benchmarks two models on CISOL: YOLOv8 (a general-purpose object detector) and TATR (Table Transformer, a TSR-specific model). Both models are trained on the CISOL training split and evaluated on the test split using the COCO mAP metric at IoU thresholds $0.5\text{:}0.95\text{:}0.05$, the same metric used in the PubTabNet ICDAR 2021 Challenge Track A.

The primary reported result is:

| Model | mAP@0.5:0.95:0.05 |
|---|---|
| TATR | not reported in paper (below YOLOv8) |
| YOLOv8 | 67.22 |
| Convergence threshold | above 67.22 |

The paper reports that YOLOv8 outperforms TATR on this dataset and attributes this to TSR-specific models generalizing poorly to new domains, consistent with a prior reproducibility study by Ajayi et al. Exact per-model and per-class numbers are not included in the paper; the authors direct readers to the public evaluation server at EvalAI for the current best results.

Annotation consistency is evaluated as a separate experiment. The four annotators each labeled the same 20 randomly selected images, and $K\text{-}\alpha$ was computed at IoU thresholds $0.5$, $0.75$, and $0.5\text{:}0.95\text{:}0.05$. The $K\text{-}\alpha$ values are converted to equivalent mAP values and plotted alongside benchmark results.

Dataset statistics are also characterized and compared to seven existing human-annotated TSR datasets (ICDAR 2013, ICDAR 2019 Track-B2, UNLV, UW3, TabRecSet, TUCD, WTW) on instance count, image count, class distribution, and annotation density. CISOL falls in the medium-size range and is distinguished by its larger average table size (more rows and columns per table) and the provision of metadata.

## What are the outcomes/conclusions?

The CISOL dataset provides a functional benchmark for TSR in a domain not previously covered. Key findings reported by the authors include:

- YOLOv8 achieves 67.22 mAP@0.5:0.95:0.05 on CISOL, outperforming TATR despite TATR being designed specifically for TSR. Results suggest that domain shift remains a significant challenge for TSR-specific models, and that general-purpose detectors can be competitive on out-of-distribution data.
- Annotation quality is moderate to high: $K\text{-}\alpha > 0.8$ at IoU 0.5 and $K\text{-}\alpha > 0.667$ at the aggregate IoU range. This indicates that the annotation guideline was sufficiently clear and that the labeling process was reproducible.
- The best benchmarked model remains below the convergence threshold, leaving room for further model improvement specific to this domain.
- Embedded tables are present in the raw data but were intentionally excluded from annotations in this release. They are identified as a prime extension target.

The authors identify several directions for future work: annotating embedded tables, extending to logical structure and cell content, and expanding to other sub-domains within civil engineering or other industries.

**Limitations:**
- The dataset covers a single narrow sub-domain (German-language steel ordering lists from 10 firms). Generalization beyond this sub-domain is unclear.
- Exact benchmarking numbers for TATR and per-class results are not in the paper; they require querying the external evaluation server.
- The dataset is openly licensed under CC-BY-4.0, which permits commercial use with attribution, but the underlying documents reflect a single geographic and industry context, limiting diversity.
- No model weights or training configurations are released for the benchmarked models.

## Reproducibility

### Models

- **YOLOv8**: general-purpose object detector (Ultralytics YOLO, January 2023 release). No architectural modifications reported. No weights released by the CISOL authors.
- **TATR (Table Transformer)**: DETR-based TSR model from Smock et al. (CVPR 2022), pre-trained on PubTables-1M. Fine-tuned on CISOL. No fine-tuned weights released.

### Algorithms

- Training details for the benchmarked models are not described in the paper beyond model selection. Hyperparameters, optimizer, learning rate, batch size, and epoch count are not reported.
- Annotation was performed using CVAT (Computer Vision Annotation Tool, CVAT.ai), a publicly available open-source tool.
- The annotation guideline is released alongside the dataset on Zenodo.
- Train/validation/test splitting used stratified sampling primarily on data origin (company), with size and type tags used to verify balance across subsets rather than as hard stratification criteria.

### Data

- **Total collected**: 3,288 document images from 24 reinforced concrete projects at 10 German structural engineering firms (2015-2023). Documents are scanned and in German.
- **Annotated subset**: 844 images, selected by stratified sampling with a preference for companies with fewer than 100 documents to ensure provenance diversity.
- **Annotation format**: COCO-JSON.
- **Classes**: table (for TD), column, row, spanning cell, header (for TSR).
- **Instance count**: over 120,000 annotated instances across both tasks.
- **Splits**: training, validation, and test sets. Stratified sampling on data origin (company) ensures provenance diversity; size and type distributions are verified to be consistent across splits.
- **License**: CC-BY-4.0.
- **Zenodo release contents** (five files, 772.5 MB total): annotation guidelines, tagged metadata, TD+TSR annotated dataset, TSR-only annotated dataset, and unlabeled images (full 3,288-image collection).
- **Anonymization**: automated pattern matching for names, locations, and personal data (length-preserving substitution), followed by manual review. Eight images removed for containing only cover page content.
- **Public availability (DOI)**: https://doi.org/10.5281/zenodo.10829550

### Evaluation

- **Primary metric**: COCO mAP at IoU $0.5\text{:}0.95\text{:}0.05$, the same metric used in the PubTabNet ICDAR 2021 Challenge.
- **Annotation quality metric**: Krippendorff's $\alpha$ computed at IoU thresholds $0.5$, $0.75$, and $0.5\text{:}0.95\text{:}0.05$ on 20 re-annotated images.
- **Convergence threshold**: $K\text{-}\alpha$ values are converted to mAP to yield a theoretical performance ceiling. The formula follows Tschirschwitz and Rodehorst (WACV 2025).
- **Evaluation server**: https://eval.ai/web/challenges/challenge-page/2257, with held-out test annotations not publicly released.
- No error bars or significance tests are reported for the benchmark results. Number of training runs and seeds are not stated.

### Hardware

- Training and evaluation hardware are not reported in the paper.
- No GPU-hour, VRAM, or inference latency figures are provided.

## BibTeX

```bibtex
@InProceedings{Tschirschwitz_2025_WACV,
  author = {Tschirschwitz, David and Rodehorst, Volker},
  title = {CISOL: An Open and Extensible Dataset for Table Structure Recognition in the Construction Industry},
  booktitle = {Proceedings of the Winter Conference on Applications of Computer Vision (WACV)},
  month = {February},
  year = {2025},
  pages = {7594-7602}
}
```