# ETD-ODv2: AI-Aided Annotation and Dataset for Layout Analysis of Long Documents > Research notes on ETD-ODv2, a 62K-page dataset for layout analysis of scanned theses and dissertations, with an AI-aided annotation framework to reduce labeling time and class imbalance. - Authors: Hunter Heidenreich - Published: 2026-03-13 - Paper: Published (WWW '23) - Dataset: ETD-ODv2 (dataset + code) - Source: Bevaya Labs, --- ## TL;DR Ahuja et al. propose an AI-aided annotation framework that uses a pre-trained object detection model to generate weak bounding-box labels, which human annotators then verify and correct. They release ETD-ODv2, a 62K-page, 300K-annotation dataset for layout analysis of electronic theses and dissertations (ETDs), including scanned documents and targeted pages for minority element classes. Models trained on the combined dataset achieve a mAP@0.50 of 0.886, up from 0.774 with digital-only data. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Resource}}$ . The headline contribution is ETD-ODv2, a new dataset that extends ETD-OD with 16.8K scanned pages and 20.2K AI-aided annotated pages. The annotation framework is in service of producing this resource. **Secondary:** $\Psi\_{\text{Method}}$ . The AI-aided annotation scheme (pre-trained model generates weak labels, optional class-based filtering, human verification) is a methodological contribution, though it is relatively straightforward and primarily described as a pipeline rather than an algorithmic novelty. ## What is the motivation? Layout analysis of long scholarly documents (theses, dissertations) faces three specific challenges: 1. **Annotation cost:** Labeling bounding boxes on document pages is slow. No LaTeX source is available for scanned documents, so automatic annotation methods (used by PubLayNet, DocBank) do not apply. 2. **Scanned document noise:** Legacy ETDs were typewritten, microfilmed, and/or scanned, introducing noise, low resolution, dilated/eroded text, and handwritten elements that cause models trained on clean digital PDFs to underperform. 3. **Class imbalance:** Certain elements (title, author, algorithm, equation number) appear on only a handful of pages per document, leading to severe frequency skew in page-level datasets. The prior ETD-OD dataset (Ahuja et al., 2022) addressed digital ETDs only and suffered from the imbalance problem. ETD-ODv2 is designed to fill both gaps. ## What is the novelty? ### AI-Aided Annotation Framework The framework has four stages: 1. **Dataset sampling:** Sample unlabeled ETDs from open-access institutional repositories and split into page images. 2. **Weak label generation:** Run a YOLOv7 model (pre-trained on ETD-OD + a small scanned seed set) on each page to produce predicted bounding boxes and class labels. 3. **Optional filtering:** Use predicted labels to select pages likely to contain minority-class elements (e.g., title, algorithm, equation number). This directly addresses class imbalance by over-sampling rare pages. 4. **Human verification and correction:** Annotators review model predictions, keeping correct ones and fixing incorrect/missing boxes. The key practical insight is that even a model trained on limited, imbalanced data produces enough correct predictions to substantially reduce annotation effort (2-3$\times$ speedup). ### ETD-ODv2 Dataset The dataset extends ETD-OD with two new subsets: - **Scanned subset:** ~16.8K pages from 100 scanned theses/dissertations, manually annotated by 5 undergraduates with author review. Captures noise, low resolution, and handwritten elements absent from digital ETDs. - **AI-aided subset:** ~20.2K pages from ~1,200 documents (scanned + digital), filtered to over-represent 12 minority element classes, then verified/corrected by 4 annotators. Combined with the original ETD-OD digital subset (25K pages), the full dataset contains: - **62,043 page images** - **300,388 bounding-box annotations** - **24 object categories** (metadata elements like title/author/degree, structural elements like paragraph/section/chapter title, and content elements like equation/table/figure/algorithm) ## What experiments were performed? ### Annotation Time Study Four annotators each labeled ~500 pages under three conditions: | Setting | Description | | :--- | :--- | | **No Model** | Classical annotation; no bounding boxes shown | | **AI-Aided-v1** | YOLOv7\_base model generates initial boxes | | **AI-Aided-v2** | YOLOv7 fine-tuned on 10K additional AI-aided pages generates boxes | Average time per page decreased by 2-3$\times$ with model assistance (AI-Aided-v1 vs. No Model). AI-Aided-v2 further reduced time beyond v1, confirming that better models yield more useful suggestions. ### Object Detection Performance YOLOv7 was trained on four dataset configurations and evaluated on a held-out test set of 9,353 pages (44,331 objects) spanning digital, scanned, and AI-aided sources: | Training Data | Images | Objects | mAP@0.50 | mAP@0.50:0.95 | | :--- | :--- | :--- | :--- | :--- | | Digital only | 21,313 | 85,540 | 0.774 | 0.573 | | Scanned only | 14,204 | 53,834 | 0.596 | 0.356 | | Digital + Scanned | 35,517 | 139,374 | 0.832 | 0.590 | | Digital + Scanned + AI-Aided | 52,690 | 256,057 | **0.886** | **0.655** | Metrics: COCO-style AP@0.50 and AP@0.50:0.95. ## What are the outcomes/conclusions? **Key results:** - The AI-aided annotation framework reduces per-page annotation time by 2-3$\times$ compared to manual labeling, and the benefit increases as the assisting model improves. - Adding scanned and AI-aided data to the training set improves mAP@0.50 from 0.774 (digital-only) to 0.886, an 11.2 percentage point gain. - Minority classes benefit the most from targeted annotation. Algorithm detection jumps from 0.368 to 0.665 AP@0.50 (a ~30 pp gain); Degree from 0.524 to 0.732 (~21 pp). - The filtering step does not hurt majority-class performance; in fact, pages selected for minority elements also contain majority elements like paragraphs, providing additional training signal across all classes. **Limitations and open questions:** - The annotation framework relies on an existing pre-trained model. If the seed model is very poor (e.g., for a completely new document domain), the weak labels may not provide enough benefit to justify the pipeline overhead. - Inter-annotator agreement is not reported. Although each scanned sample received a second review by an author, no formal IAA metric (e.g., Krippendorff's Alpha) is provided. - The test set was constructed from the same sources as the training data (ETD-OD + ETD-ODv2), so generalization to non-ETD long documents (books, legal filings, government reports) is untested. - Only YOLOv7 was evaluated as the object detection model. The authors acknowledge this but do not test other architectures (e.g., Faster R-CNN, DETR variants) to confirm the gains are model-agnostic. - The paper does not discuss annotation quality differences across the five undergraduate annotators or the four AI-aided annotators. - The GitHub repository does not specify a license, making the legal terms for dataset use unclear despite the paper being published under CC-BY-NC-4.0. ## Reproducibility ### Models - **YOLOv7** used as the detection model throughout (both for AI-aided annotation and final evaluation). - YOLOv7\_base: trained on ETD-OD + ~2K scanned pages. Specific hyperparameters not detailed. - YOLOv7\_v2: YOLOv7\_base fine-tuned on an additional 10K AI-aided pages. - The paper's conclusion claims "the code, datasets, and pre-trained models discussed in this paper are available at" the GitHub repository. However, the repo contains only a README.md file with a three-sentence description. No code, data files, or model weights are present. The repo was last updated October 2022 and has no license file. ### Algorithms - Training procedure details (learning rate, optimizer, epochs, augmentation) are not reported. - The AI-aided annotation pipeline uses pylabel (open-source bounding box widget) integrated with the pre-trained model. - Roboflow was used as the annotation platform for the scanned subset. ### Data - **Source:** ETDs from open-access US institutional repositories, uniformly sampled. Specific repositories are not enumerated. - **Scanned subset:** 100 documents, ~16.8K pages. Annotated by 5 undergraduates, reviewed by an author. - **AI-aided subset:** ~1,200 documents, ~20.2K pages. Filtered for 12 minority element classes. Annotated/corrected by 4 annotators. - **Digital subset:** From ETD-OD (Ahuja et al., 2022), ~25K pages. - **Object categories:** 24 classes compatible with ETD-OD. - Code and data are said to be available at https://github.com/Opening-ETDs/ETD-OD. However, the repository contains only a README.md and no actual code, data, or model files. Last updated October 2022. ### Evaluation - Metrics: COCO-style mAP@0.50 and mAP@0.50:0.95 (standard). - Test set: 9,353 pages / 44,331 objects, sampled from all three subsets (digital, scanned, AI-aided). - Only YOLOv7 is used, limiting the generalizability of the performance comparison. - No error bars, significance tests, or multi-run statistics are reported. - No cross-dataset evaluation (e.g., DocLayNet, PubLayNet). ### Hardware - Not reported. No mention of GPU type, training time, or inference speed. ## BibTeX ```bibtex @inproceedings{ahuja2023etdodv2, author = {Ahuja, Aman and Dinh, Kevin and Dinh, Brian and Ingram, William A. and Fox, Edward A.}, title = {A New Annotation Method and Dataset for Layout Analysis of Long Documents}, booktitle = {Companion Proceedings of the ACM Web Conference 2023}, year = {2023}, pages = {835--841}, publisher = {ACM}, doi = {10.1145/3543873.3587609}, } ```