# DSSE-200: Document Semantic Structure Extraction Dataset > DSSE-200 is a 200-page human-annotated benchmark of raster document images (magazines and academic papers) with 6 semantic classes, introduced alongside the Multimodal FCN (MFCN) for pixel-level layout segmentation. - Authors: Hunter Heidenreich - Published: 2017-07-21 - Updated: 2026-03-12 - Paper: Published Paper (CVF Open Access) - Paper: Preprint (arXiv) - Dataset: DSSE-200 Dataset - Source: Bevaya Labs, --- ## TL;DR DSSE-200 is a 200-page human-annotated benchmark of raster document images (magazines and academic papers), with pixel-level labels for 6 classes: figure, table, section heading, caption, list, and paragraph. It was introduced with a Multimodal Fully Convolutional Network (MFCN) that fuses visual features with text embedding maps. Notably, DSSE-200 consists of raster images rather than PDFs, which precludes PDF-based post-processing approaches. The dataset URL appears to be dead. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Resource}}$ : The paper introduces two reusable datasets: DSSE-200 (a 200-page human-annotated benchmark) and a 135,000-image synthetic pre-training set generated from document templates. We classify this as Resource-dominant because the note's purpose is to catalog the dataset, though the paper itself devotes more page space to its method. **Secondary:** $\Psi\_{\text{Method}}$ : The paper also introduces MFCN, a multimodal FCN that fuses visual features with a text embedding map derived from skip-gram word embeddings, plus unsupervised reconstruction and consistency auxiliary losses. The ablation study structure and SOTA comparisons are strong methodological indicators. ## What is the motivation? Document layout analysis at the time relied either on heuristic rule-based systems (exploiting PDF metadata or whitespace geometry) or on purely visual models trained on limited data. Both approaches have weaknesses: PDF-based methods fail on scanned or raster documents, while visual-only models lack the semantic grounding needed to distinguish visually similar but functionally different regions (e.g., a figure caption vs. body paragraph). The paper argues that documents are inherently multimodal, carrying both visual structure and textual meaning, and that combining both signals in a single end-to-end trainable model should improve layout segmentation. The lack of a suitable benchmark mixing magazines and academic papers motivated the release of DSSE-200. ## What is the novelty? **MFCN (Multimodal Fully Convolutional Network):** The architecture consists of four parts: an encoder, two decoders, and a bridge. The encoder learns a hierarchy of visual features using dilated convolution blocks (five 3x3 dilated convolutions with $d = 1, 2, 4, 8, 16$) and unpooling-based skip connections. The bridge module concatenates the visual feature map with a text embedding map along the channel dimension. The main decoder outputs per-pixel class probabilities, while an auxiliary decoder handles unsupervised reconstruction during training. The text embedding map is constructed by running OCR to localize words, embedding each word via a 128-dimensional skip-gram model trained on Wikipedia, averaging word embeddings within each sentence, and then filling each pixel within that sentence's bounding region with the resulting sentence vector. For an $H \times W$ document image, this yields an embedding map of size $N \times H \times W$ (where $N = 128$). Two unsupervised auxiliary tasks regularize training when labeled data is scarce: - **Reconstruction loss:** Given encoder activations $a\_l$ at layer $l$ and their reconstructions $\hat{a}\_l$ from the auxiliary decoder, the loss is: $$L\_{\text{rec}}^{(l)} = \frac{1}{C\_l H\_l W\_l} \lVert a\_l - \hat{a}\_l \rVert\_2^2, \quad l = 0, 1, 2, \ldots, L$$ - **Consistency loss:** For pixels within a bounding box region $b$ of size $H\_b \times W\_b$, the loss encourages intra-region feature consistency: $$L\_{\text{cons}} = \frac{1}{H\_b W\_b} \sum\_{(i,j) \in b} \lVert p\_{(i,j)} - p^{(b)} \rVert\_2^2$$ where $p^{(b)}$ is the mean feature vector over all pixels in box $b$: $$p^{(b)} = \frac{1}{H\_b W\_b} \sum\_{(i,j) \in b} p\_{(i,j)}$$ These auxiliary objectives allow leveraging both the 135,000 synthetic images and unlabeled real documents during semi-supervised training. **DSSE-200 Dataset:** 200 pages sampled from magazines and academic papers, with human-annotated pixel-level masks for 7 classes (6 semantic classes plus background): figure, table, section (heading), caption, list, paragraph. The images are raster (not PDF), which the authors explicitly note prevents applying PDF post-processing methods used by contemporaneous baselines. **Synthetic pre-training set:** 135,000 automatically generated document images from templates with known ground-truth layouts, used for unsupervised pre-training of the visual branch. ## What experiments were performed? The paper evaluates on three datasets: 1. **DSSE-200** (the introduced benchmark): 160 train / 40 test split, pixel-level IoU. 2. **ICDAR2015 competition dataset**: a broader document segmentation benchmark. 3. **SectLabel**: a dataset focused on scientific paper section structure. Baselines include vision-only FCN variants and several rule-based / PDF-based methods where applicable. On DSSE-200, the paper cannot compare against PDF-based methods since the documents are raster images. Ablations test: (1) vision-only FCN, (2) FCN + text embeddings, (3) adding the reconstruction auxiliary loss, (4) adding both auxiliary losses. The full model is evaluated with and without synthetic pre-training. The primary metric is **mean IoU** across all classes (pixel-level, not object-level mAP). This measures overlap at the pixel level, so results are not directly comparable to object-detection mAP numbers reported on other datasets. ## What are the outcomes/conclusions? On DSSE-200, the best vision-only architecture (Model5, with dilated blocks, unpooling, and skip connections) achieves **73.0% mean IoU**. Per-class IoU for this baseline: | Class | IoU | |---|---| | Background | 84.6% | | Figure | 83.3% | | Table | 79.4% | | Paragraph | 77.1% | | List | 66.7% | | Caption | 61.0% | | Section | 58.3% | Section headings are the hardest class, likely due to their small spatial footprint relative to surrounding text. Adding OCR-extracted text embeddings raises the mean IoU to **73.3%**, with the gains concentrated in textual classes (list +1.7%, paragraph +2.2%, section heading +1.1%). The unsupervised losses provide further additive improvement: reconstruction brings the mean to 73.9%, consistency to 75.4%, and combining both yields **75.9% mean IoU** (the paper reports only mean IoU for these ablations, not per-class breakdowns). On ICDAR2015, the binary-class MFCN achieves 94.5% (non-text) and 91.0% (text) IoU, outperforming prior baselines. On SectLabel, it achieves higher F1 than Luong et al. for section heading (0.919 vs. 0.916), caption (0.893 vs. 0.781), and list (0.793 vs. 0.712). The paper concludes that multimodal fusion of visual and text signals is beneficial for layout segmentation, especially on raster documents where PDF metadata is unavailable. **Limitations acknowledged by the authors:** - The text branch depends on OCR quality; errors in OCR propagate to the embedding map. - The 200-page test set is small, and variance across the 40 test pages is not reported. - Results on DSSE-200 are not comparable to methods that use PDF structure, since those methods cannot run on DSSE-200. **Dataset caveats:** - No explicit license is stated for DSSE-200. The source images are from magazines and academic papers with unclear copyright. Commercial use is not advisable without clarifying provenance. - The dataset URL (`http://personal.psu.edu/xuy111/projects/cvpr2017_doc.html`) appears to be a dead personal page and the data may no longer be publicly accessible. ## Reproducibility ### Models - **MFCN**: Encoder-decoder architecture following Noh et al., with modifications: dilated convolution blocks (five 3x3 convolutions at $d = 1, 2, 4, 8, 16$), unpooling-based upsampling, and refined skip connections. All convolutional layers use 3x3 kernels with stride 1, batch normalization before non-linearities. The models in the ablation study (Table 1) are trained from scratch. - **Text embeddings**: 128-dimensional skip-gram word vectors trained on the 2016 English Wikipedia dump. Out-of-vocabulary words handled following Bojanowski et al. Sentence embeddings are computed by averaging word vectors. - The paper does not provide a complete layer-by-layer specification or parameter count. - No model weights are released (as of the original publication). ### Algorithms - **Optimizer**: Adadelta with a mini-batch size of 2. - **Input preprocessing**: Per-channel mean subtraction; images resized so the longer side is at most 384 pixels. No other preprocessing. - **Semi-supervised training**: Mini-batches of synthetic and real documents alternate. For synthetic documents, both the per-pixel classification loss and unsupervised losses are active. For real (unlabeled) documents, only the unsupervised losses are active. - **Class balancing**: Per-pixel classification loss uses class weights inversely proportional to class pixel frequency. - **Auxiliary losses**: Reconstruction loss reconstructs encoder activations from the auxiliary decoder; consistency loss encourages pixels within the same bounding-box region to have similar feature representations. - **OCR engine**: Tesseract is used to extract text for the embedding map. ### Data - **DSSE-200**: 200 raster images (160 train, 40 test) from magazines and academic papers. Human-annotated pixel-level masks for 7 classes (6 semantic + background). Not PDF. - **135k synthetic images**: Automatically generated from document layout templates. Ground-truth layouts are known by construction. Used for unsupervised pre-training only. - **Public availability**: Both datasets appear to be permanently unavailable. DSSE-200 was hosted at `http://personal.psu.edu/xuy111/projects/cvpr2017_doc.html` (a PSU personal page that is now dead), and the 135k synthetic set was bundled on the same page. No mirrors, HuggingFace uploads, or alternative downloads are known to exist. This is a common fate for datasets hosted on personal academic pages from this era. - **License**: No explicit license stated for DSSE-200. ### Evaluation - **Metric**: Pixel-level mean IoU across all classes (including background). This is not the same as object-level mAP and is not directly comparable to COCO-style detection metrics. - **Splits**: 160 train / 40 test for DSSE-200. No cross-validation or multiple runs reported; statistical significance is not discussed. - **Baselines**: Vision-only FCN ablations; rule-based PDF methods (not applicable to DSSE-200 since it is raster-only). Results on ICDAR2015 and SectLabel are also reported. ### Hardware - No specific GPU type, training time, or memory requirements are reported in the paper. - As a 2017-era FCN with small batch size (2) and modest input resolution (384px), training and inference should be feasible on a single modern GPU with minimal VRAM. ## BibTeX ```bibtex @InProceedings{Yang_2017_CVPR, author = {Yang, Xiao and Yumer, Ersin and Asente, Paul and Kraley, Mike and Kifer, Daniel and Giles, C. Lee}, title = {Learning to Extract Semantic Structure from Documents Using Multimodal Fully Convolutional Neural Networks}, booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, year = {2017}, pages = {5315--5324}, doi = {10.1109/CVPR.2017.462}, } ```