# SwinDocSegmenter: An End-to-End Unified Domain Adaptive Transformer for Document Instance Segmentation > Research notes on SwinDocSegmenter, a unified Swin Transformer encoder-decoder for document instance segmentation with contrastive denoising training and domain-adaptive hybrid bipartite matching. - Authors: Hunter Heidenreich - Published: 2026-03-14 - Paper: Preprint (arXiv) - Paper: Published (ICDAR 2023) - Code: SwinDocSegmenter - Source: Bevaya Labs, --- ## TL;DR SwinDocSegmenter is a unified end-to-end transformer architecture for document instance segmentation that combines a Swin Transformer backbone with a DETR-style encoder-decoder, contrastive denoising training, and hybrid bipartite matching. It reports competitive results on PubLayNet (93.7 mAP), TableBank (98.0), and Historical Japanese (84.6), while providing the first transformer-based instance segmentation baseline on DocLayNet (76.9). The key finding is that generic MS-COCO pre-training combined with contrastive learning and adaptive matching reduces the domain bias caused by document-specific pre-training. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Method}}$ : The headline contribution is a unified transformer architecture for document instance segmentation with three components: mixed query selection with contrastive projection heads, contrastive denoising training adapted for masks, and hybrid bipartite matching for domain shift. Ablations and SOTA comparisons dominate the experimental sections. **Secondary:** $\Psi\_{\text{Evaluation}}$ : The paper provides extensive benchmarking across five datasets (PubLayNet, PRImA, HJDataset, TableBank, DocLayNet) and includes detailed ablations on backbone choice, resolution, query count, loss functions, and pre-training bias. ## What is the motivation? Document layout analysis has traditionally been addressed by detection models (bounding boxes), but instance segmentation (pixel-level, class-aware and instance-aware labels) provides richer structure for downstream document understanding tasks. The authors identify three specific gaps: 1. **Domain shift**: Transformer models pre-trained on domain-specific document datasets (e.g., PubLayNet for scientific articles) fail when applied to different document types (e.g., magazines, historical texts). The pre-training introduces bias toward common classes. 2. **No unified detection + segmentation**: Existing approaches treat detection and segmentation as separate or sequential tasks rather than jointly optimizing both. 3. **Low-frequency class penalty**: Transformers struggle with document element categories that have few training examples (e.g., separators, "other" regions), because standard matching and loss functions are biased toward high-frequency classes like text. ## What is the novelty? ### Unified Architecture The model combines a Swin Transformer Large (SwinL) backbone with a transformer encoder-decoder pair and a segmentation branch. The segmentation branch constructs a pixel embedding map (PEM) by fusing multi-scale features: $$M = Q\_e \otimes \delta(\Gamma(S\_b) + \psi(T\_e))$$ where $Q\_e$ is the query embedding from the decoder, $S\_b$ is the 1/4-resolution backbone feature map, $T\_e$ is the 1/8-resolution encoder feature map (upsampled via $\psi$), $\Gamma$ is a channel projection convolution, and $\delta$ is a segmentation head. ### Mixed Query Selection with Contrastive Projection The encoder output passes through three parallel heads (classification, detection, segmentation). Top-ranked features (by classification confidence) are selected as content queries. Two projection heads apply contrastive learning: **Low-level projection** (shallow MLP) for contrastive feature alignment: $$\mathcal{L}\_{low} = \sum\_{i=1}^{n} \sum\_{j=1}^{n'} -\log \frac{\exp(f\_i \cdot f\_j / \tau)}{\sum\_{c=1}^{k} \exp(f\_c \cdot f\_j / \tau)}$$ **High-level projection** (deep MLP) with learned prototypes $P = (p\_1, p\_2, \ldots, p\_m)$: $$\mathcal{L}\_{high} = \sum\_{i=1}^{n} \sum\_{j=1}^{n'} -\log \frac{\exp(f\_i \cdot p\_j / \phi\_j)}{\sum\_{c=1}^{k} \exp(f\_c \cdot p\_j / \phi\_j)}$$ where $\tau$ is a dataset-specific temperature and $\phi\_j$ is a concentration estimate per prototype. Content queries remain learnable (not initialized from encoder features), which the authors argue aids domain shift. ### Contrastive Denoising Training (CDN) Adapted from DINO-style denoising but extended to the segmentation task. Ground-truth boxes are randomly noised, and: - **Positive queries** (noise scale $< \lambda\_p$) are trained to reconstruct ground truth - **Negative queries** (noise scale between $\lambda\_p$ and $\lambda\_e$) are suppressed via focal loss The paper states $\lambda\_e > \lambda\_p$ but reports $\lambda\_p = 0.1$ and $\lambda\_e = 0.02$, which is an apparent contradiction in the text. The numerical values are reproduced here as published. For segmentation, the noised boxes serve as the denoising target, treating masks as the clean signal and boxes as a noised version. Noised queries are added during training and removed at inference. ### Hybrid Bipartite Matching The standard Hungarian matching is augmented with an additional mask prediction loss alongside L1 and focal loss. During domain shift from pre-trained weights, the mask prediction penalty is initially weighted more heavily, then decreased as the model converges. This addresses the inconsistency between masks predicted from different heads. ## What experiments were performed? ### Datasets | Dataset | Train Instances | Eval Instances | Classes | | :--- | :--- | :--- | :--- | | PubLayNet | 3.26M | 120K | 5 | | PRImA | 8K | 1.9K | 6 | | Historical Japanese | 181K | 31.9K | 7 | | TableBank | 2.8K | 1.4K | 1 | | DocLayNet | 91K | N/R | 11 | All models are initialized from MS-COCO Object Detection pre-trained weights. ### Metric COCO-style mAP at IoU thresholds 0.5 to 0.95 (step 0.05), plus AP@50, AP@75, and size-specific AP (small, medium, large). ### Main Results The paper compares against Layout Parser (multimodal, uses Microsoft OCR), DocSegTr (vision-only), and LayoutLMv3 (multimodal). Results are mAP (COCO-style): | Dataset | SwinDocSeg | Layout Parser | DocSegTr | LayoutLMv3 | | :--- | :--- | :--- | :--- | :--- | | PubLayNet | 93.72 | 86.7 | 90.4 | **95.1** | | PRImA | 54.39 | **64.7** | 42.5 | 40.3 | | Hist. Japanese | **84.55** | 81.6 | 83.1 | 82.7 | | TableBank | **98.04** | 91.2 | 93.3 | 92.9 | | DocLayNet (11-class) | **76.85** | N/A | N/A | N/A | SwinDocSegmenter outperforms all baselines on Historical Japanese and TableBank. On PubLayNet, LayoutLMv3 (which uses OCR text) leads at 95.1. On PRImA, Layout Parser (which also uses OCR text) leads at 64.7; the authors note that the convolution backbone and textual features give Layout Parser an advantage on this small, complex dataset. DocLayNet results are the first transformer-based instance segmentation baseline for this benchmark (compared against Mask R-CNN, Faster R-CNN, and YOLOv5 in a separate table). ### Ablation Studies **Backbone choice (PRImA):** SwinL (223M) achieves 54.4 AP vs. ResNet-50 (52M) at 36.1 AP and ViT-B (164M) at 46.1 AP. The 8-point gap over ResNet and 5-point gap over Swin-Tiny justify the parameter cost. **Input resolution (PRImA):** Performance increases with resolution: 256px (45.0 AP), 512px (50.1 AP), 1024px (54.4 AP). Higher resolutions were not feasible due to compute constraints. **Number of decoder queries (PRImA):** 300 queries (54.4 AP) outperform 100 queries (50.0 AP). The authors note SwinL could theoretically support 900-1200 queries but were limited by available compute. **Pre-training bias (PRImA):** MS-COCO pre-training (54.4 AP) substantially outperforms PubLayNet pre-training (49.4 AP). PubLayNet pre-training biases the model toward tables (70.7 vs. 49.9 AP) while severely penalizing rare classes like separators (8.6 vs. 27.6 AP). This finding validates the domain-adaptive design: generic pre-training with contrastive learning and hybrid matching outperforms domain-specific pre-training. **Loss function:** L1 + Focal Loss is the best combination, outperforming L2-based alternatives. ## What are the outcomes/conclusions? ### Key findings 1. **Generic pre-training outperforms domain-specific on cross-domain tasks:** MS-COCO pre-training with the proposed contrastive and matching mechanisms outperforms PubLayNet pre-training on PRImA (54.4 vs. 49.4 AP), reducing common-class bias. 2. **Competitive with multimodal methods using vision alone:** SwinDocSegmenter outperforms all baselines on HJDataset and TableBank, and outperforms the vision-only DocSegTr across all benchmarks. It falls short of multimodal methods (LayoutLMv3, Layout Parser) that leverage OCR text on PubLayNet and PRImA. 3. **First DocLayNet instance segmentation baseline:** The 11-class DocLayNet results (76.9 mAP) provide a reference point for future work; best classes are Text (88.2) and Table (87.4), weakest are Formula (62.3) and Title (63.3). ### Limitations 1. **PRImA performance gap:** Despite improvements, 54.4 mAP on PRImA is still relatively low, reflecting the challenge of small training sets with complex layouts. 2. **No text information:** The model is purely visual, which limits disambiguation of semantically similar regions (e.g., title vs. section header). 3. **Compute-limited exploration:** Resolution capped at 1024px and queries at 300 due to hardware constraints; higher settings could potentially improve results. 4. **Dataset-specific hyperparameters:** The temperature $\tau$ must be tuned per dataset (0.02 for PubLayNet, 0.6 for PRImA, 0.1 for HJDataset, 0.2 for TableBank), reducing out-of-the-box applicability. 5. **Weak on rare classes:** Categories with few instances ("Other" in PRImA at 7.1 AP, "Other" in HJDataset at 40.6 AP) remain challenging despite the contrastive approach. ## Reproducibility ### Models - Backbone: Swin Transformer Large (SwinL), 223M parameters total. - Architecture: Swin backbone + transformer encoder-decoder with deformable attention + segmentation branch. - Positional embeddings via 3$\times$3 convolutions. - Input resolution: 1024$\times$1024. - Decoder queries: 300. - Pre-trained from MS-COCO Object Detection weights. - Code available at [GitHub](https://github.com/ayanban011/SwinDocSegmenter) under Apache-2.0. Pre-trained checkpoints for all five benchmarks (PubLayNet, PRImA, HJDataset, TableBank, DocLayNet) are available via Google Drive links in the repo's Model Zoo. ### Algorithms - Loss: L1 + Focal Loss (detection) + mask prediction loss (segmentation) + contrastive losses ($\mathcal{L}\_{low}$, $\mathcal{L}\_{high}$). - CDN hyperparameters: $\lambda\_p = 0.1$, $\lambda\_e = 0.02$ (paper states $\lambda\_e > \lambda\_p$ but provides these contradictory values). - Temperature: dataset-specific ($\tau$ = 0.02/0.6/0.1/0.2 for PubLayNet/PRImA/HJ/TableBank). - Optimizer, learning rate, batch size, number of epochs: **not reported**. ### Data - PubLayNet, PRImA, HJDataset, TableBank, DocLayNet: all publicly available. - Standard train/eval splits used. - No data augmentation details reported. ### Evaluation - COCO-style mAP (IoU 0.5:0.05:0.95), plus AP@50, AP@75, AP\_S, AP\_M, AP\_L. - Compared against Mask R-CNN, SOLOv2, CondInst, Mask2Former, DocSegTr, and others. - No error bars, significance tests, or multi-run statistics reported. - Ablations only on PRImA; results may not generalize across all datasets. ### Hardware - No training hardware, GPU hours, or memory requirements reported. - No inference latency or throughput measurements. - Authors note compute constraints limited resolution and query exploration. ## BibTeX ```bibtex @inproceedings{banerjee2023swindocsegmenter, title={SwinDocSegmenter: An End-to-End Unified Domain Adaptive Transformer for Document Instance Segmentation}, author={Banerjee, Ayan and Biswas, Sanket and Llad{\'o}s, Josep and Pal, Umapada}, booktitle={International Conference on Document Analysis and Recognition (ICDAR)}, pages={307--325}, year={2023}, publisher={Springer}, doi={10.1007/978-3-031-41734-4_19} } ```