# GraphKD: Graph-Based Knowledge Distillation for Document Object Detection > Research notes on GraphKD, a graph-based knowledge distillation framework for document object detection that enables heterogeneous teacher-student distillation via structured instance graphs. - Authors: Hunter Heidenreich - Published: 2026-03-14 - Paper: Preprint (arXiv) - Paper: Published (ICDAR 2024) - Code: GraphKD - Source: Bevaya Labs, --- ## TL;DR GraphKD is a graph-based knowledge distillation framework for document object detection (DOD) that constructs structured instance graphs from RoI-pooled features and transfers them from a large teacher to a compact student network. By using cosine similarity for node-to-node distillation and Mahalanobis distance for edge-to-edge distillation, the framework enables heterogeneous distillation (e.g., ResNet to EfficientNet) and includes an adaptive text-node sampling strategy to reduce text-class bias. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Method}}$ : The headline contribution is a graph-based knowledge distillation framework specifically designed for document object detection. The paper proposes a structured instance graph construction from RPN proposals, a new node sampling strategy for text bias reduction, and a combined cosine/Mahalanobis distillation loss. Ablations and SOTA comparisons dominate the experimental sections. **Secondary:** $\Psi\_{\text{Evaluation}}$ : The paper provides an extensive comparative evaluation across four benchmarks, testing both homogeneous and heterogeneous distillation configurations against three competing KD methods (ReviewKD, NKD, SimKD). ## What is the motivation? Document object detection models have grown increasingly complex (e.g., SwinDocSegmenter at 223M params, LayoutLMv3 at 368M params), making deployment on edge devices impractical. Knowledge distillation offers a path to compact models, but existing KD techniques face three problems in the DOD setting: 1. **Feature imbalance**: Documents are dominated by text regions, creating a class imbalance that biases standard distillation. 2. **Missing instance relations**: Logit-based KD loses fine-grained spatial information; feature-based KD suffers from alignment problems across heterogeneous architectures. 3. **Homogeneity constraint**: Conventional layer-to-layer distillation requires matching teacher/student architectures, preventing cross-architecture transfer (e.g., ViT to CNN). The authors note this is the first application of knowledge distillation to the DOD task specifically. ## What is the novelty? ### Structured Instance Graph Rather than distilling raw feature maps or logits, GraphKD constructs a graph $G = (V, \xi)$ where: - **Nodes** ($V$): Vectorized RoI-pooled features from region proposals, categorized into "text" and "non-text" based on feature covariance. - **Edges** ($\xi$): Cosine similarity between node pairs, forming a complete, symmetric, undirected graph: $$e\_{pq} = \frac{v\_p \cdot v\_q}{\|v\_p\| \cdot \|v\_q\|}$$ Teacher and student share the same RPN proposals, ensuring aligned anchor boxes so both backbones extract features from the same regions. ### Adaptive Text Node Sampling To handle the text-dominance problem, the framework applies a mining strategy (Algorithm 1): text nodes with high classification loss in the teacher (likely to be misclassified) are merged with non-text nodes for distillation, while confident text nodes are pruned. This reduces biased edges and improves false-negative regularization. ### Graph Distillation Loss The total loss combines node losses (text and non-text) with an edge loss, all computed via Mahalanobis distance: $$\mathcal{L}\_g = \frac{\lambda\_1}{N\_{nt}} \sum\_{i=1}^{N\_{nt}} \left\|\frac{v\_i^{t,nt} - v\_i^{s,nt}}{\sigma\_{nt}}\right\| + \frac{\lambda\_2}{N\_{text}} \sum\_{i=1}^{N\_{text}} \left\|\frac{v\_i^{t,text} - v\_i^{s,text}}{\sigma\_{text}}\right\| + \frac{\lambda\_3}{N^2} \sum\_{i=1}^{N} \sum\_{j=1}^{N} \left\|\frac{e\_{ij}^t - e\_{ij}^s}{\sigma\_\xi}\right\|$$ where $\lambda\_1 = \lambda\_3 = 0.3$ (Optuna search) and $\lambda\_2 = \alpha \cdot \frac{N\_{nt}}{N\_{text}}$ adapts to the class imbalance. The key insight from ablations is that cosine similarity works best for node-to-node alignment (captures orthogonality) while Mahalanobis distance works best for edge-to-edge alignment (sensitive to outliers). A final GCN + cross-entropy loss classifies the non-text nodes into specific object categories beyond the binary text/non-text split. ## What experiments were performed? ### Datasets Four benchmarks covering diverse document types: | Dataset | Classes | Train Instances | Eval Instances | | :--- | :--- | :--- | :--- | | PubLayNet | 5 | 3.26M | 120K | | PRImA | 6 | 8K | 1.9K | | Historical Japanese | 7 | 181K | 31.9K | | DocLayNet | 11 | 994K | 66.5K | ### Distillation Configurations - **Homogeneous**: ResNet152 $\rightarrow$ ResNet101, ResNet101 $\rightarrow$ ResNet50 - **Heterogeneous**: ResNet50 $\rightarrow$ ResNet18, ResNet101 $\rightarrow$ EfficientNet-B0, ResNet50 $\rightarrow$ MobileNetV2 ### Baselines - **KD methods**: ReviewKD, NKD, SimKD (all adapted to the DOD setting) - **Supervised upper bounds**: SwinDocSegmenter (223M), LayoutLMv3 (368M), DocSegTr (168M) ### Ablations 1. **Component ablation** (Table 2): Tested edge-only, edge+non-text, edge+text, and full (edge+non-text+text) on DocLayNet with ResNet50 $\rightarrow$ ResNet18. Full model achieves 42.1 AP vs. 33.1 AP edge-only. 2. **Distance function ablation** (Table 3): Exhaustive 4$\times$4 grid of L1, L2, Cosine, and Mahalanobis for node and edge losses. Best: Cosine (nodes) + Mahalanobis (edges) at 42.1 AP. ### Metric COCO-style mAP at IoU thresholds 0.5 to 0.95 (step 0.05). ## What are the outcomes/conclusions? ### Key Results GraphKD consistently outperforms competing KD methods (ReviewKD, NKD, SimKD) across all four benchmarks and all five teacher-student configurations. The best configuration (ResNet152 $\rightarrow$ ResNet101) achieves: | Dataset | GraphKD AP | Best Competing KD AP | Supervised SOTA AP | | :--- | :--- | :--- | :--- | | PubLayNet | 88.8 | 81.1 (SimKD) | 95.1 (LayoutLMv3) | | PRImA | 41.9 | 36.2 (SimKD) | 54.3 (SwinDocSeg) | | Hist. Japanese | 79.7 | 76.7 (SimKD) | 85.0 (SwinDocSeg) | | DocLayNet | 68.9 | 64.6 (SimKD) | 72.1 (SwinDocSeg) | ### Noteworthy observations - On PRImA (a small dataset), the distilled ResNet101 (44.5M params) outperforms LayoutLMv3 (368M) by ~1.6 AP, suggesting large transformers overfit on limited data. - On DocLayNet, distilled models outperform DocSegTr and LayoutLMv3 on specific classes like "Caption," "Page-footer," and "Picture," where text-bias reduction helps. - Heterogeneous distillation (e.g., ResNet50 $\rightarrow$ MobileNetV2 at 3.4M params) is functional but incurs substantial performance drops due to double compression of RoI features. ### Limitations - The performance gap to supervised SOTA remains significant: ~7% on PubLayNet/DocLayNet and ~13% on PRImA vs. SwinDocSegmenter. - Transformer backbones cannot serve as teacher or student due to incompatible data handling at the RPN/RoI level. The authors acknowledge this as a key limitation and future work direction. - The framework is built entirely on Faster R-CNN with RPN; it does not generalize to anchor-free or DETR-style detectors. - No latency or inference speed benchmarks are reported, despite efficiency being the stated motivation. - The "first KD for DOD" claim is plausible but difficult to verify exhaustively. ## Reproducibility ### Models - All teacher/student pairs use standard backbones: ResNet18/50/101/152, EfficientNet-B0, MobileNetV2. - The detection framework is Faster R-CNN with a shared RPN between teacher and student. - A GCN is appended for final node classification (text $\rightarrow$ specific object categories). - Code is available on [GitHub](https://github.com/ayanban011/GraphKD) under MIT license, built on Detectron2. ### Algorithms - Loss penalty coefficients: $\lambda\_1 = \lambda\_3 = 0.3$ (Optuna), $\lambda\_2$ adaptive. - Node-to-node: cosine similarity loss. Edge-to-edge: Mahalanobis distance loss. - Logit matching via KL divergence. - Text node merging threshold $t$ is empirically determined (exact value not reported). - No details on optimizer, learning rate schedule, batch size, or number of training epochs. ### Data - PubLayNet, PRImA, HJDataset, DocLayNet: all publicly available. - Standard train/eval splits used (instance counts reported in Table 1). ### Evaluation - COCO-style mAP (IoU 0.5:0.05:0.95), plus AP@50, AP@75, AP\_S, AP\_M, AP\_L. - Comparisons against three KD baselines (ReviewKD, NKD, SimKD) and three supervised methods (DocSegTr, LayoutLMv3, SwinDocSegmenter). - No error bars, significance tests, or multi-run statistics reported. - The supervised baselines use different training setups (pre-training corpora, augmentations) so direct parameter-efficiency comparisons should be interpreted cautiously. ### Hardware - No training hardware, GPU hours, or memory requirements reported. - No inference latency or throughput measurements, which is a notable gap for a paper motivated by edge deployment. ## BibTeX ```bibtex @inproceedings{banerjee2024graphkd, title={GraphKD: Exploring Knowledge Distillation Towards Document Object Detection with Structured Graph Creation}, author={Banerjee, Ayan and Biswas, Sanket and Llad{\'o}s, Josep and Pal, Umapada}, booktitle={International Conference on Document Analysis and Recognition (ICDAR)}, pages={353--370}, year={2024}, publisher={Springer}, doi={10.1007/978-3-031-70543-4_21} } ```