# OG-HFYOLO: Orientation Gradient Guidance and Heterogeneous Feature Fusion for Deformation Table Cell Instance Segmentation > OG-HFYOLO is a YOLO-based instance segmentation model targeting pixel-level cell localization in deformed and warped table images, accompanied by the DWTAL dataset built from a synthetic deformation generator. - Authors: Hunter Heidenreich - Published: 2025-04-29 - Updated: 2026-03-30 - Paper: Preprint (arXiv) - Code: OG-HFYOLO Code - Dataset: DWTAL Dataset - Source: Bevaya Labs, --- ## TL;DR OG-HFYOLO extends YOLOv5-based instance segmentation with three new modules (a gradient orientation extractor, a heterogeneous kernel cross-fusion block, and a scale-aware loss) plus mask-driven NMS, targeting the problem of accurately localizing cells in physically deformed table images. The authors also construct DWTAL, a large-scale synthetically augmented dataset of deformed wired tables with pixel-level segmentation annotations, using a custom wave and cylindrical warping generator applied to existing datasets. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Method}}$ The paper's primary contribution is a new model architecture with several interacting modules. The bulk of the content covers architectural design choices, loss function derivations, and ablation experiments validating each component. **Secondary:** $\Psi\_{\text{Resource}}$ The paper also releases DWTAL, a dataset of 28,285 images across two difficulty tiers (DWTAL-s and DWTAL-l) with instance segmentation annotations for deformed table cells. This is a meaningful contribution, since no prior public dataset offered pixel-level annotations for this task. ## What is the motivation? Physical deformations in photographed or scanned tables (bending, perspective distortion, page curl from binding) disrupt the alignment between cell content and structure. Downstream tasks such as content extraction and OCR depend on accurate spatial coordinate localization of individual cells, but most existing table structure recognition methods assume near-planar inputs. Prior approaches to cell coordinate localization fall into two categories. Contour-based detection methods lose critical information when deformations are severe. Text box segmentation methods handle non-deformed wired tables but struggle with geometric distortion. Keypoint detection partially addresses the problem but corner regression accuracy degrades under heavy curvature. Instance segmentation offers pixel-level boundary descriptions that can represent irregular, curved cell shapes, but two practical barriers remained: (1) existing datasets lacked the fine-grained segmentation annotations needed to train such a model, and (2) instance segmentation methods had not been adapted to the specific challenges of deformed table cells: dense object arrangements, extreme aspect ratio variation from merged cells, and overlapping bounding boxes from complex cell shapes. ## What is the novelty? The paper introduces four technical contributions that work in concert: **Gradient Orientation-aware Extractor (GOE).** Placed after the first downsampling convolution in the backbone, GOE applies separate horizontal and vertical decoupling operators $G\_x$ and $G\_y$ (initialized as edge operators, then trained freely) to the input feature map. The gradient magnitude and gradient direction are computed as: $$I\_{GM} = \sqrt{G\_x^2 + G\_y^2}$$ $$I\_{GD} = \text{Cat}(G\_x, G\_y)$$ An orientation attention mechanism $\Phi$, inspired by HOG's orientation binning, initializes its $1 \times 1$ convolutional kernels from polar directional basis vectors: $$\mathbf{e}\_i = \begin{bmatrix} \cos \theta\_i \\ \sin \theta\_i \end{bmatrix}, \quad \theta\_i = \frac{i\pi}{n}$$ These are assembled into a weight matrix $W \in \mathbb{R}^{n \times 2 \times 1 \times 1}$ mapping gradient components to $n$ orientation channels. The final output combines both cues: $$I\_o = \text{IN}(\text{Softmax}(\Phi(I\_{GD}))) \odot I\_{GM}$$ where IN denotes instance normalization. This gives the network a geometric prior about edge directionality that standard convolutions lack. **Heterogeneous Kernel Cross Fusion (HKCF).** Inserted after skip connections in the FPN-PAN neck, HKCF addresses the extreme width and height variation of merged table cells. It uses a bottleneck to reduce channel count, then runs parallel $1 \times k$ and $k \times 1$ asymmetric convolutions (horizontal and vertical cross-convolutions) whose outputs are summed: $$HXConv^{(k)}(I) = HConv^{(k)}(I) + VConv^{(k)}(I)$$ A Channel Attention Bridge (CAB) before the cross-convolutions compensates for information loss from dimensionality reduction. Following the Heterogeneous Kernel Select Protocol from YOLO-MS, kernel sizes 3, 5, and 7 are applied at progressively deeper fusion stages. **Scale-aware loss function.** The standard YOLO mask normalization divides by instance area $A$, but $1/A$ has a second derivative $2/A^3$ that causes near-vertical gradient growth for very small cells. The authors introduce a composite weighting term $W\_s = 1 + \log(1/A)$, yielding: $$L\_{\text{scale-aware}} = \frac{1}{n} \sum\_{i=1}^{n} \left[ \left(1 + \log \frac{1}{A\_i}\right) \frac{1}{A\_i} \sum\_{(x,y) \in \text{crop}(\Omega)} \frac{1}{H'\_i W'\_i} L\_{\text{base}}(x,y) \right]$$ where $L\_{\text{base}} = L\_{\text{BCE}} + L\_{\text{Dice}}$. The $\log(1/A)$ term moderates the curvature of compensation, so gradient magnitudes scale more smoothly across the object size distribution. **Mask-driven NMS.** Standard bounding-box IoU NMS incorrectly suppresses valid cells when deformed bounding boxes substantially overlap. The proposed replacement computes IoU directly on predicted binary masks: $$\text{Mask\_IoU} = \frac{|M\_i \cap M\_j|}{|M\_i \cup M\_j|}$$ suppressing a lower-confidence mask only when this pixel-level overlap exceeds the threshold. **DWTAL dataset.** The authors construct a data generator applying wave warping (sine/cosine-based, amplitude $A \in [10, 50]$) and cylindrical warping (cosine-based along the y-axis) to mildly deformed source images from TAL-OCR and WTW, with an illumination adjustment stage. The result is two datasets: DWTAL-s (8,765 images, simpler backgrounds from TAL-OCR) and DWTAL-l (19,520 images, more complex backgrounds from WTW), both with pixel-level instance segmentation annotations. ## What experiments were performed? All experiments used a single NVIDIA RTX 3090 (24 GB). For ablation studies, no pretrained weights were used. For comparison experiments against non-YOLO models, ResNet-101 backbones were initialized from ImageNet-pretrained MSRA weights via MMDetection. Training settings: SGD optimizer, momentum 0.9, learning rate 0.001, weight decay 0.0005, $640 \times 640$ input, batch size 2 (batch size 1 for DWTAL-l comparisons), 200 epochs (ablation) and 100 epochs (comparisons). **Baselines** spanned two-stage models (Mask R-CNN, Cascade Mask R-CNN), single-stage models (SOLOv2, YOLACT), Transformer-based segmentation (Mask2Former), and YOLO variants (YOLOv5l-seg, YOLOv8l-seg, YOLOv11l-seg). **Evaluation metrics:** mask mAP@50, mask mAP@50:95, bounding box mAP@50, bounding box mAP@50:95, parameter count, and GFLOPs. **Ablation studies** isolated each of the four proposed components, examining individual and joint contributions on DWTAL-s. A separate backbone ablation compared the YOLOv5 C3 baseline against substituting YOLOv11's C3k2 and C2PSA components. An anchor mechanism ablation confirmed that anchor-based detection substantially outperforms anchor-free variants on this task. ## What are the outcomes/conclusions? On DWTAL-s, the full OG-HFYOLO model achieves mask mAP@50:95 of 74.23%, compared to 71.96% for YOLOv5l-seg, 64.4% for Mask2Former, 62.5% for Mask R-CNN, 57.5% for YOLOv8l-seg, and 57.8% for YOLOv11l-seg. On DWTAL-l, OG-HFYOLO reaches mask mAP@50:95 of 62.38% versus 61.34% for YOLOv5l-seg (the next-best on this metric) and 44.7% for Mask R-CNN. The ablation results indicate that each module delivers modest gains in isolation (GOE alone: +0.44% mask mAP@50:95; HKCF alone: +0.09%; scale-aware loss alone: +0.48%) but their joint use yields substantially larger improvements. Mask-driven NMS provides an additional +0.94% increase when added on top of the three structural modules, and is identified as the single component with the largest individual contribution in the combined setting. The authors note that the challenges in this dataset are interdependent, so the modules benefit from each other. The backbone ablation found that the simpler YOLOv5 C3 architecture outperforms YOLOv11-style components within the same training setup, which the authors take as evidence that the proposed modules are more important than the backbone choice. Anchor-based detection shows a greater than 10% absolute improvement over anchor-free variants on both datasets, suggesting that the dense, regular structure of table cell arrangements favors anchor-based priors. Qualitative results on natural scene photographs and the CamCap dataset (using a model trained exclusively on DWTAL-l) suggest some cross-domain generalization, though the evaluation is limited to visual examples rather than held-out metrics. **Limitations.** The datasets are entirely derived from two source datasets (TAL-OCR and WTW), so the diversity of deformation types and backgrounds is constrained by those sources. All images contain only a single table instance. Downstream logical coordinate recovery is left as future work. Statistical reporting (standard deviations, multiple runs) is absent. The model's parameter count (125.39 M) is higher than several baselines, which may create deployment constraints. ## Reproducibility ### Models OG-HFYOLO is built on a modified YOLOv5 backbone (CSPDarknet with C3 modules) augmented by GOE after the first downsampling layer. The neck uses FPN-PAN with HKCF blocks replacing standard skip-connection convolutions. The detection head retains YOLOv5's anchor-based design. Total parameters: 125.39 M. GFLOPs: 170 (at $640 \times 640$). Code is released under AGPL-3.0 at [https://github.com/justliulong/OGHFYOLO](https://github.com/justliulong/OGHFYOLO). The model architecture is defined in `cfg/models/segment/og-hfyolo.yaml`. A demonstration checkpoint (`best.pt`) is available via GitHub releases (v1.0.2). No fully reproduced checkpoint from the paper's complete training runs is separately documented; the released weight is a demonstration model. Environment setup is provided via `requirements.txt` or `environment.yaml` (Python >= 3.8, PyTorch >= 1.8). ### Algorithms - Optimizer: SGD, momentum 0.9 - Learning rate: 0.001 (schedule not specified) - Weight decay: 0.0005 (ablation), 0.0001 (comparison experiments) - Batch size: 2 for DWTAL-s; 1 for DWTAL-l - Training epochs: 200 (ablation), 100 (comparison) - Input resolution: $640 \times 640$ - Bounding box regression: EIoU loss replacing CIoU - Mask loss: $L\_{\text{base}} = L\_{\text{BCE}} + L\_{\text{Dice}}$, scaled by $(1 + \log(1/A)) \cdot (1/A)$ - Post-processing: Mask IoU NMS replacing bounding-box IoU NMS - GOE orientation bins: $n$ (value not stated explicitly in the paper) - HKCF kernel sizes: 3, 5, 7 at three FPN levels ### Data - DWTAL-s: 8,765 images total; 7,012 train / 1,753 test. Primarily derived from TAL-OCR with 150 collected images. Simpler backgrounds. - DWTAL-l: 19,520 images total; 15,616 train / 3,904 test. Primarily derived from WTW. More complex backgrounds. - 80/20 train/test split, stratified by deformation type. - Labels are pixel-level instance segmentation masks per table cell. - A logical coordinate annotation version has also been released (via GitHub releases, without images; images must be downloaded separately). - DWTAL is distributed in two formats. YOLO format is hosted on Google Drive (separate links for DWTAL-s and DWTAL-l). COCO format is hosted on Hugging Face at `justliulong/DWTAL` and can be downloaded via the `datasets` library. The data generator code lives in the GitHub repo at `dataprocess/data_gen.py`. Labelme-format annotation archives are also available on Google Drive. - The underlying source datasets (TAL-OCR and WTW) have their own licenses; the paper does not state an explicit license for DWTAL beyond referencing the AGPL-3.0 repo and describing it as open-source. ### Evaluation - Primary metrics: mask mAP@50 and mAP@50:95 (COCO-style AP over 101 recall points) - Secondary metrics: bounding box mAP@50 and mAP@50:95 - Non-YOLO baselines: ResNet-101 backbone, pretrained on ImageNet, fine-tuned 100 epochs via MMDetection - No error bars, significance tests, or repeated runs are reported - Comparison against Mask2Former uses ResNet-101 backbone for consistency with other non-YOLO models; a ViT-based Mask2Former is not evaluated - CamCap generalization evaluation is qualitative only ### Hardware - Training hardware: NVIDIA RTX 3090, 24 GB VRAM - Framework: Python 3.8.19, PyTorch 1.13.0, CUDA 12.4 - Training time and energy consumption are not reported - Inference speed is discussed qualitatively (OG-HFYOLO described as retaining single-stage speed) but no concrete latency numbers are given ## BibTeX ```bibtex @article{liu2025oghfyolo, title={OG-HFYOLO: Orientation Gradient Guidance and Heterogeneous Feature Fusion for Deformation Table Cell Instance Segmentation}, author={Long Liu and Cihui Yang}, journal={arXiv preprint arXiv:2504.20682}, year={2025} } ```