# NCGM: Neural Collaborative Graph Machines for Table Structure Recognition

> NCGM frames table structure recognition as a multimodal graph problem, using stacked Ego Context Extractors and Cross Context Synthesizers to enable adaptive intra- and inter-modality collaboration across geometry, appearance, and content features.

- Authors: Hunter Heidenreich
- Published: 2026-03-24
- Paper: Preprint (arXiv) <https://arxiv.org/abs/2111.13359>
- Paper: Published (CVPR 2022) <https://openaccess.thecvf.com/content/CVPR2022/html/Liu_Neural_Collaborative_Graph_Machines_for_Table_Structure_Recognition_CVPR_2022_paper.html>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## TL;DR

NCGM introduces a graph-based architecture for table structure recognition that treats three modalities (geometry, appearance, and content) as separate graphs, then alternates between per-modality context extraction and cross-modality information synthesis in stacked collaborative blocks. The approach handles distorted tables considerably better than earlier graph methods that rely on early or late fusion.

## What kind of paper is this?

**Dominant:** $\Psi\_{\text{Method}}$ (new architecture). The headline contribution is a stack of collaborative blocks composed of Ego Context Extractors (ECE) and Cross Context Synthesizers (CCS). The paper is organized around ablations, comparisons on multiple benchmarks, and an analysis of learned attention diversity.

## What is the motivation?

Table structure recognition (TSR) aims to recover the logical or physical grid structure of a table from an image, producing row/column/cell adjacency relationships. Two families of prior approaches fall short on tables with complex spanning cells or geometric distortion:

1. **Early fusion** concatenates multiple modality features before feeding them to a single graph model. This forces the model to learn a single inductive bias and ignores the fact that different modalities may be more or less informative depending on table type.
2. **Late fusion** models each modality in a separate graph and combines the outputs. This disentanglement sacrifices inter-modality interactions entirely.

The authors frame the unaddressed challenge as the "Heterogeneous TSR (Hetero-TSR)" problem: how should different modalities collaborate with each other in a way that adapts to the table at hand?

## What is the novelty?

The paper introduces Neural Collaborative Graph Machines (NCGM), built from a stack of $L$ collaborative blocks. Each block has two successive modules.

**Ego Context Extractor (ECE):** For each modality $\sim \in \\{G, A, C\\}$, a fully-connected directed graph is built over the $N$ text segment bounding boxes. The edge feature for a pair $(i, j)$ is the asymmetric function:

$$h\_e(x\_i, x\_j) = x\_i \,\|\, (x\_i - x\_j)$$

ECE aggregates global context via a Compressed Multi-head Attention (CMHA) module. Standard MHA over all $N(N-1)/2$ edge features would be $O(NM)$ in complexity. CMHA reduces this by compressing keys and values through a reshape-and-project operation:

$$MC(H) = \text{Norm}(\text{Reshape}(H, \varepsilon)\, W^h)$$

with compression ratio $\varepsilon = N/M$, cutting complexity to $O(N^2)$. ECE outputs updated per-modality context embeddings $C^{(l)} \in \\{C^G, C^A, C^C\\}$.

**Cross Context Synthesizer (CCS):** Three parallel CMHA modules each take one modality as queries and the other two as keys/values, producing inter-modality embeddings $M^{(l)}$. For example, for the content branch:

$$M\_C^{(l)} = \text{CMHA}(M\_C^{(l-1)},\; C\_A^{(l)} \cup C\_G^{(l)})$$

After $L=3$ blocks the three inter-modality streams are concatenated into collaborative graph embeddings $E \in \mathbb{R}^{N \times d\_e}$. All $N^2$ pair vectors $U = \\{u\_{i,j}\\}$ are formed by channel-wise concatenation of $e\_i$ and $e\_j$ and fed to three separate FC classifiers predicting binary same-row, same-column, and same-cell relationships.

**Training loss:** End-to-end with a combination of classification and contrastive terms for each relationship type:

$$L = L\_{cell} + L\_{col} + L\_{row}$$

$$\tilde{L} = \lambda\_1 L\_{\text{class}} + \lambda\_2 L\_{\text{con}}$$

$$L\_{con} = \|e\_{(a)} - e\_{(b)}^+\|\_2^2 + \max\\{0,\, \alpha - \|e\_{(a)} - e\_{(b)}^-\|\_2^2\\}$$

During training, Monte Carlo sampling (sample size $S=10$) is used to keep memory costs tractable. At inference, all pairs are evaluated.

**Augmented evaluation set:** The authors also release a synthesizing procedure (SciTSR-COMP-A) that applies perspective transformation and quadratic Bézier curve distortion to SciTSR-COMP images, generating a harder variant for benchmarking under realistic capture conditions.

## What experiments were performed?

**Datasets:**

- *Physical structure* (precision/recall/F1 on row/column adjacency): ICDAR-2013-P, ICDAR-2019, UNLV-P, WTW, SciTSR, SciTSR-COMP, and the new SciTSR-COMP-A.
- *Logical structure* (BLEU on TableBank; TEDS on PubTabNet): TableBank (145k train) and PubTabNet (339k train).

**Evaluation setups** follow TabStruct-Net:

- **Setup-A**: table image only (no bounding boxes or text content as input; FLAG-Net detection boxes and Tesseract OCR results are used for fair comparison).
- **Setup-B**: table image plus ground-truth (or OCR-derived) text segment bounding boxes and content.

**Baselines:** GraphTSR, DGCNN, TabStruct-Net, FLAG-Net, GTE, LGPMA, Cycle-CenterNet.

**Ablations** isolate: (a) early vs. late vs. collaborative fusion, (b) DGCNN vs. Transformer vs. ECE as the intra-modality aggregator, (c) concatenation vs. CCS as the inter-modality fusion strategy, (d) individual modality contributions (G, A, C alone and in pairs), (e) effect of block depth (1 to 9 blocks).

## What are the outcomes/conclusions?

On SciTSR-COMP (complex structures), NCGM under Setup-B achieves F1 99.0% (P 98.8%, R 99.3%), outperforming FLAG-Net. The strongest result is on the distorted set SciTSR-COMP-A: without distorted training data, NCGM exceeds the second-best FLAG-Net by approximately 11 percentage points (F1) under Setup-A and roughly 12 points under Setup-B. When trained with distorted data, the margin remains around 7 to 9 points.

**Ablation insights:**

- Geometry (G) is the dominant modality for physical structure; appearance (A) contributes meaningfully; content (C) alone is weak (F1 50.2% on SciTSR-COMP).
- ECE with individual modality inputs (99.0 F1) beats all early-fusion and late-fusion alternatives on SciTSR-COMP.
- CCS raises F1 from 98.3% (ECE + late concatenation) to 99.0%.
- Three collaborative blocks is the practical optimum; deeper networks converge slower and become prone to training collapse beyond 7 blocks.

**Limitations the authors acknowledge:**

1. Computational cost grows with the number of modalities and decoupled processing (3.1M parameters, 12.7G FLOPs for 42 bounding boxes, which is heavier than FLAG-Net at 1.9M / 3.3G FLOPs).
2. Deeper collaborative blocks risk training collapse (observed for more than 7 blocks beyond 50 epochs).
3. Nested tables, where row/column boundaries are ambiguous, remain a failure mode.

The paper does not discuss inference latency beyond FLOPs counts, nor does it report results on FinTabNet or the PubTables-1M benchmark family that has since become standard.

## Reproducibility

| Resource | Type | License | Link |
|---|---|---|---|
| Preprint (arXiv) | Paper | arXiv-nonexclusive-distrib-1.0 | [arxiv.org/abs/2111.13359](https://arxiv.org/abs/2111.13359) |
| Published (CVPR 2022) | Paper | Open Access (CVF) | [CVF Open Access](https://openaccess.thecvf.com/content/CVPR2022/html/Liu_Neural_Collaborative_Graph_Machines_for_Table_Structure_Recognition_CVPR_2022_paper.html) |
| Code | Code | Not released | N/A |
| Pretrained weights | Model | Not released | N/A |

### Models

- Backbone: ResNet-18 (conv1 to conv2\_2) plus three 3$\times$3$\times$64 conv layers for appearance features; RoI Align for per-box pooling.
- Geometry: 4-dim normalized bounding box coordinates projected by a $d$-dim FC layer.
- Content: word2vec embeddings followed by a 7$\times$1$\times$d conv for sequential modeling.
- Hidden size $d=64$; 8 attention heads; $d\_k = d\_v = 8$; $d\_m = 64$.
- 3 collaborative blocks (L=3); FC classifiers have 3 layers at 256 dimensions plus a 2-dim softmax head.
- Total: 3.1M parameters. No pretrained weights or code released.

### Algorithms

- Framework: PyTorch.
- Input table images resized to $512 \times 512$.
- Pre-trained on SciTSR for 10 epochs; fine-tuned on each target benchmark for 50 epochs.
- Optimizer: not explicitly named; learning rate initialized at $1 \times 10^{-4}$, divided by 10 on loss plateau.
- Loss weights: $\lambda\_1 = \lambda\_2 = 1$; margin $\alpha = 1$ in contrastive loss.
- Monte Carlo sampling size $S=10$ during training; full pairing at inference.
- No data augmentation beyond the optional SciTSR-COMP-A distortion procedure (perspective transform + Bézier curve warp).

### Data

- Training: SciTSR (12k), WTW (10.97k), ICDAR-2019 (600), PubTabNet (339k), TableBank (145k), and partial splits for ICDAR-2013 and UNLV (80/20 random 5-fold).
- Bounding box alignment: cell-level datasets (ICDAR-2019, UNLV, WTW) are converted to text-segment level via Tesseract OCR; text-segment-level datasets (ICDAR-2013, SciTSR) use parsed GT boxes directly.
- SciTSR-COMP-A synthesis procedure is described in the appendix but the augmented images are not separately distributed.

### Evaluation

- Physical structure: precision, recall, and F1 on pairwise adjacency (same-row, same-column, same-cell).
- Logical structure: BLEU (TableBank, following Li et al.) and TEDS (PubTabNet, following Zheng et al.).
- ICDAR-2013 and UNLV: averaged over 10 random 5-fold splits.
- No error bars or significance tests reported for main results. No seed sensitivity study.
- Baselines use different input formats in some cases (flag-net detection vs. GT boxes), creating potential inconsistencies at the dataset level.

### Hardware

- Training: one Nvidia Tesla V100 GPU with 32 GB memory.
- No wall-clock training time reported.
- FLOPs: 12.7G for a table with 42 text segment boxes; parameters: 3.1M.
- No inference latency or throughput numbers reported.

## BibTeX

```bibtex
@inproceedings{liu2022neural,
  title={Neural Collaborative Graph Machines for Table Structure Recognition},
  author={Hao Liu and Xin Li and Bing Liu and Deqiang Jiang and Yinsong Liu and Bo Ren},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2022}
}
```