# TRivia: Self-supervised Fine-tuning of VLMs for Table Recognition

> TRivia is a self-supervised fine-tuning framework that uses GRPO with table QA-based rewards to train VLMs on unlabeled table images, producing TRivia-3B, a compact open-source model that outperforms Gemini 2.5 Pro and MinerU2.5 on real-world table recognition benchmarks.

- Authors: Hunter Heidenreich
- Published: 2025-12-02
- Updated: 2026-03-26
- Paper: Preprint (arXiv) <https://arxiv.org/abs/2512.01248>
- Code: TRivia (code) <https://github.com/opendatalab/TRivia>
- Model: TRivia-3B (model weights) <https://huggingface.co/opendatalab/TRivia-3B>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## TL;DR

TRivia is a self-supervised fine-tuning framework that enables vision-language models (VLMs) to learn table recognition from unlabeled table images. It uses Group Relative Policy Optimization (GRPO) with table question-answering as a proxy reward, sidestepping the need for costly human annotations or proprietary model distillation. The resulting TRivia-3B, fine-tuned from Qwen2.5-VL-3B, outperforms Gemini 2.5 Pro and MinerU2.5 on three standard table recognition benchmarks.

## What kind of paper is this?

**Dominant:** $\Psi\_{\text{Method}}$: The core contribution is a self-supervised fine-tuning framework (TRivia) built around a training algorithm combining GRPO, response-consistency sampling, and attention-guided QA generation. The paper includes extensive ablations establishing which components drive performance.

**Secondary:** $\Psi\_{\text{Evaluation}}$: The paper evaluates across four benchmarks (PubTabNet, OmniDocBench, CC-OCR, OCRBench v2), compares against 12 baselines spanning three model categories, and includes a robustness analysis to QA noise and complex table types.

## What is the motivation?

Table recognition (TR), the task of converting table images into structured representations such as HTML or Markdown, is a core component of document parsing pipelines. Recent VLMs have pushed TR quality forward significantly, but the gap between open-source and proprietary systems remains large.

The root cause is data. Acquiring labeled TR data follows three established paradigms, each with a ceiling:

1. **Synthetic data** (rendered HTML tables) offers scale but misses real-world visual diversity.
2. **Human-annotated real-world data** captures complexity but is expensive and slow to produce.
3. **Distillation from proprietary models** (e.g., Gemini 2.5 Pro) is still costly, may violate service agreements, and hard-caps downstream performance at the teacher's level.

Even large-scale open-source efforts like MinerU2.5, trained on millions of samples with manual annotations and Gemini 2.5 Pro distillation, cannot exceed Gemini's own performance. This motivates a different question: can unlabeled table images from the wild, which are cheap to collect at scale, be used to train TR models that go beyond the limits of labeled data?

## What is the novelty?

TRivia introduces a closed-loop self-supervised fine-tuning pipeline built around three components:

**1. Table QA-driven GRPO**

Rather than predicting full HTML markup as a supervised target, TRivia treats TR as a reinforcement learning (RL) problem. The TR model acts as the policy and generates multiple recognition responses $o\_1, \ldots, o\_R$ for each table image $I$. A separate LLM $M\_{\text{LLM}}$ then answers table-specific questions based on each response, and the reward is defined as the average F1-score across the QA set:

$$\text{Reward}(o\_j) = \frac{1}{|QA|} \sum\_{(q,a) \in QA} \text{F1}(M\_{\text{LLM}}(q; o\_j),\, a)$$

GRPO optimizes relative reward differences across the response group rather than chasing absolute scores, which means imperfect QA pairs tend to cancel out during within-group normalization. To prevent destabilization from invalid (illegal) recognition outputs, which receive zero reward by default, TRivia filters those responses out before computing advantages.

**2. Response-consistency sampling**

Not all unlabeled images contribute equally to learning. TRivia selects samples that cause the model to produce diverse recognition outputs, since GRPO benefits most from samples where relative advantages are informative. For each image $I$, the TR model generates $K$ recognition outputs $\\{o\_i\\}\_{i=1}^K$, and the consistency score is defined as the average pairwise TEDS similarity:

$$\text{Consistency}(I) = \frac{2}{K^2 - K} \sum\_{1 \le i < j \le K} \text{TEDS}(o\_i, o\_j)$$

Images with lower consistency scores (more disagreement across responses) are prioritized for training. To keep the process practical, this sampling is done offline once rather than re-evaluated online. Images with consistency scores below 0.4 tend to be noisy or non-tabular and are discarded; the remaining images are uniformly sampled across score intervals from 0.4 to 1.0.

**3. Attention-guided diverse QA generation**

QA pairs serve as the supervisory signal, so they must cover different table regions and be verifiable. Naive single-pass prompting often covers only part of a table, while multi-pass prompting generates paraphrased duplicates from the same region. TRivia instead uses the VLM's own attention weights during answer generation to identify the visual source of each QA pair:

$$VS((q,a);\, I, M\_{\text{QA}}) = \\{v \mid A\_{M\_{\text{QA}}}(v \mid a) > \tau\_A,\; v \in \mathcal{V}\\}$$

Candidate QA pairs are then selected greedily so that no two selected pairs share significant visual overlap (IOU below a threshold $\tau\_{\text{IOU}}$). Before selection, each candidate is cross-checked by a separate VLM $M\_{\text{Val}}$ to verify that it can be answered correctly with the image but not without it.

## What experiments were performed?

**Benchmarks.** TRivia-3B is evaluated on four benchmarks:
- **PubTabNet**: academic papers with digital tables.
- **OmniDocBench v1.5**: 512 samples from digital PDFs with diverse table types.
- **CC-OCR**: 300 scanned/photographed images, including handwritten, long, and complex tables.
- **OCRBench v2 (table parsing subset)**: 700 images with varied real-world layouts.

The primary metric is TEDS (Tree Edit Distance-based Similarity), which captures both content accuracy and structure. S-TEDS is a structure-only variant.

**Baselines.** The paper compares against 12 models in three categories:
- Expert TR models: SLANNet-plus, UniTable.
- General-purpose VLMs: InternVL3.5-241B, Qwen2.5-VL-72B, Qwen3-VL-235B, Gemini 2.5 Pro, GPT-4o, GPT-5.
- Document-parsing VLMs: dots.ocr, DeepSeek-OCR, PaddleOCR-VL, MinerU2.5.

**Ablations.** The paper ablates each of the three main TRivia components:
- Removing attention-guided QA generation significantly drops performance on structurally complex tables.
- Replacing response-consistency sampling with random sampling slows convergence and reduces final TEDS from 63.5 to 52.0 on the analysis set.
- Removing illegal-sample filtering destabilizes GRPO rewards, increasing convergence steps by approximately 25% and reducing final TEDS by 3 points.

The paper also compares the QA-based approach directly against distillation alternatives: using Qwen2.5-VL-72B pseudo-labels for SFT decreases TEDS by 8.37 on average, and GRPO with those same pseudo-labels still lags by 4.92 TEDS compared to TRivia.

## What are the outcomes/conclusions?

TRivia-3B achieves 89.88 average TEDS across OmniDocBench, CC-OCR, and OCRBench v2, compared to 88.93 for Gemini 2.5 Pro and 86.82 for MinerU2.5. It is the top-performing model overall, with the only gap being on CC-OCR where Gemini 2.5 Pro scores 85.56 versus 84.90 for TRivia-3B.

Several secondary findings are notable:

- TRivia-3B annotations can be used to distill into a smaller Stage-2 model via standard SFT with nearly no performance loss (89.99 vs 89.88 overall TEDS), which contrasts sharply with the degraded results from distilling Qwen2.5-VL-72B pseudo-labels.
- The improvements are most pronounced on challenging table types: rotated tables gain +21.76 TEDS over the base Stage-2 model, with large gains on large tables and borderless merged-cell tables as well.
- TRivia is robust to moderate QA noise: injecting 20% incorrect QA pairs causes only limited degradation, explained by the relative reward normalization in GRPO canceling out uniformly wrong signals.
- A preliminary multi-task study shows TRivia is compatible with key information extraction (KIE), suggesting the approach may generalize beyond TR.

**Limitations.** The pipeline relies on large auxiliary models (Qwen2.5-VL-72B as $M\_{\text{QA}}$ and InternVL3-78B as $M\_{\text{Val}}$), which imposes significant compute cost during data preparation even though TRivia-3B itself is compact. The method is validated only on table recognition and one KIE task; broader applicability to other document understanding tasks remains to be demonstrated.

## Reproducibility

### Models

- **TRivia-3B**: fine-tuned from Qwen2.5-VL-3B-Instruct. 3 billion parameters. Model weights are released on HuggingFace at [opendatalab/TRivia-3B](https://huggingface.co/opendatalab/TRivia-3B) (Apache-2.0). Training code is at [opendatalab/TRivia](https://github.com/opendatalab/TRivia) (Apache-2.0). A Huggingface demo is available at [opendatalab/TRivia-3B (Space)](https://huggingface.co/spaces/opendatalab/TRivia-3B).
- Visual encoder: supports image resolutions from $256 \times 28 \times 28$ to $1280 \times 28 \times 28$, yielding 256 to 1280 visual tokens per image.

### Algorithms

Training proceeds in three stages:

| Stage | Data | Trainable params | Samples | Batch | Epochs |
|---|---|---|---|---|---|
| 1 (OTSL warm-up) | Synthetic (PubTabNet, SynthTabNet, MMTab) | LLM only | 700K | 32 | 1 |
| 2 (SFT) | Real-world | All | 50K | 32 | 2 |
| 3 (TRivia/GRPO) | TRivia curated unlabeled | All | 50K | 128 | 1 |

Stage 3 hyperparameters: learning rate $2 \times 10^{-7}$ (ViT), $1 \times 10^{-6}$ (MLP + LM); GRPO group size $G = 16$; sampling temperature 1.2; sequence length 8192; constant LR scheduler.

- QA generation: Qwen2.5-VL-72B-Instruct as $M\_{\text{QA}}$, attention layer 72, $\tau\_A = 0.01$, $\tau\_{\text{IOU}} = 0.3$.
- Validity cross-checking: InternVL3-78B as $M\_{\text{Val}}$.
- Answer LLM: Qwen3-8B as $M\_{\text{LLM}}$ during GRPO.

### Data

- **Stage 1**: PubTabNet (200K), SynthTabNet (4 subsets, 100K each), MMTab (100K), converted to OTSL format.
- **Stage 2**: approximately 50K real-world samples from open-source datasets and web sources.
- **Stage 3**: 100K unlabeled PDF table images collected from web; after response-consistency filtering, 50K selected; QA generation yields a final dataset of 48,470 images with an average of 28.3 QA pairs each.
- Dataset construction details and prompt templates are provided in the supplementary appendix of the paper.

### Evaluation

- **Metrics**: TEDS (full) and S-TEDS (structure only), both computed via tree edit distance against reference HTML.
- **Benchmarks**: OmniDocBench v1.5 (512 samples), CC-OCR (300 samples), OCRBench v2 table subset (700 samples), PubTabNet, FinTabNet.
- **Inference**: vLLM used for compatible models; temperature 0.2 for general-purpose VLMs to reduce repetitions; Gemini 2.5 Pro evaluated with thinking mode enabled.
- The authors acknowledge that PubTabNet and FinTabNet in-domain performance may drop after Stage 2 real-world adaptation, as those benchmarks are only present in Stage 1 training.

### Hardware

- All experiments run on 8x A100 80GB GPUs.
- Stage 1: approximately 1 day. Stage 2: approximately 2 hours. Stage 3 (GRPO): approximately 2 days.
- TRivia-3B is designed for offline deployment and can run on hardware that fits a 3B parameter VLM.

## BibTeX

```bibtex
@article{zhang2025trivia,
  title={TRivia: Self-supervised Fine-tuning of Vision-Language Models for Table Recognition},
  author={Junyuan Zhang and Bin Wang and Qintong Zhang and Fan Wu and Zichen Wen and Jialin Lu and Junjie Shan and Ziqi Zhao and Shuya Yang and Ziling Wang and Ziyang Miao and Huaping Zhong and Yuhang Zang and Xiaoyi Dong and Ka-Ho Chow and Conghui He},
  journal={arXiv preprint arXiv:2512.01248},
  year={2025}
}
```