# Fox: Focus Anywhere for Fine-grained Multi-page Document Understanding > Research notes on Fox, a multi-page document understanding pipeline that uses dual vision vocabularies and position-aware prompts for region-level OCR, translation, and summarization. - Authors: Hunter Heidenreich - Published: 2026-03-14 - Paper: Preprint (arXiv) - Code: Fox (Evaluation Scripts) - Benchmark: Fox Benchmark - Project: Fox Project Page - Source: Bevaya Labs, --- ## TL;DR Fox is an LVLM pipeline that combines dual vision vocabularies (CLIP-style and SAM-style) with position-aware prompts to enable fine-grained, region-level document understanding on single and multi-page inputs. The authors introduce cross-vocabulary hybrid data synthesis to break specific-vocabulary bias, and a 9-sub-task benchmark for evaluating fine-grained document parsing. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Method}}$: The headline contribution is a new multi-vocabulary LVLM pipeline and data synthesis strategy for fine-grained document understanding. Most of the paper describes the architecture, data generation, and the position-aware prompting scheme. **Secondary:** $\Psi\_{\text{Resource}}$: The paper introduces a bilingual benchmark with 9 fine-grained sub-tasks (region OCR, color-guided OCR, multi-page OCR, cross-page VQA, etc.), though its scale is modest (112 English + 100 Chinese pages). ## What is the motivation? Existing LVLMs for document understanding face several limitations: 1. **CLIP-style vocabularies lose dense text.** Patch-based models (UReader, TextMonkey, LLaVA-NeXT, InternVL-V1.5) decompose large document pages into many small patches, each independently encoded. CLIP's contrastive pre-training compresses sparse visual features, preventing lossless recovery of dense document content. 2. **Multi-vocabulary models suffer from vocabulary bias.** Models like Vary stack a SAM-style document vocabulary alongside CLIP but tend to activate only one branch depending on the input type, leading to incomplete utilization of available visual knowledge. 3. **No fine-grained region interaction.** Prior document LVLMs operate at the page level. Users cannot point to a specific region (paragraph, line, figure) and request OCR, translation, or summarization of just that area. 4. **Multi-page context is limited.** High token counts per page (e.g., InternVL-V1.5 produces 3,328 tokens per page) make multi-page input impractical. ## What is the novelty? Fox introduces three key ideas: ### 1. Position-aware prompts for "focus anywhere" Users specify regions of interest via three prompt types: click points, dragged bounding boxes, and color-coded boxes. These are normalized to spatial coordinates and combined with language instructions. Full-page OCR is reframed as "foreground focus" by providing the full page bounding box as input. ### 2. Cross-vocabulary data synthesis To break the vocabulary-specific bias (where only one vision encoder activates per input), the authors synthesize hybrid training data: - **Figure-text interleaved data:** Natural images are rendered onto document pages at random positions. Occluded text boxes are masked. This produces samples requiring both CLIP (for natural image understanding) and Vary-tiny (for text recognition). - **Color-text hybrid data:** Three random text boxes are painted in red, blue, and green. CLIP handles color recognition while Vary-tiny handles character recognition, forcing cross-vocabulary collaboration. ### 3. Efficient multi-page extension Each 1024$\times$1024 page is compressed to 256 image tokens by the SAM-style branch. With dual vocabularies, each page produces $v\_i \in \mathbb{R}^{256 \times 2048}$ after concatenation. All pages' tokens are unified into a single sequence: $$Q = \mathcal{LLM}(\\{v\_i'\\}\_{i=1}^N, (L^{\text{instruct}}, \Psi(\\{r\_i\\}\_{i=1}^N)))$$ where $\Psi(\cdot)$ normalizes spatial coordinates and $N$ is the number of input pages (up to 8 in experiments). Vision vocabulary weights are frozen during multi-page fine-tuning. ### Training objective Standard causal language modeling: $$\mathcal{L}\_t = -\mathbb{E}\_{(Q,V) \sim D} \log P\_\theta(q\_m | q\_{100B), Qwen-VL-Max (>100B). ### Key results **Dense English page OCR:** Fox (1.8B params) achieves edit distance 0.046, outperforming all baselines except Qwen-VL-Max (>100B) on F1/BLEU/METEOR. Fox beats Vary-toy (same param count) by 2.8% F1. **Dense Chinese page OCR:** Fox achieves edit distance 0.061 and leads on most metrics, including against Qwen-VL-Max. **Region focus tasks:** Region-level OCR yields F1 0.957 (English) and 0.955 (Chinese). Color-guided OCR achieves F1 0.940 (English) and 0.884 (Chinese), confirming cross-vocabulary activation. **Multi-page (8 pages):** Multi-region OCR F1 0.946; cross-page VQA accuracy 0.827. **Translation/summary:** METEOR 0.366 for region translation, ROUGE-L-F 0.282 for summary. The authors attribute modest scores to the small 1.8B language model. ## What are the outcomes/conclusions? Fox demonstrates that dual vision vocabularies can be effectively combined through synthetic cross-vocabulary data, without modifying the pre-trained vocabulary weights. The position-aware prompting scheme enables fine-grained, format-agnostic interactions (robust to single-column, multi-column, and interleaved layouts) on up to 8-page documents. **Limitations acknowledged by the authors:** - The CLIP branch is low-resolution, limiting natural image understanding quality within documents. - The language model (Qwen-1.8B) is small, constraining translation and summarization performance. **Limitations not acknowledged:** - The benchmark is very small (212 pages total) and not independently reproducible since it is manually collected. - No comparison with other fine-grained or region-level document understanding methods (there were few at the time of writing). - Multi-page evaluation uses only 8 pages; scaling behavior beyond this is unknown. - No evaluation on standard document VQA benchmarks (DocVQA, InfoVQA, etc.) that would allow direct comparison with the broader literature. - Training data includes GPT-3.5-generated annotations (translations, summaries), introducing potential quality and reproducibility concerns. ## Reproducibility ### Models - **Architecture:** Dual-encoder (CLIP-ViT-224px + Vary-tiny SAM-style ViT at 1024$\times$1024), embedding linear layers, Qwen-1.8B LLM. - **Parameter count:** 1.8B (language model). Vision vocabulary sizes not specified separately. - **Frozen components:** Both vision encoders are frozen throughout training. Only the embedding linear layers and LLM are trained. - **Model weights:** Not released. Reproducing results requires full retraining on 48$\times$ A800 GPUs. ### Algorithms - **Optimizer:** AdamW - **Learning rate:** 1e-4 (pre-training), 2e-5 (SFT) - **Scheduler:** Cosine annealing - **Batch size:** 4 per device - **Epochs:** 1 (both stages) - **Input resolution:** 1024$\times$1024 (SAM-style branch), 224$\times$224 (CLIP branch) - **Image rendering hyperparameters:** Scaling ratios for figure-text interleaved synthesis: $\alpha=0.3$, $\beta=0.9$, $\gamma=0.4$, $\eta=0.9$. - **Warmup, gradient clipping, mixed precision:** Not reported. - **Pre-training data:** ~10M samples total across region-level tasks (foreground OCR 1M, region OCR 1M, line OCR 600K, color OCR 1M, translation 500K, summary 500K), cross-vocabulary tasks (~1.6M), multi-page tasks (800K), plus NLP/caption data. - **SFT data:** 10K samples per task type, prompts rewritten 10$\times$ via GPT-3.5, plus LLaVA-80K. - **Training code:** Not released. The GitHub repository contains only evaluation scripts and benchmark data. ### Data - **Training data:** PDF pages from e-books, CC-MAIN, and arXiv, parsed with Python. Natural images from BLIP558K, Laion-COCO, RegionChat. Layout annotations from PubLayNet (33K) + PaddleOCRv2 pseudo-labels (1M). - **GPT-3.5 dependency:** Region-level translation and summary annotations were generated via GPT-3.5, introducing non-determinism and potential API deprecation risk for reproduction. - **Benchmark:** 112 English + 100 Chinese manually collected pages, 1000+ words per page. Available for download via [HuggingFace](https://huggingface.co/datasets/ucaslcl/Fox_benchmark_data). - **Public availability:** Benchmark data released under CC-BY-NC-4.0. Training data pipeline described but raw data not bundled. No data generation scripts are provided. ### Evaluation - **Metrics:** Edit distance, F1, precision, recall, BLEU, METEOR, ROUGE-L (R, P, F), accuracy. - **Baselines:** Not all are tested on all sub-tasks. Multi-page and region-level tasks have no external baselines (only Fox is evaluated). - **No error bars or variance reporting.** Single-run results only. ### Hardware - **Training:** 48$\times$ NVIDIA A800 GPUs, per-device batch size 4. Both pre-training and SFT run for 1 epoch. - **GPU-hours:** Not reported. Given 48 GPUs and ~10M pre-training samples at batch 4, the compute cost is substantial but unquantified. - **Inference:** Not detailed. Each page produces 256 tokens per vision branch (512 total after concatenation), so multi-page inference on 8 pages requires 4,096 vision tokens plus the language instruction. ## BibTeX ```bibtex @article{liu2024fox, title={Fox: Focus Anywhere for Fine-grained Multi-page Document Understanding}, author={Liu, Chenglong and Wei, Haoran and Chen, Jinyue and Kong, Lingyu and Ge, Zheng and Zhu, Zining and Zhao, Liang and Sun, Jianjian and Han, Chunrui and Zhang, Xiangyu}, journal={arXiv preprint arXiv:2405.14295}, year={2024} } ```