# Ocean-OCR-3B: A Multimodal LLM for General OCR > Analysis of Ocean-OCR, a 3B-parameter multimodal LLM using NaViT and Qwen-2.5-3B for OCR across dense documents, scene text, and handwriting - Authors: Hunter Heidenreich - Published: 2025-01-01 - Updated: 2026-03-04 - Paper: Ocean-OCR Paper - Code: Ocean-OCR Code - Model: Ocean-OCR Models - Source: Bevaya Labs, --- ## TL;DR Ocean-OCR is a 3B-parameter multimodal LLM built around a NaViT vision encoder plus an MLP projector into a Qwen-2.5-3B language model, trained with a three-stage pipeline and a large OCR-heavy data mixture. The authors report strong results on standard OCR benchmarks and on three "practical scenario" evaluations (dense bilingual documents, scene text, bilingual handwriting), including competitive or better performance than specialized OCR systems in their setup. --- ## What kind of paper is this? **Dominant:** $\Psi_{\text{Method}}$ (new model recipe for "general OCR" via dynamic-resolution vision encoder + OCR-centric data + staged training). **Secondary:** $\Psi_{\text{Impact}}$ (real-world style comparisons vs TextIn API and PaddleOCR across scenarios), plus a smaller $\Psi_{\text{Resource}}$ component (large curated in-house + synthetic data, plus constructed eval sets, though release details are not the headline). One reasonable superposition: $$\text{Paper} \approx 0.65\,\Psi_{\text{Method}} + 0.20\,\Psi_{\text{Impact}} + 0.15\,\Psi_{\text{Resource}}$$ ## What is the motivation? - Mainstream MLLMs are strong at reasoning, but their OCR/perception is often insufficient for dense text and text-heavy images, which blocks text-related tasks. - Prior approaches include sliding windows, cropping, token compression, and dynamic tiling, but the authors claim OCR performance is still short of practical requirements and often limited to OCR-specific settings rather than general-purpose use. ## What is the novelty? - **Architecture choice for resolution variability:** Use a NaViT vision encoder so the model can process images of any resolution and produce a variable number of visual tokens. - **Token-count control for high-res images:** Compress adjacent $2 \times 2$ visual tokens into 1 token to reduce compute while trying to preserve key information. - **Data and pipeline emphasis:** Build a large multimodal dataset mixture (including OCR and OCR-augmented captioning) and train with a 3-phase pipeline (projector alignment, full pretraining, SFT). ## What experiments were performed? - **General VLM benchmarks:** MMMU, MMBench (EN/CN), MathVista, MME, SEEDBench, RealWorldQA, HallusionBench, evaluated via VLMEvalKit in zero-shot using original configs. - **Open OCR benchmarks:** DocVQA, TextVQA, ChartQA, OCRBench. - **Practical OCR scenarios (constructed evals):** - Dense bilingual document extraction (100 EN papers + 100 ZH papers), metrics include normalized edit distance, F1, precision, recall, BLEU, METEOR. - Scene text OCR benchmark (260 natural images, half Chinese and half English, from MSRA-TD500; pseudo labels via PaddleOCR then manual correction). - Bilingual handwriting benchmark with 4 granularities/sources, 100 samples per category, evaluated with the same metric set. ## What are the outcomes/limitations? **Key outcomes:** - General benchmarks: Ocean-OCR reports MMMU 42.0, MMBench-EN 75.3, MMBench-CN 73.0, MathVista 55.6, MME 2094, SEEDBench 72.5, RealWorldQA 61.2, HallusionBench 46.0. - OCR benchmarks (Table 3): DocVQA 91.4, TextVQA 80.0, ChartQA 84.6, OCRBench 82.7, average 84.7. - Dense bilingual documents (Table 4): Ocean-OCR achieves edit distance 0.057 (EN) and 0.062 (ZH), with strong F1/precision/recall and BLEU/METEOR on both languages. - Scene text (Table 5): Ocean-OCR reports edit distance 0.113 with F1 0.875 and METEOR 0.754, outperforming several MLLM baselines in this setup. **Limitations:** - **Training compute and hyperparameters are not specified:** optimizer, learning rate schedule, batch sizes, training tokens/steps, GPUs, training time are unstated. - **"First MLLM to outperform TextIn and PaddleOCR"** is evaluated on their constructed scenario datasets and their chosen TextIn endpoints; for example, on dense EN document edit distance TextIn is slightly lower (0.055 vs 0.057), while Ocean-OCR is much better on ZH in that table. - The paper mentions future work to build a medical report benchmark, implying that scenario is currently anecdotal/case-based rather than a standardized eval. --- ## Model ### High-level architecture - LLaVA-style 3-part stack: **NaViT** vision encoder + **MLP projector** + **Qwen-2.5-3B** LLM. - NaViT provides variable-resolution input handling and yields a variable number of visual tokens per image. ### Visual token compression - For high-resolution images, adjacent $2 \times 2$ tokens are compressed into a single token to reduce token count and compute. ### LLM choice rationale - They choose Qwen-2.5-3B for "ease of use and practical deployment," balancing capability and size. --- ## Data ### Training data inventory (Table 1) Public vs in-house counts (in M samples, as reported): **Alignment + pretrain:** - Pure text: 150.7M (in-house only) - Caption: 33.2M public, 49.1M in-house - Interleaved: 19.1M public, 28.7M in-house - OCR: 12.4M public, 7.8M in-house **SFT:** - General QA (Cauldron): 3.6M public - OCR QA: 3.0M public, 1.9M in-house **Totals:** 71.3M public, 238.2M in-house. ### Mixture and sourcing details - Pretraining keeps a 50/50 mix of pure text and vision-language data. - **Interleaved data:** OBELICS plus an in-house parsed corpus from books/papers, with an approximate 4:6 OBELICS:in-house ratio. - **Caption data:** open datasets (DenseFusion-1M, Synthdog, DreamLIP, InternVL-SA-1B-Caption) plus synthetic captions where OCR hints are injected using PaddleOCR + GPT-4o (images sourced from Wukong and LAION-2B). - **OCR data:** open OCR datasets (DocStruct4M, RenderedText, AnyWord-3M, TinyChartData) plus synthetic scene/PDF/handwriting OCR (PDFs from in-house e-books; handwriting rendered from pure-text corpora). ### SFT data construction and filtering - General VQA SFT uses Cauldron with filtering; they put data into a dialogue template and use Qwen2-VL-72B to judge response accuracy and filter low-quality QA pairs. - OCR SFT expansions include synthesized scene OCR QA (COCO-Text, ICDAR2019 ArT, Incidental Scene Text) using GPT-4o to generate "realistic" text VQA, plus synthetic handwriting and in-house PDF-derived data. --- ## Algorithms / Training ### Three-phase pipeline 1. **Vision-language alignment:** freeze vision encoder + LLM; train the projector to map visual tokens into the text feature space; NaViT is used as the vision encoder. 2. **Vision-language pretraining:** unfreeze all modules (NaViT, MLP projector, LLM) and train jointly to build multimodal knowledge while preserving language capability. 3. **Supervised fine-tuning:** update all components; goal is instruction following while maintaining general ability. ### Optimization objective - Across all stages, they use next-token prediction loss on text tokens. ### Missing implementation details - No batch size, learning rate schedule, optimizer, weight decay, training steps/tokens, or hardware/training time are provided in the paper. --- ## Evaluation ### Evaluation harness and protocol - All evaluations use VLMEvalKit, run zero-shot, and follow models' original configurations for consistency. ### General benchmarks (Table 2) | Benchmark | MMMU | MMBench-EN | MMBench-CN | MathVista | MME | SEEDBench | RealWorldQA | HallusionBench | |-----------|------|------------|------------|-----------|-----|-----------|-------------|----------------| | Ocean-OCR | 42.0 | 75.3 | 73.0 | 55.6 | 2094 | 72.5 | 61.2 | 46.0 | ### Open OCR benchmarks (Table 3) | Model | DocVQA | TextVQA | ChartQA | OCRBench | Average | |-------|--------|---------|---------|----------|---------| | Ocean-OCR | 91.4 | 80.0 | 84.6 | 82.7 | 84.7 | **Note on baselines:** GOT is described as OCR-specific and not compatible with complex instructions or general tasks; TextIn and PaddleOCR are treated as specialized OCR baselines. ### Practical scenario evaluations **Dense bilingual document extraction (Table 4)** - **Dataset:** 100 English paper images + 100 Chinese paper images - **Metrics:** normalized edit distance, F1, precision, recall, BLEU, METEOR - **Ocean-OCR results (EN / ZH):** - Edit distance: 0.057 / 0.062 - F1: 0.937 / 0.962 - Precision: 0.932 / 0.956 - Recall: 0.956 / 0.974 - BLEU: 0.906 / 0.912 - METEOR: 0.945 / 0.916 - **TextIn endpoints used:** PDF-to-markdown for document extraction; multipage endpoint for scene + handwriting. **Scene text recognition (Table 5)** - **Dataset:** 260 natural images, split evenly Chinese/English, sampled from MSRA-TD500; pseudo GT via PaddleOCR then manual verification and correction. - **Ocean-OCR results:** - Edit distance: 0.113 - F1: 0.875 - Precision: 0.875 - Recall: 0.887 - BLEU: 0.420 - METEOR: 0.754 **Handwritten recognition (Table 6)** - **Dataset:** paragraph-level real (CASIA-HWDB, GNHK), line-level real (CASIA-HWDB, BRUSH), plus paragraph/line synthetic, bilingual; 100 samples per category; same metric set. --- ## Hardware / Production Hardware, training time, throughput, and serving details are not reported in the paper.