# DeepSeek-OCR: Optical Compression for Document Understanding > Analysis of DeepSeek-OCR, a VLM treating text images as compression media with DeepEncoder and 3B MoE decoder for token-efficient OCR - Authors: Hunter Heidenreich - Published: 2025-10-01 - Updated: 2026-03-04 - Paper: DeepSeek-OCR Paper - Code: DeepSeek-OCR Code - Model: DeepSeek-OCR Model - Source: Bevaya Labs, --- ## TL;DR DeepSeek-OCR is a VLM that treats **an image of text as a compression medium**: an encoder ("DeepEncoder") maps high-res document images into a small number of **vision tokens**, and a **3B MoE decoder** reconstructs the text. The authors report $\\approx 97\\%$ OCR decoding precision when the **text-token count is $\\le 10\\times$ the vision-token count**, and $\\approx 60\\%$ at $\\approx 20\\times$ compression; on OmniDocBench it aims to be competitive while using relatively few vision tokens. * * * ## What kind of paper is this? **Dominant:** $\\Psi_{\\text{Method}}$ (new encoder design + multi-resolution token budgeting + end-to-end OCR system). **Secondary:** $\\Psi_{\\text{Evaluation}}$ (compression study on Fox; benchmarking on OmniDocBench); $\\Psi\_{\\text{Resource}}$ (code/weights released; curated OCR training data). ## What is the motivation? - Long-context LLM compute scales quadratically with sequence length, so "just feed more text" is expensive. - A document image can represent many words, suggesting **vision tokens** could serve as a cheaper "compressed representation" of long text. - Prior open-source VLM vision encoders (dual-tower, tiling, NaViT-style adaptive resolution) each have issues for this use case: deployment complexity, too many vision tokens, high activation memory, or long packed sequences. ## What is the novelty? - **Contexts optical compression framing:** treat OCR as a compression-decompression mapping between **document pixels $\\rightarrow$ small vision-token sequence $\\rightarrow$ reconstructed text**, and study how many vision tokens are "enough" for a given amount of text. - **DeepEncoder:** serially connects a **window-attention perception module** (SAM-base) to a **dense global-attention knowledge module** (CLIP-large) using a **$16\\times$ token compressor** so high-res inputs do not explode dense attention cost. - **Multi-resolution token modes** (Tiny/Small/Base/Large + dynamic tiling modes "Gundam") to vary token budgets and study compression ratios. ## What experiments were performed? - **Vision-text compression study:** Fox benchmark, English docs with 600--1300 tokenizer tokens; evaluate OCR precision under **64** or **100** vision tokens (Tiny/Small). Reports precision vs. compression ratio. - **OCR practical performance:** OmniDocBench; compares edit distance across categories (English/Chinese; text/formula/table/order; and per-document-type breakdown) across multiple model modes (Tiny $\\rightarrow$ Gundam-M). Includes comparisons to pipeline OCR systems and other end-to-end VLM OCR models. - **Qualitative "deep parsing" demos:** secondary calls to parse charts, natural images, chemical formulas, and simple geometry with a unified prompt style. ## What are the outcomes/limitations? **Outcomes (reported):** - Fox compression: within $\\approx 10\\times$ compression, decoding precision reaches $\\approx 97\\%$; around $\\approx 20\\times$ compression precision is still $\\approx 60\\%$. - OmniDocBench: at **100 vision tokens**, the model is reported to surpass GOT-OCR2.0 (which uses 256 tokens/page in their table); at higher token modes (400, <800), it approaches or beats stronger baselines while using fewer tokens than some competitors. - The paper positions this as evidence that compact decoders can learn to reconstruct text from heavily compressed visual latents, and proposes this as a direction for long-context handling. **Limitations / caveats (explicit or strongly implied):** - Fox evaluation notes formatting mismatches between output and ground truth can understate "true" OCR accuracy (their prompt-output format may not match the benchmark exactly). - Degradation beyond $\\approx 10\\times$ is attributed to (a) more complex layouts in longer docs and (b) blur at 512--640 resolutions. Their proposed "fix" for (a) is rendering text into a single layout page, and they frame (b) as related to intentional forgetting. - They state there is **no SFT stage**, so the model is "not a chatbot" and some capabilities require completion-style prompts. - Several future-eval items are explicitly left open (digital-optical text interleaved pretraining, needle-in-a-haystack testing). * * * ## Model ### High-level architecture ![DeepSeek-OCR model architecture diagram showing two-stage encoder with SAM perception module, 16x token compressor, and CLIP global knowledge module feeding into a 3B MoE decoder](https://labs.bevaya.ai/images/ocr/deepseek-ocr-model-architecture-figure-3.webp) - End-to-end VLM: **DeepEncoder (encoder)** + **DeepSeek-3B-MoE (decoder)**. - **DeepEncoder params:** $\\approx 380\\text{M}$, composed of: - **SAM-base** (patch size 16), $\\approx 80\\text{M}$ params, used as perception module dominated by **window attention**. - **CLIP-large**, $\\approx 300\\text{M}$ params, used as knowledge module with **dense global attention**. - A **$16\\times$ token compressor** between them. - **Decoder:** DeepSeek-3B-MoE (routed MoE), **570M activated parameters** at inference; activates **6 of 64 routed experts + 2 shared experts**. ### Token compressor details - Implemented as a **2-layer convolution module** for $16\\times$ downsampling of vision tokens: - Each conv layer: kernel size 3, stride 2, padding 1. - Channel increase: 256 $\\rightarrow$ 1024. - Example: a 1024x1024 input yields $1024/16 \\times 1024/16 = 4096$ patch tokens into the first stage; after the compressor, tokens reduce to $4096/16 = 256$ before dense global attention. > **Why this design?** The paper positions it between three common encoder families (dual-tower, tile-based, NaViT-style), avoiding their pitfalls (too many tokens, very low native resolution, huge activations): ![Comparison diagram of VLM encoder types: dual-tower, tile-based, NaViT-style, and DeepSeek-OCR's approach](https://labs.bevaya.ai/images/ocr/deepseek-ocr-vlm-types-figure-2.webp) ### Multi-resolution modes and token budgets ![Diagram showing DeepSeek-OCR's multi-resolution strategy with Tiny/Small/Base/Large native modes and Gundam dynamic tiling](https://labs.bevaya.ai/images/ocr/deepseek-ocr-multi-resolution-tiling-figure-4.webp) - **Native modes** (single image, no tiling, fixed token budget): - Tiny: $512 \\times 512$ $\\rightarrow$ 64 tokens (resize) - Small: $640 \\times 640$ $\\rightarrow$ 100 tokens (resize) - Base: $1024 \\times 1024$ $\\rightarrow$ 256 tokens (padding) - Large: $1280 \\times 1280$ $\\rightarrow$ 400 tokens (padding) - **Dynamic modes** (global + tiles, variable token budget): - Gundam: $n \\times 640 \\times 640$ tiles + $1024 \\times 1024$ global $\\rightarrow$ $n \\times 100 + 256$ tokens, $n \\in [2,9]$. If both sides $< 640$, it degrades to Base. - Gundam-M: $n \\times 1024 \\times 1024$ tiles + $1280 \\times 1280$ global $\\rightarrow$ $n \\times 256 + 400$ tokens (trained by continued training for load-balancing). - Positional encodings are dynamically interpolated to support all modes with a single model; tiling follows InternVL2.0. - For padded native modes, valid tokens: $$N_{\\text{valid}}=\\left\\lceil N_{\\text{actual}}\\times\\left[1-\frac{\max(w,h)-\min(w,h)}{\max(w,h)}\right]\\right\\rceil$$ (Eq. 1). * * * ## Data ### OCR 1.0 data (documents + scene text) - **30M PDF pages** across $\\approx 100$ languages; Chinese+English $\\approx 25\\text{M}$ pages, other languages $\\approx 5\\text{M}$. - Two annotation types: - **Coarse** annotations extracted via `fitz` for broad OCR coverage, especially minority languages. - **Fine** annotations: **2M pages Chinese + 2M pages English**, labeled using layout (PP-DocLayout) + OCR systems (MinerU, GOT-OCR2.0) to create detection+recognition interleaved ground truth; for minority languages they describe a "model flywheel" using patch OCR to label patches after layout processing, producing **600K** samples. - **3M Word documents**: extract content to create high-quality image-text pairs (notably helpful for formulas and HTML tables). - Natural scene OCR: - Sources: LAION, Wukong; labels via PaddleOCR. - **10M** Chinese + **10M** English samples. - Prompts can control whether detection boxes are output. ### OCR 2.0 data (charts, chemistry, geometry) - **Charts:** 10M rendered images (pyecharts + matplotlib); labels are **HTML tables** (explicitly not OneChart dictionary format) to save tokens. - **Chemical formulas:** 5M image-text pairs; source SMILES from PubChem; render via RDKit; output target is SMILES. - **Plane geometry:** 1M samples; generation follows "Slow Perception"; uses a perception ruler size of 4 per line segment; includes translation-invariant augmentation. ### General vision + text-only mix - General vision data for caption/detection/grounding is included to preserve a general vision interface; authors state it is **20%** of training data. - Text-only pretraining data is **10%**, processed to **8192** tokens (also the full model sequence length). - Overall mix: **70% OCR**, **20% general vision**, **10% text-only**. * * * ## Algorithms / Training ### Stage A: Train DeepEncoder - Objective: next-token prediction using a compact language model, following Vary-style training. - Data: all OCR 1.0 + OCR 2.0 + **100M** general samples from LAION. - Training: 2 epochs, batch size 1280, AdamW, cosine annealing, learning rate $5\\times10^{-5}$, sequence length 4096. ### Stage B: Train full DeepSeek-OCR - Platform: HAI-LLM. - Pipeline parallelism: 4 partitions: - PP0: SAM + compressor as "vision tokenizer", **frozen**. - PP1: CLIP part as embedding layer, **unfrozen**. - PP2/PP3: DeepSeek3B-MoE has 12 layers, split 6 + 6. - Compute: **20 nodes**, each **8x A100-40G**; DP=40; global batch size 640. - Optimizer/schedule: AdamW, step-based scheduler, initial LR $3\\times10^{-5}$. - Throughput: text-only $\\approx 90\\text{B}$ tokens/day; multimodal $\\approx 70\\text{B}$ tokens/day. - Gundam-M is trained via continued training on a trained model (they cite load balancing / speed issues if trained jointly). ### Prompts / interfaces (examples shown in paper) ```text Free OCR. <|grounding|>Convert the document to markdown. Parse the figure. ``` (These are used to control layout output and deep parsing behavior.) * * * ## Hardware / Production - Data-generation throughput claims: - Abstract: "200k+ pages per day" on a single A100-40G. - Intro: "33 million pages per day" using 20 nodes (each 8x A100-40G). - The paper frames "contexts optical compression" as enabling a **forgetting-like mechanism**: render older dialogue rounds into images and progressively downsize them so older context gets blurrier and cheaper (Figure 13). * * * ## Evaluation ### Fox benchmark: compression ratio vs precision - Setup: English Fox docs, ground-truth tokens 600--1300 (using DeepSeek-OCR tokenizer, vocab size $\\approx 129\\text{k}$), 100 pages; evaluate Tiny (64) and Small (100) modes. - Reported table highlights (precision, compression): - 600--700 tokens: 64 toks $\\rightarrow$ 96.5% at 10.5$\\times$; 100 toks $\\rightarrow$ 98.5% at 6.7$\\times$ - 1200--1300 tokens: 64 toks $\\rightarrow$ 59.1% at 19.7$\\times$; 100 toks $\\rightarrow$ 87.1% at 12.6$\\times$ ### OmniDocBench: edit distance vs tokens - Metric: edit distance (lower is better), across English/Chinese and subcategories (text/formula/table/order), plus overall; "Tokens" is avg vision tokens per page, with parentheses indicating "valid" tokens for padded modes. - DeepSeek-OCR modes (tokens shown as reported): - Tiny: 64 - Small: 100 - Base: 256 (182 valid) - Large: 400 (285 valid) - Gundam: 795 - Gundam-M @200dpi: 1853 - Document-type breakdown suggests some categories (slides, books, reports) work well with low tokens, while newspapers require Gundam-class modes due to much higher text-token counts (4--5k).