# Dolphin: A 322M Document Parsing VLM with Analyze-then-Parse Pipeline > Analysis of Dolphin, a 322M-parameter document image parsing VLM using an analyze-then-parse pipeline with parallel type-specific element recognition - Authors: Hunter Heidenreich - Published: 2025-05-15 - Updated: 2026-03-04 - Paper: Dolphin Paper - Code: Dolphin Code - Model: Dolphin 322M Model - Model: Dolphin 322M v1.5 Model - Model: Dolphin 4B v2 Model - Source: Bevaya Labs, --- ## TL;DR Dolphin is a 322M-parameter document image parsing VLM that uses an **analyze-then-parse** pipeline: Stage 1 predicts a **reading-order layout element sequence** (type + bounding box), Stage 2 parses each element **in parallel** using **type-specific prompts**. Trained on **30.27M** multi-granularity samples (page-level + element-level), heavily weighted toward formula elements, it reports normalized **Edit Distance 0.0114** on Fox-Page-EN, **0.0131** on Fox-Page-ZH, and **0.1028** on the introduced Dolphin-Page benchmark, while running at **0.1729 FPS** due to parallel element parsing. * * * ## What kind of paper is this? **Dominant:** $\Psi_{\text{Method}}$ (introduces two-stage analyze-then-parse design with reading-order layout anchors and parallel type-specific element recognition). **Secondary:** $\Psi_{\text{Resource}}$ (30.27M training set across multiple granularities; introduces Dolphin-Page and Dolphin-Block benchmarks). **Also present:** $\Psi_{\text{Evaluation}}$ (broad comparisons across multiple benchmarks, though no new metric proposed). Rough superposition: `0.65 Method + 0.20 Resource + 0.15 Evaluation`. ## What is the motivation? **End-to-end autoregressive limitations:** Full-page generation methods face quadratic attention costs and can degrade layout structure on long or complex documents containing mixed elements (text, tables, formulas, figures). **Pipeline coordination overhead:** Multi-stage toolchains require careful integration and coordination across separate specialized models, introducing complexity and potential error propagation. **Goal:** Achieve robust **structured extraction** from document images with mixed elements, producing machine-readable markup that maintains layout structure while enabling efficient parallel processing. ## What is the novelty? **Two-stage analyze-then-parse pipeline:** - **Stage 1 (Analyze):** Generate a sequence of layout elements in **natural reading order**, each with type and bounding box. These elements become **anchors** for the recognition stage. Prompted with: "Parse the reading order of this document." - **Stage 2 (Parse):** Crop each predicted element region and parse all elements **in parallel**, guided by **type-specific prompts** (e.g., "Parse the table in the image" for table regions, outputting HTML structure). > **Contrast to end-to-end models:** Dolphin avoids full-page autoregressive generation by decomposing into layout planning + parallel element recognition, reducing long-context burden and enabling batched processing of detected regions. > **Contrast to pipeline tools:** Unlike tools that route different element types to completely separate specialized models, Dolphin uses a **unified model** for all element types, differentiating behavior via type-specific prompts rather than separate model architectures. **Multi-granularity training:** Combines page-level parsing data with element-level recognition data (cropped blocks). This allows the model to learn both layout structure and fine-grained content recognition, with heavy weighting toward formula elements (23M of 30.27M samples). **Element-centric data strategy:** Stage 1 training *avoids treating formulas as independent layout elements* to preserve broader context for formula recognition, while Stage 2 includes extensive formula-specific training data. ## What experiments were performed? **Page-level parsing benchmarks:** - Fox-Page (English + Chinese): text-heavy document pages - Dolphin-Page: "complex document" benchmark introduced by the authors **Element-level recognition benchmarks:** - Text blocks: FoxBlock-EN, FoxBlock-ZH, DolphinBlock - Formulas: SPE (Simple Printed Expression), SCE (Simple Chinese Expression), CPE (Complex Printed Expression) - Tables: PubTabNet (7,904 test samples), PubTab1M (10,000 test samples) **Baselines:** Comparisons include Nougat, GOT-OCR2.0, StructEqTable, Vary, OFA, and Donut across different element types. **Ablations:** Parallel vs. sequential element decoding (throughput comparison). ## What are the outcomes/limitations? **Outcomes (reported):** Page-level Edit Distance (normalized): - Fox-Page-EN: **0.0114** - Fox-Page-ZH: **0.0131** - Dolphin-Page: **0.1028** (Table 1) *Note: Section 5.2 text claims 0.1283, inconsistent with table* - Average across benchmarks: **0.0575** - Throughput: **0.1729 FPS** Element-level performance: - Paragraph ED: **0.0029** (FoxBlock-EN), **0.0121** (FoxBlock-ZH), **0.0136** (DolphinBlock) - Formula CDM: **0.9850** (SPE), **0.9685** (SCE), **0.8739** (CPE) - Table TEDS: **0.9515** (PubTabNet), **0.9625** (PubTab1M) Parallel decoding speedup: **1.8$\times$** (0.1729 vs. 0.0971 FPS) with similar accuracy. **Limitations:** - **Vertical text support:** Limited capability for vertical text layouts common in some Asian language documents. - **Multilingual capacity:** Beyond Chinese and English, additional languages require further development. - **Handwriting recognition:** Current version does not handle handwritten text effectively. - **Parallel processing bounds:** Despite parallel element parsing, speedup is constrained by: - Preprocessing overhead (detection, cropping, batching) - **Max 16 elements per batch** due to GPU memory limits, requiring multiple passes on element-dense pages - This limits practical throughput gains compared to theoretical maximum from full parallelization **Open questions:** - Minor inconsistency between Table 1 (Dolphin-Page ED: 0.1028) and Section 5.2 text (claims 0.1283) raises questions about which result is authoritative. - No discussion of how reading-order prediction quality affects downstream parsing accuracy. * * * ## Model **Architecture:** Encoder-decoder transformer following Donut design. **Encoder:** - **Swin Transformer** backbone - Window size: 7 - Layer configuration: [2, 2, 14, 2] (4 stages) - Attention heads: [4, 8, 16, 32] **Decoder:** - **mBART** architecture - 10 layers - Hidden size: 1024 **Initialization:** Donut pretrained weights. **Total parameters:** 322M. ## Data **Training set:** 30.27M samples total, combining page-level and element-level data. | Data Type | Samples | Notes | |-----------|---------|-------| | Mixed Documents | 0.12M | Page-level parsing | | HTML | 4.37M | Page-level parsing | | LaTeX | 0.5M | Page-level parsing | | Markdown | 0.71M | Page-level parsing | | Table elements | 1.57M | Element-level cropped regions | | Formula elements | 23M | Element-level cropped regions (76% of dataset) | **Data strategy:** Heavy weighting toward formula elements reflects the complexity and importance of mathematical content recognition in scientific documents. **Layout representation:** Stage 1 avoids treating formulas as independent layout elements to maintain broader context during formula recognition, while Stage 2 uses extensive formula-specific training for detailed parsing. ## Algorithms / Training **Training objective:** Standard **cross-entropy** token prediction loss. **Dynamic instruction-tuning:** For each training sample, randomly select one applicable task from 5 task types based on available annotations to form question-answer pairs. This prevents the model from memorizing fixed prompt-response patterns. **Task prompts (Table 6):** ```text Page-level Layout Analysis: Parse the reading order of this document. Text Paragraph/Formula Parsing: Read text in the image. Table Parsing: Parse the table in the image. Text Spotting: Detect and recognize all the text lines in the image. Text Box Query: Read the text in the image within the specified box [x1,y1,x2,y2]. ``` **Training configuration:** - Hardware: **40 A100 GPUs** - Epochs: **2** - Per-device batch size: **16** (via gradient accumulation) - Optimizer: AdamW - Learning rate: **5e-5** with cosine decay schedule ## Evaluation **Metrics:** - **Edit Distance (ED):** Normalized character-level edit distance for text and page-level comparisons - **CDM (Character Detection Metric):** Character-level edit distance for formula evaluation - **TEDS (Tree Edit Distance-based Similarity):** Structural similarity over HTML table representations **Formula benchmarks:** - SPE (Simple Printed Expression) - SCE (Simple Chinese Expression) - CPE (Complex Printed Expression) **Table benchmarks:** - PubTabNet: 7,904 test samples - PubTab1M: 10,000 test samples **Page-level benchmarks:** - Fox-Page-EN: English text-heavy documents - Fox-Page-ZH: Chinese text-heavy documents - Dolphin-Page: Complex multi-element documents (introduced by authors) **Element-level benchmarks:** - FoxBlock-EN/ZH: English and Chinese text paragraphs - DolphinBlock: Mixed element types ## Hardware / Production **Input preprocessing:** 1. Preserve aspect ratio 2. Resize longer edge to **896 pixels** 3. Pad to **896 $\times$ 896** square 4. Normalize bounding box coordinates within padded space **Inference pipeline:** 1. Stage 1: Full-page forward pass to predict layout elements (type + bbox) 2. Crop detected elements from original image 3. Stage 2: Batch element crops (max **16 per batch**) for parallel recognition 4. Reassemble elements according to predicted reading order **Throughput analysis:** - Sequential decoding: 0.0971 FPS - Parallel decoding: 0.1729 FPS (**1.8$\times$ speedup**) - Speedup limited by preprocessing overhead and batch size constraints - For pages with $>16$ elements, multiple batches required **Memory constraints:** Max 16 elements per batch on typical GPU configurations; element-dense pages require multiple forward passes through Stage 2.