# SmolDocling: A 256M VLM for Document Conversion > Analysis of SmolDocling, a 256M-parameter vision-language model using DocTags markup for end-to-end document conversion with layout, tables, code, and formulas - Authors: Hunter Heidenreich - Published: 2025-03-01 - Updated: 2026-03-04 - Paper: SmolDocling Paper - Model: SmolDocling-256M-preview - Source: Bevaya Labs, --- ## TL;DR SmolDocling is a 256M-parameter vision-language model that converts document page images directly into **DocTags**, a structured XML-style markup encoding content, layout, and element locations in a single sequence. Built on SmolVLM-256M with a document-specific token vocabulary, large synthetic datasets (tables, charts, code, formulas), and a curriculum training recipe, it matches or exceeds much larger VLMs on OCR, layout analysis, and structure extraction while remaining in the same compute budget as classic ensemble systems. * * * ## What kind of paper is this? **Dominant:** $\\Psi\_{\\text{Method}}$ -- introduces SmolDocling model, DocTags output format, and curriculum training strategy for end-to-end document conversion. **Secondary:** $\\Psi\_{\\text{Resource}}$ -- contributes several new datasets (DocLayNet-PT, SynthDocNet, SynthChartNet, SynthCodeNet, SynthFormulaNet) with instruction-tuned variants. ## What is the motivation? Converting complex PDFs into structured, machine-readable formats remains challenging: - PDFs prioritize print rendering over semantics or reading order - Real documents have complex layouts: multi-column, forms, tables, charts, code, equations - Existing solutions split into two camps: - **Ensembles:** pipelines of OCR + layout + table + post-processing; efficient but brittle - **Large VLMs:** single multimodal models for Q&A over documents; compute-heavy, prone to hallucinations, focused on Q&A rather than faithful conversion - Data gap: few open multimodal datasets jointly cover layout, tables, equations, code, and charts at page level with consistent markup - Target: a small, efficient VLM for **full document conversion** (content + structure + spatial grounding) with open datasets and standardized markup ## What is the novelty? - **SmolDocling architecture** on **SmolVLM-256M**: - 256M total parameters: SigLIP-base vision encoder ($\\sim$93M) + lightweight SmolLM-2 language backbone ($\\sim$135M) - Aggressive pixel shuffle: compresses each 512$\\times$512 patch to 64 visual tokens (4096 pixels per token) - **DocTags markup vocabulary:** - XML-style tokens separating text content from structure - Blocks: ``, ``, ``, ``, ``, `<section_header>`, `<list_item>`, `<page_header>`, `<page_footer>`, `<picture>`, `<otsl>`, `<code>`, `<document_index>`, plus `<ordered_list>` / `<unordered_list>` - Spatial grounding: every block embeds bounding box via `<loc_x1><loc_y1><loc_x2><loc_y2>` on 0--500 coordinate grid - Tables: **OTSL** vocabulary (`<fcel>`, `<ecel>`, `<lcel>`, `<ucel>`, `<xcel>`, `<nl>`, plus header tags `<ched>`, `<rhed>`, `<srow>`) - Code: `<_programming-language_>` subtype tags preserve indentation and line breaks - Pictures: `<image_class>` tags with detailed class taxonomy (charts, molecular structures, logos, signatures, diagrams) - **Unified representation:** cropped elements (tables, code, equations, charts) use identical DocTags structure as when embedded in full pages, improving alignment between local and global tasks - **Curriculum training:** 1. Extend tokenizer with DocTags vocabulary 2. Freeze vision encoder, train language model to output DocTags 3. Unfreeze vision encoder, jointly train on document pretraining + task-specific datasets 4. Final fine-tuning on all datasets including instruction-style examples - **Instruction interface:** simple textual prompts such as "Convert this page to Docling," "Convert chart to table," "Convert formula to LaTeX," "OCR the text in a specific location `<loc_...>`" (Table 6, p. 20) Architecture diagram (Figure 1, p. 2) shows: SigLIP encoder $\\rightarrow$ projection and pooling $\\rightarrow$ concatenation with text embeddings $\\rightarrow$ LLM autoregressively emits DocTags. ## What experiments were performed? **General setup:** - Input images standardized to **144 DPI** - Maximum sequence length: **8192 tokens** (up to 3 pages per forward pass) - For text-only metrics, DocTags and HTML outputs converted to Markdown; tables excluded from full-page text evaluation **1) Text recognition (full pages)** - Dataset: DocLayNet test pages (excluding tables) - Baselines: Qwen2.5-VL 7B, GOT 580M, Nougat base 350M - Metrics: edit distance, F1, precision, recall, BLEU, METEOR **2) Code listing OCR** - Dataset: SynthCodeNet rendered code snippets - Metrics: text similarity on plaintext outputs (line breaks, indentation) - Evaluation only reported for SmolDocling (other models not trained for this task) **3) Equation recognition** - Dataset: Im2LaTeX-230k test formulas (normalized LaTeX) - Baselines: Qwen2.5-VL, GOT, Nougat **4) Layout analysis** - Dataset: DocLayNet test set (6 classes: Text, Section Heading, List Item, Table, Picture, Formula) - Metric: mAP[0.5:0.95] - Baselines: Qwen2.5-VL-7B, human agreement **5) Table structure recognition** - Datasets: FinTabNet, PubTables-1M (cropped table images, $\\sim$72 DPI with compression artifacts) - Baselines: SmolVLM-2.2B, Granite Vision, TableFormer - Metric: **TEDS** with and without cell text (structure-only in parentheses) **6) Chart extraction** - Task: convert cropped charts to tables - Baselines: Phi-3.5-vision, Granite Vision, SmolVLM-2.2B, Molmo-E - Metric: TEDS between predicted and ground-truth HTML tables **7) Qualitative analyses** - Figure 2 (p. 4): DocTags snippets for tables with captions, code blocks, equations, nested lists, image classes - Figure 3 (p. 15): treemap of dataset contributions by type and size - Figures 4--6 (pp. 16--18): visual samples of SynthChartNet, SynthCodeNet, SynthFormulaNet - Figure 7 (p. 21): molecule recognition comparison (GOT-OCR 2.0, Qwen2.5-VL-72B, SmolDocling) - Table 7 (pp. 22--24): layout bounding box overlays (SmolDocling vs. Qwen2.5-VL) on six DocLayNet pages ## What are the outcomes/limitations? **Outcomes (reported):** - **Text OCR (full pages, DocLayNet):** - SmolDocling: edit distance 0.48, F1 0.80, precision 0.89, recall 0.79, BLEU 0.58, METEOR 0.67 - Qwen2.5-VL 7B: edit distance 0.56, F1 0.72, BLEU 0.46, METEOR 0.57 - GOT 580M: edit distance 0.61, F1 0.69 - Nougat 350M: edit distance 0.62, F1 0.66 - SmolDocling strictly better on all metrics despite being significantly smaller - **Code OCR (SynthCodeNet):** - SmolDocling: edit distance 0.11, F1 0.92, precision 0.94, recall 0.91, BLEU 0.87, METEOR 0.89 - No comparison baselines (effectively sets benchmark) - **Equation OCR (Im2LaTeX-230k):** - SmolDocling: edit distance 0.11, F1 0.95, precision 0.96, recall 0.95, BLEU 0.83, METEOR 0.89 - GOT: slightly higher on BLEU/METEOR but very close overall - Qwen2.5-VL: edit distance 0.22, F1 0.89 - Nougat: edit distance 0.62, F1 0.60 - **Layout analysis (DocLayNet, mAP[0.5:0.95]):** - SmolDocling: 0.231 overall - Qwen2.5-VL-7B: 0.133 overall - Human agreement baseline: 0.82 - SmolDocling clearly better than Qwen2.5-VL but far below human performance - **Table structure recognition:** - FinTabNet: TEDS 0.52 (with text), 0.81 (structure only) - PubTables-1M: TEDS 0.65 (with text), 0.88 (structure only) - TableFormer (52M): 0.89 FinTabNet, 0.84 PubTables-1M (uses external OCR) - Structure-only scores indicate grid reconstruction works reasonably well; suffers from low resolution of table crops - **Chart extraction:** - SmolDocling: TEDS 0.75 - Granite Vision: 0.95 - Molmo-E: 0.54 - Phi-3.5-vision: 0.40 **Limitations:** - **Layout grounding:** bounding boxes can be imprecise, especially for colorful layouts or terminal output; recall can be low on some document types - **Tag integrity:** occasional missing closing tags or partially emitted location tags can break downstream parsing - **Repetition loops:** sometimes repeats tokens and fabricates extra cells or text segments late in sequence (typical autoregressive failure) - **Charts and molecules:** chart reconstruction quality is dataset-dependent; molecule recognition experiments (Figure 7) show general VLMs do not match specialized models like MolGrapher or DECIMER on structure fidelity - **Long-sequence robustness:** layout grounding and consistency need improvement for complex multi-page documents The results suggest a small VLM with well-designed markup and datasets can match or approach much larger models on several document tasks, but spatial precision and long-sequence handling remain open challenges. * * * ## Model ### Architecture - **Vision encoder:** SigLIP base patch-16/512, $\\sim$93M parameters - **Language model:** lightweight SmolLM-2 variant, $\\sim$135M parameters - **Total parameters:** 256M - **Visual pipeline:** - Images tiled into 512$\\times$512 patches - Pixel shuffle compresses each patch to 64 visual tokens - Special tokens separate sub-images when multiple pages packed - Pixel-to-token ratio: 4096 pixels per token - **Fusion:** projected visual embeddings concatenated with text token embeddings from instruction prompt; autoregressive decoder (SmolLM-2) generates DocTags sequence (Figure 1, p. 2) - **Tokenizer:** DocTags tokens added to vocabulary before training; model trained to emit them exactly ## Data Treemap visualization of dataset sizes and types shown in Figure 3 (p. 15). ### Pretraining / multi-task document data - **DocLayNet-PT:** 1.4M pages from DocFM corpus with weak annotations for layout, table structure, language, topic, figure classification encoded in DocTags - **DocMatix with DocTags conversion:** 1.3M documents extended from DocMatix with full document conversion to DocTags plus DocVQA questions - **DocLayNet-PT-Instruct:** instruction-tuned version of DocLayNet-PT for document-centric instructions - **The Cauldron:** other SmolVLM pretraining datasets retained for general multimodal skills ### Task-specific datasets **Layout / conversion:** - DocLayNet v2: 60K pages with human-corrected layout annotations (from 76K total pages) - WordScape pages with tables: 63K pages with structure-preserving XML ground truth - **SynthDocNet:** 250K synthetic pages from Wikipedia with diverse layouts; table lists and text blocks synthesized with precise annotations **Tables:** - PubTables-1M, FinTabNet, WikiTableSet (EN), WordScape tables - Original structures converted to OTSL interleaved with cell text - Total: $\\sim$1.5M+ tables **Charts (SynthChartNet):** - Source: 90K tables from FinTabNet; each rendered into multiple chart types - Counts: $\\sim$5K line charts, 380K pie charts, 380K bar charts, 77K stacked bar charts - Renderers: Matplotlib, Seaborn, Pyecharts - Total: $\\sim$2.5M charts (Figure 4, p. 16) **Code (SynthCodeNet):** - 9.3M rendered snippets from permissively licensed code datasets (The Stack, CodeNet) - Rendered with LaTeX listings and Pygments at 120 DPI - Random fonts, sizes, colors, line numbers, syntax highlighting - Covers 56 programming languages (Figure 5, p. 17) **Equations (SynthFormulaNet):** - 5.5M normalized LaTeX formulas combining public datasets plus 4.7M equations from arXiv LaTeX sources - Rendered at 120 DPI with randomized fonts and optional equation numbers (Figure 6, p. 18) **Instruction data:** - From DocLayNet-PT pages: instructions like "Perform OCR at bbox," "Identify page element type at bbox," "Extract all section headers," "Detect footer elements," plus full conversion instructions (Table 6, p. 20) ## Algorithms / Training ### Curriculum 1. Extend tokenizer with DocTags vocabulary 2. Freeze vision encoder and train remaining network to output DocTags (ensuring language model learns new markup) 3. Unfreeze vision encoder and jointly train on: - Document pretraining data (DocLayNet-PT, DocMatix with DocTags) - Task-specific datasets (tables, code, equations, charts) 4. Final fine-tuning on union of all datasets including instruction-style examples to align with conversion instructions and no-code actions ### Optimization - **Optimizer:** AdamW - **Learning rate:** $2 \\times 10^{-4}$ for most of network; $2 \\times 10^{-6}$ for vision encoder after unfreezing - **Gradient clipping:** 1.0 - **Warmup ratio:** 0.03 - **Training epochs:** 4 ### Sequence handling - Maximum sequence length: 8192 tokens (up to 3 pages concatenated) - Pages may be interleaved with text prompts or instructions The paper does not present detailed ablations on curriculum stages or DocTags design; evidence is indirect via task metrics. ## Evaluation ### Text recognition (DocLayNet) SmolDocling outperforms Nougat, GOT, and Qwen2.5-VL across all similarity scores; improvements especially strong in F1 and BLEU. Tables excluded from these metrics (table OCR quality evaluated separately via TEDS). ### Code and equation OCR - **Code:** strong scores on SynthCodeNet; authors treat this as defining a new benchmark - **Equations:** SmolDocling and GOT roughly tied; both significantly above Nougat; SmolDocling slightly ahead of Qwen2.5-VL ### Layout - SmolDocling achieves higher mAP than Qwen2.5-VL in all six classes - Human mAP far higher; both models struggle with exact box alignment and class confusion - Example visualizations (Table 7, pp. 22--24): - Multi-column pages handled reasonably, with occasional missed paragraphs or misclassified text - Terminal-style manual pages and colorful reports are harder (low recall for Qwen2.5-VL, misaligned boxes for SmolDocling) ### Tables TEDS scores indicate SmolDocling reconstructs table **structure** well; text transcription for low-resolution crops weaker. Structure-only scores (0.81, 0.88) suggest downstream systems can recover accurate HTML or spreadsheets once text is OCR'ed. ### Charts SmolDocling sits in middle of pack on chart-to-table conversion; authors attribute this to inconsistencies in how different datasets represent chart semantics. ### Molecule recognition Preliminary experiment only: SmolDocling, GOT-OCR 2.0, and Qwen2.5-VL do not reconstruct molecules as accurately as domain-specific models like MolGrapher or DECIMER. Authors suggest better SMILES tokenization and explicit atom/bond encoding might be necessary for competitive performance. ## Hardware / Production ### Training - **Setup:** 64 $\\times$ NVIDIA A100 80GB GPUs - **Duration:** $\\sim$38 hours per epoch - **Total epochs:** 4 ### Inference - **Implementation:** vLLM - **Throughput:** 0.35 seconds per page on single A100 - **VRAM usage:** 0.489 GB for inference The compute footprint is similar to strong ensemble systems rather than large VLMs, while supporting multimodal instructions and unified DocTags output.