# DocMark: Adaptive Markup Language Generation for Visual Document Understanding > Analysis of DocMark, an InternVL-based model that generates structured markup as intermediate context for grounded document question answering - Authors: Hunter Heidenreich - Published: 2025-05-01 - Updated: 2026-03-04 - Paper: DocMark Paper - Code: DocMark Code - Project: DocMark Models & Datasets - Source: Bevaya Labs, --- ## TL;DR DocMark trains an InternVL-based multimodal model to convert document images into structured markup (Plain Text, Markdown, LaTeX, HTML, JSON, TikZ) and then uses that markup as intermediate context for grounded question answering. The paper introduces **DocMark-Pile** (3.8M pretraining pairs for markup conversion) and **DocMark-Instruct** (624k instruction-following examples with extracted-context rationales), showing consistent gains over InternVL2 across document QA and OCR-related benchmarks. * * * ## What kind of paper is this? **Dominant:** $\Psi_{\text{Method}}$ (introduces a training and inference pipeline: pretrain for markup conversion, then fine-tune with a two-round "extract context then answer" procedure). **Secondary:** $\Psi_{\text{Resource}}$ (contributes large-scale structured datasets: DocMark-Pile with 3.8M samples and DocMark-Instruct with 624k examples, with domain coverage and task taxonomy documented in Table 3, p.13). * * * ## What is the motivation? Visual document understanding requires layout-sensitive perception and text comprehension, but common instruction datasets often provide only short answers, which encourages hallucinations and weak spatial grounding. Many documents have source-code-like structure; representing documents as markup can preserve semantics and layout structure in a form that LLMs can parse reliably (Markdown/HTML/LaTeX/JSON/TikZ examples in Figure 2, p.4). The authors frame the gap as missing contextually grounded training data and missing pipelines that generalize across diverse document types and layouts. * * * ## What is the novelty? **Adaptive markup language generation:** The model decides which markup format to emit as context (instead of forcing a fixed format), then uses it to answer questions via a two-round prompt template (Sec. 3.2.1, p.5). **DocMark-Pile (3.8M):** Multi-task pretraining dataset spanning six markup formats, intended to teach structural parsing rather than only OCR. **DocMark-Instruct (624k):** Instruction dataset with a chain-of-thought-like context extraction step. It is synthesized by (1) converting images to markup with the pretrained model, then (2) using ChatGPT-3.5 to extract the relevant snippet needed to answer each question (Sec. 3.2.2, p.6; prompt in Figure 7, p.16). **"Unclear" abstention in context extraction:** When parsed markup does not contain answer-relevant information, the context annotation is "unclear" (Figure 7, p.16). * * * ## What experiments were performed? **Markup generation capability (Table 1, p.7):** Text recognition, formula recognition (HME100K), chart structural extraction (ChartQA-SE AP@strict/slight/high), key information extraction (FUNSD, SROIE), and TikZ code consistency (DaTikZ-test). **Downstream document understanding (Table 2, p.8):** TextVQA, DocVQA, InfoVQA, ChartQA, AI2D, OCRBench, WebQA, MathVision. **Ablation on training data strategy (Sec. 4.4, p.8; Figure 5):** Removing DocMark-Pile or DocMark-Instruct reduces performance, especially on text-dense and structured tasks. **Supplementary ablation on CoT prompting versus CoT fine-tuning (Table 5, p.14):** Naive CoT prompting can hurt vanilla models; DocMark training makes the two-round process effective. **Token accounting (Figure 6, p.15):** Context tokens are a small fraction of total tokens relative to image tokens under their dynamic resolution strategy. * * * ## What are the outcomes/limitations? ### Outcomes DocMark-2B improves over InternVL2-2B on multiple downstream tasks (Table 2, p.8): | Benchmark | DocMark-2B | InternVL2-2B | | --------- | ---------- | ------------ | | TextVQA val | 74.8 | 73.2 | | DocVQA val | 87.8 | 86.0 | | InfoVQA val | 61.2 | 57.6 | | ChartQA | 79.8 | 76.4 | | AI2D | 82.5 | 74.2 | | OCRBench | 813 | 783 | | WebQA | 70.1 | 66.5 | | MathVision | 18.8 | 15.7 | DocMark-8B shows similar improvements, with gains in ChartQA (84.2 versus 82.2) and WebQA (78.9 versus 77.5) while roughly matching InternVL2-8B on DocVQA (89.8 versus 90.8) and trailing slightly on InfoVQA (68.3 versus 72.8). Qualitatively, generated markup context reduces errors that look like hallucinations in baseline models (Figure 1, p.2). ### Limitations **Style attribute preservation:** Generated representations may not preserve font size, colors, or other visual styling well; the method prioritizes structured content and layout over visual appearance (Supplement C.1, p.13). **Markdown conversion filtering:** Dense technical content requires filtering when formulas cannot be compiled or translated cleanly (Sec. 3.1.1, p.4). **Webpage scraping noise:** Real webpage scraping introduces noise; the paper mitigates this with a "webpage summarization" task that extracts main elements (Sec. 3.1.1, p.4). **Annotation dependency:** DocMark-Instruct context is produced with an external LLM (ChatGPT-3.5) and a specific prompt template (Figure 7, p.16), raising questions about annotation bias and transfer to other instruction styles. > **Contrast to end-to-end VLM OCR:** Unlike end-to-end models that directly generate text from images, DocMark explicitly generates intermediate structured markup before answering questions, providing a form of interpretable context. This trades inference cost (two-round generation) for reduced hallucination and improved grounding in document structure. * * * ## Model ### Base architecture Built on **InternVL** architecture (Sec. 4.1, p.6): - Visual encoder: **InternViT** - LLM backbones: **InternLM-2B** and **InternLM-8B** variants - MLP layers project image features into visual tokens, then concatenate with text tokens ### Input resolution and tokenization strategy Uses InternVL2-style **dynamic resolution tiling**: split an input image into multiple $448 \times 448$ sub-images, with **max 12** sub-images during training (Sec. 4.1, p.7). ### Two-round adaptive context generation Template (Sec. 3.2.1, p.5): **Round 1 (context extraction):** - $Q_1$: "To answer the question: ``, extract relevant context from the image." - $A_1$: `` where `` is one of {txt, txt_gd, md, latex, html, json, tikz}. **Round 2 (answer generation):** - $Q_2$: "Based on the image and extracted context, answer the question: ``." - $A_2$: `` Figure 4 (p.6) visualizes this pretrain-to-finetune flow. * * * ## Data ### DocMark-Pile (pretraining, 3.8M) Six markup targets (Figure 2, p.4): plain text, Markdown, LaTeX, HTML, JSON, TikZ. #### Plain text (TXT) and text grounding (TXT_GD) Natural scene text datasets: TextOCR, COCO-Text, HierText, LSVT, RCTW-17, CTW1500 (Sec. 3.1.1, p.3). Uses bounding boxes to train text grounding: emit text plus coordinates. #### Markdown (dense documents, tables) **Dense documents:** Sourced from arXiv with source and compiled rendering, converted to images and Markdown; samples with unconvertible formulas are filtered (Sec. 3.1.1, p.4). **Tables:** PubTabNet, SynthTabNet, TabRecSet, TableGraph, SciTSR; HTML annotations converted to Markdown (Sec. 3.1.1, p.4). #### LaTeX (math-heavy documents, handwritten formulas) **Math/science textbooks and exam pages:** Uses Mathpix for LaTeX annotations (Sec. 3.1.1, p.4). **Handwriting:** HME100K, MathWriting; also renders LaTeX expressions via Matplotlib and augments with affine transforms and Gaussian noise (Sec. 3.1.1, p.4). #### HTML (webpage screenshot to code and summarization) WebSight synthetic dataset (HTML and screenshots) plus scraped real websites (Sec. 3.1.1, p.4). Adds a "webpage summarization" task to reduce noise by extracting main page elements. #### JSON (KIE and charts) **KIE datasets converted to JSON:** CORD, POIE, SVRD, WildReceipt, XFUND (Sec. 3.1.1, p.5). **Charts represented as structured dictionaries:** Title, source, axis labels, values; uses PlotQA metadata and synthetic chart rendering (Sec. 3.1.1, p.5). #### TikZ (scientific/geometry diagrams) Uses DaTikZv1 and DaTikZv2; also converts textbook figures to TikZ with an existing model, recompiles and filters failures to ensure image--code consistency (Sec. 3.1.1, p.5). ### DocMark-Instruct (fine-tuning, 624k) Domain breakdown (Table 3, p.13): | Domain | Datasets | Size | | ------ | -------- | ---- | | Text-based QA | TextVQA, STVQA, EST-VQA | 63k | | Document-based QA | DocVQA, InfoVQA, Docmatix, in-house | 105k | | Chart-based QA | ChartQA, PlotQA, DVQA | 267k | | KIE | POIE, SROIE, FUNSD, XFUND | 3k | | Webpage-based QA | WebSRC, in-house | 63k | | Mathematical QA | Geo170k, Geometry3k, MultiMath, in-house | 117k | #### Construction pipeline (Sec. 3.2.2, p.6) 1. Pretrained DocMark converts document images to markup. 2. ChatGPT-3.5 receives: question, known answer, and full markup; it extracts the minimal relevant snippet to justify the answer. 3. If markup does not support the question, output is "unclear" (Figure 7, p.16). * * * ## Algorithms / Training ### Stage 1: Pretraining (DocMark-Pile) - Duration: **1 epoch** - Learning rate: **$2 \times 10^{-5}$** - Total batch size: **128** - Adds **LLaVA-558K** image-caption data to maintain alignment between visual and text tokens (Sec. 4.1, p.7) ### Stage 2: Fine-tuning (DocMark-Instruct and general instruction data) - Duration: **1 epoch** - Learning rate: **$1 \times 10^{-5}$** - Total batch size: **128** - Mixes DocMark-Instruct with ShareGPT4V and text-only multi-turn conversation data to preserve general chat ability (Sec. 4.1, p.7) ### Objective Concatenates the two-round dialogue during training and applies cross-entropy on both $A_1$ (context) and $A_2$ (final answer) (Sec. 3.2.3, p.6). ### Inference Run $Q_1$ to generate context, then $Q_2$ to generate answer (Sec. 3.2.3, p.6). * * * ## Evaluation ### Markup generation and specialized tasks (Table 1, p.7) DocMark-2B versus DocMark-8B key reported numbers: | Task | Metric | DocMark-2B | DocMark-8B | | ---- | ------ | ---------- | ---------- | | Text recognition | Benchmark score | 247 | 252 | | HME100K | Accuracy | 67.2 | 73.4 | | ChartQA-SE | AP@strict | 73.8 | 75.4 | | ChartQA-SE | AP@slight | 85.2 | 85.5 | | ChartQA-SE | AP@high | 86.5 | 87.1 | | FUNSD | F1 | 64.1 | 68.5 | | SROIE | F1 | 54.2 | 59.6 | | DaTikZ-test | Consistency | 0.39 | 0.64 | ### Downstream document understanding (Table 2, p.8) See [What are the outcomes/limitations?](#what-are-the-outcomeslimitations) section for comprehensive benchmark results. ### Ablations Removing DocMark-Pile materially reduces performance, especially for text-dense and structured document QA; adding DocMark-Instruct improves further (Sec. 4.4, p.8; Figure 5). Supplement Table 5 (p.14) suggests naive CoT prompting harms vanilla models (not trained for context extraction), while DocMark training makes CoT-like inference beneficial. ### Token efficiency Figure 6 (p.15): Image tokens dominate total token count under dynamic resolution; context tokens are a relatively small increment compared to image and original QA tokens. * * * ## Hardware / Production The paper specifies dynamic image tiling ($448 \times 448$, max 12 tiles) and batch sizes, but does not provide GPU type, wall-clock training time, or throughput/latency measurements (Sec. 4.1, p.7). * * * ## Are model weights shared publicly? **Yes.** The authors point to public releases on [GitHub](https://github.com/Euphoria16/DocMark) and host models via a [Hugging Face collection](https://huggingface.co/collections/HanXiao1999/docmark-adaptive-markup-language-generation-68284f713d046200532085e3). ### Under what licensing? **Unclear.** The public repository and Hugging Face model cards do not explicitly state a license. On GitHub, absence of a license means default copyright applies (authors retain all rights; others are not granted reuse, derivative, or distribution rights). Hugging Face instructs users to "seek out and respect a project's license"; if none is declared, permissions are not clearly communicated. * * * ## Is data shared publicly? **Yes.** DocMark-Pile and DocMark-Instruct datasets appear to be posted publicly on [Hugging Face](https://huggingface.co/collections/HanXiao1999/docmark-adaptive-markup-language-generation-68284f713d046200532085e3). The paper states DocMark-Pile is built from "publicly available datasets" plus synthetic methods. ### Under what licensing? **Unclear.** The dataset pages and cards do not explicitly state a license. Practically, DocMark-Pile aggregates many upstream sources (arXiv-derived content, multiple public table datasets), so downstream use may also depend on each upstream dataset's license even if the aggregate were later licensed. * * * _Note: This analysis follows the Bevaya research note guidelines and classification taxonomy. For academic or production use, consult the original paper and verify claims through independent evaluation._