# NVIDIA Nemotron Parse 1.1: Lightweight End-to-End Document Parsing > Analysis of Nemotron Parse 1.1, an 885M encoder-decoder VLM for document parsing with token-compressed variant and multi-token inference - Authors: Hunter Heidenreich - Published: 2025-11-01 - Updated: 2026-03-04 - Paper: Nemotron Parse 1.1 Paper - Model: Nemotron Parse v1.1 Model - Model: Nemotron Parse v1.1-TC Model - Demo: NIM Container - Source: Bevaya Labs, --- ## TL;DR Nemotron Parse 1.1 is a lightweight 885M-parameter encoder-decoder VLM for end-to-end document parsing that outputs formatted text (Markdown/LaTeX), bounding boxes, and semantic classes in reading order. A token-compressed variant (Nemotron-Parse-1.1-TC) offers faster inference with minimal quality trade-offs. The authors report competitive OCR and table extraction performance across several benchmarks and release model weights plus an optimized NIM container. * * * ## What kind of paper is this? **Dominant:** $\Psi_{\text{Method}}$ This paper introduces an updated end-to-end document parsing model (v1.1) with specific architectural innovations: a token-compressed speed variant (TC), multi-token inference training, and removal of decoder positional embeddings. The core contribution is the method itself: a single model interface that produces formatted text, bounding boxes, and semantic classes through prompt-controlled outputs. **Secondary:** $\Psi_{\text{Resource}}$ The work releases two model variants with weights, a production-ready NIM container, and a subset of the training data (Nemotron-VLM-v2 dataset). It provides the NVpdftex pipeline for synthetic data generation, making the resource contribution significant. * * * ## What is the motivation? Document OCR for downstream LLM and retrieval workflows requires more than plain text extraction. Systems need layout preservation, reading order, semantic block types (captions, footnotes, section headers), table structure, mathematical formulas, and multi-column/page handling. Pipeline approaches can be brittle and introduce latency through multiple stages, while existing end-to-end models may underperform on specific subtasks when asked to produce all outputs simultaneously. Nemotron Parse targets comprehensive extraction in a single model with good throughput characteristics. * * * ## What is the novelty? **Unified output interface:** A single model produces (a) formatted text in Markdown with LaTeX for formulas/tables, (b) bounding boxes with relative coordinates, and (c) semantic classes for layout elements. Prompts control which outputs are generated, enabling eight valid combinations from three binary axes. **Token-compressed variant (TC):** Applies additional pixel-shuffle operations to reduce vision token sequence length, improving inference speed with what the authors describe as "minimal quality degradation." **Decoder without positional embeddings:** The decoder is trained and evaluated without positional embeddings to support longer-context inference. The authors argue that causal masking provides positional cues and visual tokens already encode 2D spatial structure, avoiding interference with document layout. **Multi-token inference training:** The model is trained to predict $m$ tokens per decoding step using additional linear projection heads. During inference, this operates in greedy mode without token verification. The authors claim this training strategy benefits even single-token inference quality. * * * ## What experiments were performed? **Internal reading-order/OCR test set (Table 2):** Evaluation on 789 human-labeled PDF pages from magazines, books, and Common Crawl. Metrics are word error rate (WER) and F1 score, comparing against Kosmos-2.5 and GOT in plain and markdown modes. **GOT benchmark (Table 3):** OCR accuracy and reading-order metrics comparing Nemotron Parse variants against multiple systems including Gemini Flash 2.0, Marker, SmolDocling, and others. **OmniDocBench v1.0 English subset (Table 4):** Category-level metrics for overall, text, formula, table, and order accuracy across many models. **Table extraction benchmarks (Tables 5-6):** TEDS and S-TEDS metrics on RD-TableBench, PubTabNet, and OmniDocBench tables (Table 5). RD-TableBench "table similarity" scores comparing against Reducto, cloud vendor OCR, and several LLM-based parsers (Table 6). **Multilingual OCR evaluation (Table 7):** WER and F1 scores across multiple languages using an NVpdftex-derived test set of 10,000 scientific dense documents per language. **Throughput measurement (Table 8):** Tokens per second and approximate pages per second on a single H100 GPU for both base and TC variants. * * * ## What are the outcomes/limitations? **Key outcomes (as reported):** Internal test set results show Nemotron-Parse-MIP and Nemotron-Parse-TC-MIP achieving lower WER and high F1 relative to Kosmos-2.5 and GOT (Table 2). On the GOT benchmark, both variants place in the top tier, with the authors noting only Gemini Flash 2.0 outperforming them (Table 3). Table extraction achieves competitive TEDS/S-TEDS scores, with RD-TableBench TEDS in the mid-80s (Table 5) and table similarity around mid-80s compared to other systems (Table 6). The TC variant improves throughput from 3800 to 4500 tokens/sec (Table 8), which the authors interpret as roughly 4 vs. 5 pages/sec for average document lengths. **Limitations and caveats explicitly noted:** **Formula scoring artifact on OmniDocBench:** Since the model outputs Markdown format, simple equations may not be wrapped in LaTeX math delimiters. These are penalized in the formula category even when represented correctly in Markdown, artificially lowering formula scores. **Multilingual scope:** The authors report stronger performance for Chinese, Japanese, and Korean in scientific PDFs and standard documents, but limited support for "in-the-wild" images and documents in those languages. **Multi-token inference risks:** The greedy decoding approach operates "without token verification," which could amplify errors in dense text regions. The paper does not quantify this potential error propagation. **Missing reproducibility details:** The paper lacks a comprehensive training hyperparameter table (optimizer configuration, learning rate schedule, batch sizes, training steps) and detailed ablation studies for individual design choices beyond high-level architectural descriptions. * * * # Reproducibility Details ## Model ### Architecture The model has **885M total parameters** with a compact **256M-parameter language decoder**. The vision encoder initializes from **RADIO** using a **ViT-H/16** backbone (**657M parameters**). A vision neck applies horizontal convolution kernels of size $1 \times 4$ with stride $1 \times 4$, reducing both hidden dimensionality and sequence length. For a $1648 \times 2048$ image, the sequence length reduces to **3200 tokens**, and the RADIO summary token is concatenated. The **token-compressed (TC) variant** applies an additional pixel-shuffle operation to the compressed sequence, reducing it to **833 tokens**, described as a total $\times 16$ reduction in sequence length. The decoder uses an **mBART-style architecture** reduced to **10 layers** with tied weights. ### Positional embeddings The decoder is trained and evaluated **without positional embeddings** to enable large-context inference. The authors argue that causal masking provides implicit positional cues, and visual tokens already encode 2D spatial structure, so explicit position encodings are unnecessary and potentially harmful for document layout understanding. ### Multi-token inference For predicting $m$ tokens simultaneously, the training procedure adds $(m-1) \times 2$ additional linear projection layers. Training uses teacher forcing for embeddings of later tokens. Inference operates in greedy mode without verification of the predicted tokens. * * * ## Data ### Training data blend The training mixture combines synthetic, public, and human-annotated sources. Table 1 in the paper lists examples including: - Multilingual arXiv (8.3M samples) - Wikipedia OCR data (9.5M samples) - Multilingual synthetic OCR data (3.5M samples) - Table datasets: PubTables, FinTabNet, TabRecSet - DocLayNet - Common Crawl samples ### NVpdftex pipeline The core dataset generation pipeline is inspired by Nougat-style LaTeX rendering but claims tighter alignment through TeX Live instrumentation. The pipeline intercepts node/character creation and page output to extract character-level bounding boxes, semantic labels, and reading order directly from the typesetting engine. A repository link is provided in the paper. For multilingual support, machine translation is applied to NVpdftex content into **six languages**, with LaTeX-level augmentations including font variations, color modifications, and layout changes. ### Data augmentation **DocLayNet augmentation:** The authors augment DocLayNet with autolabeled reading order, text inside images, and Markdown formatting. This includes LaTeX formatting for tables and formulas. **Common Crawl annotation:** Common Crawl samples receive human annotation for plaintext, bounding boxes, and semantic classes. Additional autolabeling covers text-inside-images and formatting. Low-quality predictions are filtered using edit distance heuristics. * * * ## Algorithms / Training ### Prompts and output specification The model uses three prompt axes that define outputs, yielding eight valid combinations: 1. **Text formatting:** ``, ``, `` 2. **Bounding boxes:** ``, `` 3. **Classes:** ``, `` (used only with bbox) The **maximal-information prompt (MIP)** is: ` `. ### Output format Bounding boxes use **relative coordinates** in a **1024 $\times$ 1280** scale. The output schema for MIP is: ``` (text) ``` ### Reading order specification **Base model:** Uses canonical ordering starting with Page-Header, then Text/Section-Header/List-Item/Title/Formula in reading order, followed by Footnotes/Page-Footers/Tables/Pictures/Captions at the end. **TC model:** The authors claim improved ordering that places "floating" elements (tables, figures) within the natural page reading flow rather than at the end. * * * ## Evaluation ### Internal reading-order/OCR test set (Table 2) Evaluated on 789 human-labeled PDF pages from magazines, books, and Common Crawl. Metrics reported: WER (lower is better) and F1 (higher is better). Comparisons include masking/unmasking headers and footers depending on baseline model capabilities. ### GOT benchmark (Table 3) OCR accuracy and reading-order metrics comparing multiple systems. Nemotron Parse 1.1 and TC variants show strong performance in this benchmark. ### OmniDocBench v1.0 (Table 4) English subset evaluation with category-level metrics: overall, text, formula, table, and order. The paper notes the formula scoring artifact: Markdown-formatted simple equations without LaTeX delimiters are penalized even when correct. ### Table extraction benchmarks (Tables 5-6) **Table 5:** TEDS and S-TEDS on RD-TableBench, PubTabNet, and OmniDocBench tables. **Table 6:** RD-TableBench "table similarity" scores comparing Nemotron Parse against Reducto, cloud vendor OCR systems, and multiple LLM-based document parsers. ### Multilingual OCR (Table 7) WER and F1 scores per language on NVpdftex-derived dense scientific documents. Reported values range around WER 0.03--0.06 and F1 0.96--0.98 across tested languages. ### Qualitative examples The paper includes figures showing: - **Figure 1 (p.10):** Layout analysis with bounding boxes and semantic classes - **Figure 2 (p.11):** OCR with mathematical formula formatting - **Figure 3 (p.11):** Complex table extraction rendered as LaTeX * * * ## Hardware / Production Model weights are released in **fp32** and **bf16** formats with **vLLM** support. A **NIM container** is also available for production deployment. Throughput measurements on a single **H100 GPU** in **bf16** precision: - **Nemotron Parse 1.1:** 3800 tokens/sec - **Nemotron Parse 1.1-TC:** 4500 tokens/sec The authors interpret these speeds as approximately **4 pages/sec** vs. **5 pages/sec** based on analysis of 10,000 pages averaging 1000 tokens per page.