# Nougat: Notes

> OCR-free encoder-decoder transformer converting rasterized scientific PDF pages into structured markup preserving math and tables

- Authors: Hunter Heidenreich
- Published: 2023-08-01
- Updated: 2026-03-04
- Code: Nougat Code <https://github.com/facebookresearch/nougat>
- Model: Nougat Small Model <https://huggingface.co/facebook/nougat-small>
- Model: Nougat Base Model <https://huggingface.co/facebook/nougat-base>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## TL;DR

Nougat is an OCR-free, encoder--decoder transformer that converts rasterized pages of scientific PDFs (including scanned pages) into a lightweight markup language that preserves mathematical expressions and tables. It couples a Swin Transformer visual encoder with an mBART-style decoder, trains on a large automatically-aligned arXiv/PMC/IDL corpus of $\\approx 8.2$M pages, and substantially outperforms a strong GROBID + LaTeX-OCR baseline on text, math, and tables.

* * *

## What kind of paper is this?

**Dominant:** $\\Psi\_{\\text{Method}}$ (end-to-end OCR-free visual encoder--decoder for document $\\rightarrow$ markup conversion and a repetition-robust decoding scheme).

**Secondary:** $\\Psi_{\\text{Resource}}$ (releases code, trained models, and a dataset generation pipeline pairing PDFs with source markup); $\\Psi_{\\text{Evaluation}}$ (defines a modality-aware evaluation setup for plain text, math, and tables with multiple MT-style metrics).

## What is the motivation?

-   Scientific knowledge is mostly stored as PDFs; embedded text often exists but:
    -   Math and tables lose semantic structure.
    -   Many documents (books, scans) have no embedded text at all.
-   Traditional OCR (e.g., Tesseract) works line-by-line and cannot capture 2D structure needed for math layout (superscripts, fractions, matrices).
-   Existing scholarly pipelines (e.g., GROBID $\\rightarrow$ S2ORC) capture body text but drop or flatten equations and tables.
-   Goal: build a **single model** that:
    -   Works directly from page images (so it supports scanned PDFs).
    -   Outputs a **structured markup** with math and tables.
    -   Is trainable without manual page-level annotations, using PDF+source pairs.

## What is the novelty?

-   **Architecture:** Adapts Donut's OCR-free encoder--decoder design to scientific PDFs, using a **Swin Transformer** encoder and **mBART** decoder specialized to scientific tokenization (architecture diagram and high-level flow in Figure 1, p.2).
-   **Data pipeline** for paired PDF page $\\rightarrow$ markup:
    -   Converts LaTeX $\\rightarrow$ HTML via LaTeXML, then HTML $\\rightarrow$ custom lightweight markdown that preserves math & tables (Figure 3, p.4).
    -   Automatically aligns PDF page text with source paragraphs using TF-IDF + linear SVM page prediction, decision-tree-style splits, and fuzzy matching with a quality threshold; produces $\\approx 7.5$M arXiv pages plus PMC/IDL pages.
-   **Augmentation for scanned-like robustness:** Heavy image augmentation (bitmap, erosion/dilation, affine transformations, grid/elastic distortion, brightness/contrast changes, compression, noise, blur), visualized in Figure 2 (p.3).
-   **Repetition-robust training and decoding:**
    -   Introduces **anti-repetition token perturbation** during training.
    -   Adds a **logit-variance--based heuristic** to detect when decoding has collapsed into a repetition loop and terminate early (Figure 6, p.8).
-   **Modality-aware evaluation:** Separately evaluate "All text", "Plain text", "Math", and "Tables" for both baselines and Nougat, surfacing where the model actually struggles.

## What experiments were performed?

-   **Training corpus:**
    -   **arXiv**: 7,511,745 pages.
    -   **PubMed Central (PMC)**: 536,319 pages (XML-based).
    -   **Industry Documents Library (IDL)**: 446,777 pages with high-quality OCR (plain text only).
    -   Total $\\approx 8.2$M pages (Table A.1, p.13).
-   **Baselines:**
    -   Embedded PDF text (extracted text layer from digital PDFs).
    -   GROBID $\\rightarrow$ XML, with formulas converted back from Unicode to LaTeX; small inline formulas sometimes mis-tagged as text.
    -   GROBID + LaTeX-OCR (pix2tex) on formula bounding boxes to get LaTeX math.
-   **Models:**
    -   **Nougat small**: 250M parameters, max sequence length 3584, 4-layer decoder (pretrained base model).
    -   **Nougat base**: 350M parameters, 10-layer decoder, max sequence length 4096.
-   **Metrics:** Normalized character-level edit distance (Levenshtein), BLEU, METEOR, Precision, Recall, F1 on tokens.
-   **Qualitative evaluation:** Example pages with dense math (Figure 5, p.6) and scanned books/theses (Figures B.1--B.3, pp.14--16) to show performance on non-digital PDFs and mobile-camera scans; additional pages with tables and quantitative results (Figure B.4, p.17).

## What are the outcomes/limitations?

### Outcomes

On the arXiv test set, **Nougat base** improves strongly over both PDF text and GROBID baselines:

-   **All text modality:**
    -   PDF: edit distance 0.255, BLEU 65.8.
    -   GROBID: edit distance 0.312, BLEU 55.6.
    -   Nougat small: edit distance 0.073, BLEU 88.9, F1 92.9.
    -   Nougat base: edit distance 0.071, BLEU 89.1, F1 93.1.
-   **Plain text:** Nougat base achieves edit distance 0.058, BLEU 91.2, METEOR 94.6, F1 95.7.
-   **Math:** lower numbers, as expected; Nougat base still reaches F1 $\\approx 76.5$ with BLEU 56.9 vs GROBID+LaTeX-OCR BLEU 0.3 and F1 9.7.
-   **Tables:** Nougat base edit distance 0.211, BLEU 69.7, F1 78.0.

Qualitatively:

-   For dense math pages, the model outputs LaTeX that renders visually close to the original (Figure 5, p.6); bounding boxes and decorative elements are skipped.
-   For old scanned textbooks and NASA reports, output is noisy but legible and structurally sensible (Figures B.1--B.2).
-   For modern mobile-camera scans of theses, the model handles skew, lighting artifacts, and page curvature reasonably well (Figure B.3).

### Limitations

-   **Repetition / collapse:**
    -   About **1.5%** of test pages fall into repetition loops under greedy decoding; frequency increases on out-of-domain documents.
    -   Heuristic detection helps but does not eliminate the issue; the authors flag this as the main challenge for future work.
-   **Page-local context only:**
    -   Model processes **one page at a time** with no document-level context, causing:
        -   Inconsistent bibliography styles and numbering.
        -   Section numbers that skip or hallucinate.
-   **Language coverage:**
    -   Training data is almost entirely **English**.
    -   Latin-based languages work but special characters are mapped to nearest Latin equivalents.
    -   Non-Latin scripts lead to immediate repetitions or failure.
-   **Data quality:**
    -   Ground truth markup includes artifacts from LaTeXML and splitting heuristics (extra numbering, missing figures/tables, truncated text).
    -   Authors argue that large corpus size compensates, but no ablation on data quality is provided.
-   **Throughput:**
    -   On an NVIDIA A10G (24GB), they can process **6 pages in parallel**; with average $\\approx 1400$ tokens per page, mean generation time is **19.5s per batch** (no inference optimizations).
    -   This is much slower than traditional pipelines (GROBID $\\approx 10.6$ PDFs/s) but works for scanned docs and preserves math.

* * *

## Model

### Architecture

-   **Encoder--decoder transformer** following Donut (Figure 1, p.2).
-   **Input:** rasterized page image $x \\in \\mathbb{R}^{3 \\times H_0 \\times W_0}$ at **96 DPI**.
-   **Preprocessing:**
    -   Crop page margins.
    -   Resize to fixed canvas (H, W) = (896, 672).
    -   If smaller than canvas, pad to fixed size.
-   **Visual encoder:**
    -   Swin Transformer **base** model, initialized from image pretraining.
    -   Splits image into non-overlapping windows; hierarchical self-attention across windows.
    -   Outputs sequence of patch embeddings $z \\in \\mathbb{R}^{d \\times N}$, where $N$ is number of patches.
-   **Text decoder:**
    -   Transformer decoder with cross-attention over encoder outputs; implementation based on **mBART**.
    -   Uses tokenizer from **Galactica** (Taylor et al.) tuned for scientific text (equations, citations).
    -   Autoregressive generation of markup tokens; projection to vocabulary logits $\\ell \\in \\mathbb{R}^v$.
-   **Sequence lengths & sizes:**
    -   Base model: max sequence length **4096 tokens**, 10 decoder layers, total **350M parameters**.
    -   Small model: max sequence length **3584**, 4 decoder layers, total **250M parameters** (starting from pretrained base).
-   **Output format:**
    -   Lightweight markdown-like markup that supports:
        -   Headings.
        -   Inline & display LaTeX math.
        -   LaTeX tables.
        -   Bold/italic.
        -   Algorithms and references (citations as numeric markers).

### Repetition detection (inference-time logic)

-   Let $\\ell_i$ be the **max logit** over the vocabulary for token $i$.
-   Compute sliding-window variance over logits, window size $B = 15$: $\\mathrm{VarWin}\_B[\\ell](x)$.
-   Then compute the variance of this signal from position $x$ to end: $\\mathrm{VarEnd}\_B[\\ell](x)$.
-   If $\\mathrm{VarEnd}\_B[\\ell](x)$ drops below threshold (6.75) and stays low for the rest of the sequence, classify output as collapsed repetition (Figure 6).
-   During incremental decoding:
    -   Use only last 200 tokens and **half the threshold** for early detection.
    -   After generation completes, re-run on full sequence to confirm / re-classify.

* * *

## Data

### Sources and composition

-   **arXiv:**
    -   1,748,201 articles with LaTeX sources and compiled PDFs.
    -   After processing and page-level alignment, yields **7.51M pages** (main training/eval set).
-   **PubMed Central (PMC):**
    -   Open-access non-commercial subset; XML + PDF.
    -   Parsed into the same markup format as arXiv; used mainly for **pretraining** due to noisier math/tables (often embedded as images).
-   **Industry Documents Library (IDL):**
    -   Public health--related industry documents collected by UCSF.
    -   Use OCR-IDL annotations (text only, no formatting) as an additional pretraining corpus to teach basic OCR on scanned docs.

### LaTeX to markup pipeline (arXiv)

(Figure 3, p.4 shows an example of LaTeX $\\rightarrow$ HTML $\\rightarrow$ markdown $\\rightarrow$ PDF.)

1.  **LaTeXML:**
    -   Convert LaTeX sources to HTML5.
    -   Normalize: expand macros, standardize whitespace, add optional brackets, normalize tables, canonicalize references/citations.
2.  **HTML to custom markdown:**
    -   Parse HTML into lightweight markup supporting:
        -   Sections/headings.
        -   Inline/display LaTeX math.
        -   LaTeX tables.
        -   Algorithms and citations.
    -   Remove ambiguity in math where possible but some variability remains (e.g., `\frac` vs `\over`, different bold commands).

### Page splitting and alignment

-   **Figure/table handling:**
    -   Use **pdffigures2** to detect and temporarily remove figures/tables and captions from PDFs.
    -   Match captions back to source via Levenshtein distance on captions; re-insert removed elements at the **end of each page** after splitting.
-   **Bag-of-words page index prediction:**
    -   Extract text lines from PDFs with MuPDF; strip headers/footers/page numbers.
    -   Compute TF-IDF features; train a **linear SVM** to classify lines by page index.
    -   Split LaTeX source into paragraphs; predict page number for each paragraph.
-   **Boundary optimization:**
    -   Predicted page indices ideally form a staircase; noise introduces mismatches.
    -   Use a decision-tree-like search over paragraph indices, minimizing a Gini-style impurity measure to choose splitting boundaries (visualized in Figure 4, p.5).
-   **Fuzzy alignment check:**
    -   Around each predicted split, compare source text with:
        -   Last sentences of previous PDF page.
        -   First sentences of next PDF page.
    -   Use fuzzy string matching (normalized Levenshtein) to score candidate split points.
    -   Keep pages whose **average alignment score $\\ge 0.9$** at both boundaries.
    -   This yields an acceptance rate of about **47% of all pages**.

### Ground truth artifacts

-   LaTeXML may:
    -   Number subsections in the markup even when PDF shows unnumbered headings.
    -   Drop figures/tables or represent equations as images.
-   Page splitting sometimes:
    -   Includes text from previous page.
    -   Cuts off words or misses "invisible" formatting tokens (bold, italics, section markers).
-   PMC inline math often appears as Unicode or italic text; display equations/tables frequently come as images and are ignored.

* * *

## Algorithms / Training

### Image augmentation

To simulate scanned/low-quality docs, apply a **fixed-probability mixture** of augmentations per page (Figure 2, p.3):

-   Bitmap conversion.
-   Erosion / dilation.
-   Affine transforms: shift, scale, rotate.
-   Grid distortion, elastic transform.
-   Random brightness/contrast changes.
-   Image compression artifacts.
-   Gaussian noise, Gaussian blur.

Implemented using **Albumentations** library.

### Anti-repetition token perturbation

During training, to make the decoder more robust to previous token errors:

-   For each training example, sample a random token and replace it with another random token.
-   Repeat replacement while random samples fall below a probability threshold (10%), creating a small number of corrupted tokens.

Authors report:

-   No performance degradation on in-domain data.
-   $\\approx 32\\%$ reduction in failed page conversions due to repetition on out-of-domain documents.

### Optimization

-   **Optimizer:** AdamW.
-   **Training schedule:**
    -   Train for **3 epochs** with effective batch size **192 pages**.
    -   Initial learning rate $5 \\times 10^{-5}$.
    -   Every 15 updates, multiply LR by 0.9996 until reaching final LR $7.5 \\times 10^{-6}$.
    -   Authors mention training instabilities as motivation for the relatively low LR.
-   **Decoding:**
    -   Greedy decoding (no beam search, no sampling) to simplify analysis and avoid exposure to degenerate outputs introduced by stochastic sampling.

* * *

## Evaluation

### Metrics and modalities

-   **Character-level normalized edit distance** (Levenshtein / #chars).
-   **BLEU** and **METEOR** borrowed from MT evaluation.
-   **Precision, Recall, F1** over tokens.
-   Evaluate over:
    -   **All text** (single stream).
    -   **Plain text**.
    -   **Math**.
    -   **Tables**.

### Main quantitative results (arXiv test set)

From Table 1 (p.7):

-   **All text (no modality split)**
    -   PDF: edit distance 0.255, BLEU 65.8, METEOR 82.1, F1 79.2.
    -   GROBID: edit distance 0.312, BLEU 55.6, METEOR 71.9, F1 73.0.
    -   Nougat small: edit distance 0.073, BLEU 88.9, METEOR 92.8, F1 92.9.
    -   Nougat base: edit distance 0.071, BLEU 89.1, METEOR 93.0, F1 93.1.
-   **Tables**
    -   GROBID: edit distance 0.626, BLEU 25.1, METEOR 64.5, F1 69.7.
    -   Nougat small: edit distance 0.220, BLEU 68.5, METEOR 78.6, F1 77.3.
    -   Nougat base: edit distance 0.211, BLEU 69.7, METEOR 79.1, F1 78.0.
-   **Plain text**
    -   GROBID + LaTeX-OCR: edit distance 0.363, BLEU 57.4, METEOR 69.2, F1 75.9.
    -   Nougat small: edit distance 0.058, BLEU 91.0, METEOR 94.3, F1 95.7.
    -   Nougat base: edit distance 0.058, BLEU 91.2, METEOR 94.6, F1 95.7.
-   **Math**
    -   GROBID + LaTeX-OCR: edit distance 0.727, BLEU 0.3, METEOR 5.0, F1 9.7.
    -   Nougat small: edit distance 0.117, BLEU 56.0, METEOR 74.7, F1 76.9.
    -   Nougat base: edit distance 0.128, BLEU 56.9, METEOR 75.4, F1 76.5.

### Qualitative examples

-   **Figure 5 (p.6):** side-by-side original vs. Nougat-rendered math-heavy page (Sorscher et al.). Equations, alignment, and text are reproduced with minor markup differences; decorative equation boxes are skipped.
-   **Appendix B figures (pp.14--17):**
    -   Old calculus textbook (Figure B.1): shows OCR errors on barely legible exponents and repetition loops triggered by punctuation mistakes.
    -   NASA report (Figure B.2): longer, dense paragraphs; model mostly tracks text accurately, though math and typographic details occasionally degrade.
    -   Mobile-scanned thesis pages (Figure B.3): demonstrates robustness to camera artifacts.
    -   Pages with tables (Figure B.4): show how tables and plots are represented in the markup.

* * *

## Hardware / Production

-   **Training hardware:** not specified in detail (no GPU counts or training hours given).
-   **Inference throughput** (Section 5.5):
    -   Machine: NVIDIA **A10G** GPU with 24GB VRAM.
    -   Batch: 6 pages in parallel.
    -   Average output length: $\\approx 1400$ tokens/page.
    -   Mean generation time: **19.5 seconds per 6-page batch** (no optimizations).
-   **Comparison to traditional pipelines:**
    -   GROBID: $\\approx 10.6$ PDFs/s on unspecified hardware.
    -   Nougat is **much slower** but supports scanned docs and provides semantically richer outputs for math and tables.