# DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understanding

> Research note on DocPedia, an OCR-free multimodal model that uses frequency-domain processing (DCT) for high-resolution document understanding.

- Authors: Hunter Heidenreich
- Published: 2023-11-20
- Updated: 2026-03-04
- Paper: DocPedia Paper <https://arxiv.org/abs/2311.11810>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## What kind of paper is this?

-   This is a **big idea paper** in multimodal document understanding.
-   It proposes **DocPedia**, an OCR-free multimodal model, focusing on
    **frequency-domain processing** instead of the pixel-based approach.
-   The paper addresses the challenge of high-resolution document parsing and
    aligns **perception and comprehension** tasks by introducing a dual-stage
    training method.

## What is the motivation?

-   The primary problem is to improve document parsing accuracy in
    high-resolution images while preserving both visual and textual information,
    without relying on OCR.
-   DocPedia's approach could substantially benefit **OCR-free document
    parsing** applications by providing a framework that maintains both
    perception and logical reasoning capabilities.
-   The potential impact includes improved performance in **information
    extraction, visual question answering (VQA)**, and complex multimodal
    document processing tasks, especially in fields needing OCR-free methods for
    dense, complex documents.

## What is the novelty?

-   Traditional document parsing relies on OCR, leading to error accumulation
    and limited comprehension capabilities when handling high-resolution images.
-   Previous models like LLaVAR and UniDoc operate on lower resolutions or do
    not fully integrate language models, which limits comprehension.
-   **Main Contributions**:
    -   Proposes a **frequency domain** approach (Discrete Cosine Transform,
        DCT) to capture more detail at high resolutions with fewer visual
        tokens.
    -   Introduces a **dual-stage training strategy** combining perception and
        comprehension tasks.
    -   Demonstrates **superior performance** on key benchmarks, confirming the
        efficacy of joint learning and the frequency-based approach.

## What experiments were performed?

-   The paper theorizes that processing in the frequency domain captures more
    visual and textual information while minimizing token counts, which is
    effective for high-resolution images.
-   The dual-stage training approach balances lower-level perception tasks
    (e.g., text recognition) and higher-level comprehension tasks (e.g., logical
    reasoning).
-   **Experiments**:
    -   DocPedia achieves **significant accuracy improvements** (e.g., 40% in
        DocVQA, 28% in FUNSD) over state-of-the-art methods on high-resolution
        document benchmarks.
    -   Results confirm the benefits of **frequency domain processing** and the
        training approach for handling high-density text and visual elements in
        complex documents.

## Dataset

-   **Pre-training data**:
    -   **PPT images**: 600K images from Common Crawl.
    -   **PDF images**: 325K images from arXiv, each assigned one of five OCR
        tasks (text detection, recognition, spotting, paragraph reading,
        full-text reading).
    -   **Image-caption pairs**: 595K pairs for natural scene perception,
        sourced from LLaVA data.
-   **Fine-tuning data**:
    -   Uses pre-training datasets with additional **VQA datasets** (e.g.,
        DocVQA, ChartVQA) and expanded answers generated by ChatGPT.
    -   **Instruction tuning**: 158K instances from LLaVA instruction-tuning
        data, enriching tasks with semantic reasoning.

## Methodology Highlights

-   **Frequency-domain approach**:
    -   Images converted from RGB to YCbCr, followed by **JPEG DCT extraction**
        to obtain DCT coefficients.
    -   **Downscaling**: Y channel via 8×8 blocks; Cb and Cr channels downscaled
        via 16×16 blocks and then upsampled.
    -   **Concatenation and processing**: Y, Cb, and Cr channels concatenated
        and passed through a 1×1 convolution, then fed into the **Swin
        Transformer**.
-   **Two-stage training strategy**:
    -   **Stage 1**: Text-aware pre-training with frozen LLM (Vicuna), focusing
        on OCR-based tasks to align frequency features with the language model.
    -   **Stage 2**: Context-aware fine-tuning, unfreezing LLM and integrating
        higher-level semantic tasks for robust multimodal comprehension.

## Training Details

-   **Learning rate strategy**: One-cycle strategy with peak learning rates set
    to **1e-3 for pre-training (PT)** and **1e-5 for fine-tuning (FT)**.
-   **Batch sizes**: 64 for PT, 8 for FT.
-   **Optimizer**: AdamW.
-   **Hardware**: Trained on **8 A100 GPUs** for one epoch each in PT and FT
    stages.
-   **Token count vs. Resolution**: 1,600 tokens in **RGB** (1280×1280) achieves
    **29.54% accuracy on DocVQA**; **DCT** (2560×2560) achieves **47.08%
    accuracy**.