# CC-OCR: A Comprehensive and Challenging OCR Benchmark for Evaluating Large Multimodal Models in Literacy > CC-OCR is a four-track OCR benchmark covering multi-scene text reading, multilingual OCR, document parsing, and key information extraction across 39 subsets and 7,058 fully annotated images, designed to evaluate large multimodal models on literacy tasks. - Authors: Hunter Heidenreich - Published: 2026-03-30 - Paper: Preprint (arXiv) - Paper: Published (ICCV 2025) - Benchmark: CC-OCR Benchmark (Code + Data) - Dataset: CC-OCR Dataset (HuggingFace) - Source: Bevaya Labs, --- ## TL;DR CC-OCR is a four-track OCR evaluation benchmark from Alibaba Group and South China University of Technology covering 39 subsets, 7,058 fully annotated images, and 10 languages. It systematically evaluates large multimodal models (LMMs) on text reading, multilingual OCR, document parsing (including tables and formulas), and key information extraction, revealing specific weaknesses in text grounding, multi-orientation robustness, and structured output generation. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Evaluation}}$: the primary contribution is a new benchmark and evaluation protocol. The paper introduces CC-OCR as a measurement framework for assessing LMM literacy capabilities across multiple tasks, scenarios, and languages. The headline novelty is "how we measure" these models, and the paper's center of gravity lies in the benchmark design, annotation methodology, and systematic model evaluation rather than in proposing a new model or training recipe. **Secondary:** $\Psi\_{\text{Resource}}$: the benchmark itself represents a reusable evaluation asset: 7,058 annotated images spanning newly collected and re-annotated data, including the SOLD dataset (re-annotated open-category KIE data), all released to the research community. ## What is the motivation? Large multimodal models have demonstrated strong general visual understanding, but their performance on OCR-centric tasks that require fine-grained text perception, structured output generation, and multilingual reading is less well understood. Existing benchmarks for evaluating LMMs on OCR tasks suffer from several limitations: narrow scope (focusing on only one subtask such as line-level text recognition or formula detection), restricted language coverage (many are English-only or at best English and Chinese), and inconsistent annotation standards that make cross-model comparison difficult. Benchmarks such as OCRBench emphasize line-level recognition, while FOX and DocLocal4K focus on document images but omit natural scenes and are restricted to one or two languages. KOSMOS2.5-Eval covers parsing but not key information extraction. The gap is a comprehensive benchmark that can simultaneously assess all major OCR-relevant capabilities of general-purpose LMMs. CC-OCR is constructed to fill that gap. ## What is the novelty? The main contribution is the CC-OCR benchmark itself, characterized by three design principles: diversity of scenarios, practicality through use of real-world captured images, and deliberate challenge construction targeting known weak points of LMMs. **Four tracks:** The benchmark organizes evaluation into (1) multi-scene text reading (natural scenes, documents, web/UGC images in English and Chinese); (2) multilingual text reading across ten languages from four language families; (3) document parsing covering full-page document structuring, table recognition, and formula recognition; and (4) key information extraction in both constrained-category and open-category settings. **Challenge dimensions** are orthogonal to the track structure and include orientation-sensitivity (multi-directional and curved text), grounding (predicting word-level bounding boxes), natural noise, artistic/stylized text, multilingual diversity, mathematical and chemical formula expressions, and structured output formats (LaTeX, HTML, JSON, SMILES). **SOLD dataset:** A notable sub-contribution is SOLD (Structurally-rich Open Layout Dataset), created by re-annotating two existing open-category KIE datasets (SIBR and HUST-CELL) with a bottom-up annotation scheme that produces end-to-end JSON representations, including hierarchical structures and table-format key-value groups. The original link-based annotation format of those datasets was unsuitable for directly evaluating LMMs. **Evaluation metrics** are tailored per track. For OCR text sequences, the authors use a full-text multi-set matching metric (Eval-Trans) that accounts for the fact that LMMs may predict text in different orders: $$ \text{Recall} = \frac{\sum\_{i}^{N} \min(c\_i, c'\_i)}{\sum\_{i}^{N} c\_i}, \quad \text{Precision} = \frac{\sum\_{i}^{N} \min(c\_i, c'\_i)}{\sum\_{i}^{N} c'\_i} $$ where $u\_i$ is a basic unit (character for CJK scripts, word for Latin/Cyrillic/Arabic), $c\_i$ is its count in the ground-truth sequence, and $c'\_i$ is its count in the predicted sequence. F1 is computed from recall and precision. For document content structuring and formula recognition, normalized edit distance (NED) is used: $$ \text{NED} = \frac{1}{N} \sum\_{i=1}^{N} \left(1 - \frac{\text{EditDist}(P\_i, G\_i)}{\max(\text{len}(P\_i), \text{len}(G\_i))}\right) $$ For table parsing, the benchmark adopts Tree Edit Distance-based Similarity (TEDS), which accounts for both structural similarity and cell content accuracy: $$ \text{TEDS}(T\_{pred}, T\_{gt}) = 1 - \frac{\text{EditDist}(T\_{pred}, T\_{gt})}{\max(|T\_{pred}|, |T\_{gt}|)} $$ For key information extraction, field-level F1 score is used, where a key-value pair is treated as a single field and any character mismatch counts as a failed extraction. The paper also introduces a repetition ratio $R\_{rep}$ to quantify hallucination in the form of repeated output, defined as the fraction of images for which the length of the longest continuous repetitive string exceeds 25% of the total output length. ## What experiments were performed? Nine LMMs are evaluated: five generalist models (GPT-4o-2024-08-06, Gemini-1.5-Pro-002, Claude-3.5-Sonnet-20241022, Qwen2-VL-72B, InternVL2-76B) and four specialist models (KOSMOS2.5, TextMonkey, Florence-2, GOT). Specialist models are those trained specifically for OCR or document understanding tasks. The evaluation covers all four tracks. For multi-scene OCR, each model is prompted to output the full text content of the image. For multilingual OCR, prompts are in the target language. For document parsing, models are prompted to produce structured output (LaTeX for documents and formulas, HTML for tables, SMILES for molecular formulas). For KIE, models must fill in a provided JSON schema. Three additional analyses probe specific challenge dimensions: text grounding (word-level bounding box prediction on English multi-scene subsets), multi-orientation robustness (rotating multi-scene images by 0, 90, 180, and 270 degrees), and repetition hallucination. ## What are the outcomes/conclusions? **Overall ranking:** Gemini-1.5-Pro achieves the highest average score (73.0), ranking first in three of four tracks. Qwen2-VL-72B ranks second overall (68.7) and first in the KIE track. Generalist LMMs consistently outperform specialist models, which the authors attribute to their larger parameter counts and broader training data. **Multi-scene OCR:** Gemini-1.5-Pro scores 83.25, followed by Qwen2-VL-72B at 77.95. Performance in natural scenes is approximately 15 percentage points lower than in document scenes for most models, suggesting that text in natural environments remains a more difficult regime. **Multilingual OCR:** The performance gap across languages is substantial. Asian languages (Japanese, Korean, Vietnamese) and Arabic show the weakest results across most models. Japanese scores are particularly low, which the authors attribute to the prevalence of vertical text in their collected images. Latin-family languages are generally easier, though German and Italian show lower scores, possibly due to special characters. **Document parsing:** Gemini-1.5-Pro leads with 62.37. Even the top-performing model scores below 70 on individual document parsing sub-tasks (67.17 on document content structuring and 67.93 on table recognition), which the authors note as evidence of the benchmark's difficulty. All models perform substantially better on English documents than Chinese documents; Gemini-1.5-Pro, for instance, scores about 5 percentage points higher on English tables than Chinese tables. Handwritten formula recognition is a clear weak point: nearly all models score below 50, with molecular formula recognition being a partial exception for Gemini-1.5-Pro. **Key information extraction:** Qwen2-VL-72B (71.76 F1) and Gemini-1.5-Pro (67.28 F1) lead. GPT-4o and Claude-3.5-Sonnet perform notably weaker on the EPHOIE dataset. The highest-performing model still falls short of practical application thresholds, which the authors attribute to real-world noise, complex document structures, and the strict JSON output requirement. **Text grounding:** Only four of the nine evaluated models support word-level grounding output. Even for these, performance drops sharply relative to recognition-only results. Gemini-1.5-Pro achieves the highest grounding score at 60.98%; all others remain below 50%. This finding indicates that fine-grained spatial understanding remains a major gap. **Multi-orientation robustness:** Most models degrade significantly when images are rotated. InternVL2-76B shows the largest drop (38.92 percentage points), followed by GOT (34.86). Gemini-1.5-Pro is a notable outlier, declining by only 3.80 points, suggesting substantially better rotation invariance. **Repetition hallucination:** TextMonkey produces repetitive output in 33.93% of images, and KOSMOS2.5 in 10.64%. Among generalist models, InternVL2-76B shows the highest repetition rate at 5.94%, while Claude-3.5-Sonnet shows the lowest at 0.09%. The authors identify this as a practical failure mode worth monitoring. **Failure mode observations:** GPT-4o exhibits both hallucination (inventing plausible but incorrect text for long fields such as addresses) and instruction noncompliance (reformatting dates despite explicit prompts). These qualitative failures are documented alongside quantitative results. ## Reproducibility ### Models No new model is introduced. The benchmark evaluates nine existing LMMs. The commercial APIs evaluated are GPT-4o-2024-08-06, Gemini-1.5-Pro-002, and Claude-3.5-Sonnet-20241022. The open-source models are KOSMOS2.5, TextMonkey, Florence-2, GOT, InternVL2-76B (76 billion parameters), and Qwen2-VL-72B (72 billion parameters). Model-specific details (parameter counts, architecture) are described in the original model papers and not repeated here. ### Algorithms Evaluation uses the metrics described above (Eval-Trans F1, NED, TEDS, field-level F1, $R\_{rep}$). Normalization code for the KIE evaluation and evaluation scripts for all tracks are released at [AlibabaResearch/AdvancedLiterateMachinery](https://github.com/AlibabaResearch/AdvancedLiterateMachinery/tree/main/Benchmarks/CC-OCR) under MIT license, with VLMEvalKit integration supported via the HuggingFace dataset. The annotation pipeline for the SOLD dataset uses a bottom-up re-annotation scheme with rule-based multi-round quality rectification, LLM-assisted error correction, key normalization, and secondary manual review. ### Data CC-OCR comprises 7,058 fully annotated images across four tracks. Approximately 41% of the data are sourced from real-world applications and released for the first time. Data sources fall into three categories: existing benchmarks with qualified annotations (open-source, re-annotated as needed), newly collected data, and re-annotated data. All document parsing data and all Chinese multi-scene data are newly collected. KIE constrained-category data is re-annotated from SROIE, CORD, EPHOIE, and POIE. The open-category SOLD dataset is re-annotated from SIBR and HUST-CELL. The benchmark data is publicly available on HuggingFace at [wulipc/CC-OCR](https://huggingface.co/datasets/wulipc/CC-OCR) under an MIT license; this hosts a TSV-format version of the data integrated with VLMEvalKit. The dataset combines images from multiple source licenses: subsets derived from open-source benchmarks retain their original licenses (e.g., TotalText, IC15, CORD, SROIE are individually licensed), while newly collected and re-annotated images (including all document parsing data, Chinese multi-scene data, and SOLD) are released by the authors under the MIT license. The overall dataset package on HuggingFace carries MIT, though downstream users should verify the original source licenses for any open-source subsets they use independently. ### Evaluation Benchmark statistics: multi-scene OCR has 2,750 images; multilingual OCR has 1,500 images (150 per language); document parsing has 800 images (300 documents, 300 tables, 200 formulas); KIE has 2,008 images (1,008 constrained, 1,000 open-category). For text evaluation, basic units are defined per script: characters for CJK languages (Chinese, Japanese, Korean), words for Latin/Cyrillic/Arabic. The repetition threshold $T\_{rep}$ is set to 5 (a continuous unit must appear more than 5 times consecutively to qualify as repetitive). No statistical uncertainty measures (confidence intervals, significance tests, multiple-seed runs) are reported. The evaluation is a single-pass inference with no reported variance. A limitation noted by the authors is that most specialist models cannot produce LaTeX or HTML output, so the document parsing track primarily reflects generalist model performance. Results for specialist models on that track are incomplete or require post-hoc conversion (indicated by asterisks in Table 4). ### Hardware The paper does not report hardware specifications, inference latency, GPU counts, or cost estimates for the model evaluations. No deployment or inference requirements are discussed. ## BibTeX ```bibtex @InProceedings{Yang_2025_ICCV, author = {Yang, Zhibo and Tang, Jun and Li, Zhaohai and Wang, Pengfei and Wan, Jianqiang and Zhong, Humen and Liu, Xuejing and Yang, Mingkun and Wang, Peng and Bai, Shuai and Jin, Lianwen and Lin, Junyang}, title = {CC-OCR: A Comprehensive and Challenging OCR Benchmark for Evaluating Large Multimodal Models in Literacy}, booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)}, month = {October}, year = {2025}, pages = {21744-21754} } ```