# ModernBERT > Encoder-only transformer trained on 2T tokens with 8192 context length, bringing modern architecture improvements to BERT-family models - Authors: Hunter - Published: 2025-01-03 - Source: Bevaya Labs, --- # Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference **Paper**: [Smarter, Better, Faster, Longer](https://aclanthology.org/2025.acl-long.127/) **Presenter**: Hunter **Note Taker**: Hunter **Date**: January 3, 2025 --- ## Overview ModernBERT is an updated encoder-only (bidirectional) transformer trained on 2 trillion tokens with native 8,192-token context support. It applies a range of modern architecture improvements (Flash Attention 2, RoPE positional embeddings, GeGLU activations, and alternating local/global attention) to the BERT family. The result is strong performance on classification and retrieval tasks across text and code, while being faster and more memory-efficient than prior encoders. --- ## Novelty - Flash Attention 2 for memory-efficient long-context attention. - Rotary positional embeddings (RoPE) replacing absolute position embeddings, enabling length generalization. - GeGLU activations replacing standard GELU in the feed-forward layers. - Alternating local (128-token window) and global attention layers to reduce cost at long contexts while preserving global information. - Trained on 2T tokens including code, representing a large training budget increase over prior BERT-family models. - Native 8,192 sequence length support, vs. 512 for original BERT. --- ## Learnings - Modern architecture improvements (RoPE, Flash Attention, GeGLU) collectively produce a substantial Pareto improvement over older encoders on both speed and accuracy. - Long-context support is practically important: many real-world retrieval and classification tasks involve documents that BERT's 512-token limit truncates. - Including code in pretraining improves performance on code-related tasks and does not harm general language understanding. - The alternating local/global attention scheme is a practical efficiency trade-off that retains most of the accuracy of full attention at longer contexts. --- ## Notes for Application - An 8,192-token encoder would directly benefit document-level extraction tasks where pages or multi-page sections exceed 512 tokens; worth evaluating as a backbone or retriever. - RoPE-based encoders generalize better to lengths not seen during training, which could help with variable-length insurance forms. - For dense retrieval pipelines (e.g., RAG over policy documents), ModernBERT's multi-vector retrieval improvements may offer gains over older bi-encoder setups. - The code-trained variant could be relevant if structured data (tables, JSON-like content) is extracted from documents, since code pretraining often improves structured understanding.