# Byte Latent Transformer > Entropy-based dynamic byte patching replaces fixed tokenization for robust, vocabulary-free language modeling - Authors: Yosh - Published: 2025-04-25 - Source: Bevaya Labs, --- # Byte Latent Transformer: Patches Scale Better Than Tokens **Paper**: [Byte Latent Transformer: Patches Scale Better Than Tokens](https://aclanthology.org/2025.acl-long.453/) **Presenter**: Hunter **Note Taker**: Yosh **Date**: April 25, 2025 --- ## Overview Replaces fixed-vocabulary tokenization with entropy-based grouping of bytes into variable-length patches. Patch boundaries are chosen by a small byte-level autoregressive model using either a global entropy constraint or an approximate monotonic constraint; an incremental patcher ensures that patching on streaming input matches patching on the full sequence. The model stack uses two small transformers (local encoder/decoder) plus a larger latent transformer operating over patches. Reported performance is comparable to Llama 3 on language benchmarks and stronger in settings like character permutation. A wrapper path is described: initialize from an existing model (e.g., Llama 3) and add a BLT wrapper. --- ## Novelty - Dynamic, vocabulary-free byte patching driven by predicted next-byte entropy (vs. static tokenizers, BPE, or space-based rules). - Incremental patcher property for streaming/online encoding consistency. - Architecture combining local transformers for patch formation with a latent transformer consuming patch sequences. - Weight-initialization strategy that reuses pretrained weights and layers them under a BLT wrapper. --- ## Learnings - Tokenizers’ brittleness (noise sensitivity, orthographic gaps, multilingual inequity) is a motivating factor for patching. - BLT matches or exceeds Llama 3 on several evaluations; particularly robust to character-level perturbations. - Edge behavior observed: patch lengths can expand on repeated or similarly structured phrases (e.g., multiple-choice options). - The wrapper approach suggests compatibility with existing model ecosystems while moving toward tokenizer-free inputs. --- ## Notes for Application - Patch-based byte encodings could be feasible for noisy or multilingual document text where tokenizers struggle. - The incremental patcher may support streaming OCR ingestion while keeping encodings consistent across chunks. - A wrapper initialized from existing Llama-based checkpoints offers a migration path for current document models; worth testing alongside standard tokenizers. - Monitor repeated, formulaic sections in forms (e.g., option lists): similar structure may trigger longer patches; evaluate effects on latency and memory.