Byte Latent Transformer (BLT)
A Byte Latent Transformer (BLT) is a tokenizer-free language model architecture that processes raw UTF-8 bytes directly by dynamically grouping them into variable-length patches based on next-byte entropy rather than relying on a fixed vocabulary of subword tokens. Lightweight local byte encoders and decoders bridge raw bytes to a global latent transformer, allocating more compute and new patch boundaries wherever predictions are uncertain (such as complex morphology, code, or noisy text) and fewer steps over predictable spans. It eliminates vocabulary bottlenecks and tokenization artifacts, though it remains an emerging architecture rather than a replacement for standard BPE in most production models.
Origin and context
Subword tokenizers such as BPE and WordPiece have long caused LLM failure modes on character-level manipulation, multilingual scripts, arithmetic, and typographic noise, yet pure byte-level transformers were computationally prohibitive because every byte consumed a full transformer step. In December 2024, researchers at Meta FAIR introduced the Byte Latent Transformer (BLT), training models up to 8B parameters on 4T bytes. By using a lightweight byte-level language model to estimate next-byte entropy and starting a new latent patch only when uncertainty spikes above a threshold, BLT matched Llama-3 token-based scaling efficiency while improving robustness on character-level and low-resource tasks.
Why it matters
Standard tokenizers allocate equal global transformer compute to predictable boilerplate tokens and high-entropy decision points, while freezing vocabulary heuristics at pretraining time. Dynamic entropy patching demonstrates that compute can be allocated adaptively along the sequence dimension based on information density, removing the rigid vocabulary embedding table and improving resilience to orthographic noise and multilingual scripts.
Example
When processing the string `function calculate_total_price()`, a BLT local encoder sees high entropy after `function ` and `calculate_`, starting new latent patches at those semantic boundaries, while grouping the highly predictable continuation `total_price()` into a single variable-length patch so the heavy global latent transformer only executes a few steps across the entire span.
Maturity and evidence
Byte Latent Transformer is rated maturity 2 (emerging): Meta FAIR released weights, code, and 8B-scale scaling evaluations, and independent researchers have analyzed dynamic byte patching, but almost all deployed frontier models continue to rely on BPE subword tokenizers and existing token-oriented serving kernels.
Limits and open questions
Related terms
References
- Byte Latent Transformer: Patches Scale Better Than TokensPagnoni et al. / Meta FAIR (arXiv) · 2024-12-13 · class A
- Meta Lingua & Byte Latent Transformer Official ImplementationMeta Research / GitHub · 2024-12-13 · class B
- Dynamic Vocabulary and Byte-Level Patching Analysis in Subword-Free Language Models (TACL / ACL Survey)Independent NLP Researchers / arXiv · 2024-12-20 · class A
Last updated: 2026-10-19
This term is also covered in the Skills Atlas as transformer architecture skill.
This term is also covered in the Skills Atlas as large language models skill.
This term is also covered in the Skills Atlas as model training skill.
This term is also covered in the Skills Atlas as tokenization skill.
Referencing this analysis in documentation, research, or an engineering write-up? Copy a formatted link or citation below:
Show citation text & canonical URL
[Byte Latent Transformer (BLT) — AI Glossary — Skills Intelligence](https://www.skillsintelligence.tools/glossary/term/byte-latent-transformer)Skills Intelligence. (2026). Byte Latent Transformer (BLT) — AI Glossary. Skills Intelligence AI Glossary. Retrieved from https://www.skillsintelligence.tools/glossary/term/byte-latent-transformer