Multi-Head Latent Attention (MLA)
Multi-Head Latent Attention (MLA) is an attention architecture, introduced in DeepSeek-V2, that compresses keys and values into a shared low-dimensional latent vector via down-projection before caching, and reconstructs per-head keys and values during attention computation. By caching only the compressed latent representation alongside a small decoupled rotary positional vector, MLA shrinks the KV cache footprint to levels comparable with or smaller than Grouped-Query Attention (GQA) while maintaining the modeling capacity of standard Multi-Head Attention. It is an architectural change requiring pretraining or upcycling, not a drop-in runtime quantization filter for existing checkpoints.
Origin and context
To reduce KV cache memory in large transformers, most post-2023 models replaced Multi-Head Attention (MHA) with Multi-Query Attention (MQA) or Grouped-Query Attention (GQA), trading off head diversity for smaller caches. In May 2024, DeepSeek-AI introduced Multi-Head Latent Attention in DeepSeek-V2 (and later DeepSeek-V3 and R1). By applying low-rank joint compression to keys and values and absorbing the up-projection matrices into the query and output projections during inference, MLA reduced KV cache size by over 90% relative to standard MHA while outperforming GQA of equivalent cache size. Independent engines including vLLM, SGLang, TensorRT-LLM, and FlashMLA subsequently implemented dedicated MLA kernels.
Why it matters
In long-context and high-concurrency LLM serving, the KV cache dominates HBM consumption and memory bandwidth. MLA shows that low-rank latent compression of the KV state space can achieve smaller memory footprints than GQA without forcing multiple query heads to share identical key-value projections, improving both perplexity and maximum serving batch size.
Example
During decoding in a DeepSeek-V3 deployment on SGLang or vLLM with FlashMLA kernels, the model caches only a 512-dimensional compressed latent KV vector and a 64-dimensional decoupled RoPE key vector per token instead of storing 128 separate key and value heads. At attention time, the query heads are multiplied with the absorbed up-projection weights (`W_UQ^T * W_UK`), computing multi-head attention directly against the compressed latent cache without materializing the expanded key tensor in HBM.
Maturity and evidence
Limits and open questions
Because Rotary Position Embeddings (RoPE) are incompatible with matrix absorption across the low-rank projection, MLA must split queries and keys into a non-positional compressed part and a separate decoupled RoPE sub-vector. Furthermore, during prefill where matrix absorption increases FLOPs, engines must switch between unpacked compute-efficient kernels and absorbed memory-efficient decode kernels.
Related terms
References
- DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language ModelDeepSeek-AI (arXiv) · 2024-05-06 · class A
- TransMLA: Multi-Head Latent Attention Is All You Need (Upcycling GQA Models to MLA)Meng et al. / Independent Researchers (arXiv) · 2025-02-11 · class A
- FlashMLA: Efficient MLA Decoding Kernel for Hopper GPUsDeepSeek-AI / GitHub · 2025-02-24 · class B
Last updated: 2026-10-19
This term is also covered in the Skills Atlas as transformer architecture skill.
This term is also covered in the Skills Atlas as inference optimization skill.
This term is also covered in the Skills Atlas as long context modeling skill.
This term is also covered in the Skills Atlas as kv cache optimization skill.
Referencing this analysis in documentation, research, or an engineering write-up? Copy a formatted link or citation below:
Show citation text & canonical URL
[Multi-Head Latent Attention (MLA) — AI Glossary — Skills Intelligence](https://www.skillsintelligence.tools/glossary/term/multi-head-latent-attention)Skills Intelligence. (2026). Multi-Head Latent Attention (MLA) — AI Glossary. Skills Intelligence AI Glossary. Retrieved from https://www.skillsintelligence.tools/glossary/term/multi-head-latent-attention