KV-Aware Routing
KV-aware routing chooses where an LLM request will execute using information about reusable attention key-value cache and worker load. A replica that already holds the matching prompt prefix can avoid recomputing that part of prefill. The router balances this saving against queueing and other work. It selects a serving destination for a model; semantic model routing instead decides which model should answer.
Origin and context
Distributed serving makes cache locality a scheduling problem: a useful prefix on one accelerator is not automatically available on another. Preble studied distributed prompt scheduling that accounts for both reuse and computation. Mooncake built a cache-centered serving architecture, while NVIDIA Dynamo introduced a router that tracks reusable state across workers. Their designs show why cache placement and load balancing need to be considered together.
Why it matters
Repeated system prompts and shared document prefixes offer reuse opportunities, but round-robin routing can scatter those requests across cold replicas. Conversely, always choosing the warmest replica can overload it. Cache-aware scheduling tries to retain the computational benefit without concentrating all demand on one worker. Its success should be measured with latency and useful throughput under the intended traffic mix, not cache-hit rate alone.
Example
Consider an illustrative service with three replicas. One holds a long shared prefix but has a deep queue; another has less cached state and spare capacity. A useful routing policy estimates whether saved prefill time outweighs waiting on the busy replica. Compare it against load-only routing using the same request trace, include cold starts and evictions, and record first-token latency, completion latency, errors and the fraction meeting service targets.
Maturity and evidence
Maturity 3 reflects multiple published scheduling designs and an available engineering implementation. The core idea is established, while cache indexing, transfer and scoring details remain system-specific. A successful experiment on repeated-prefix traffic does not imply the same gains when most prompts are unique or the cache is frequently evicted.
Limits and open questions
Cache information can become stale, tracking and transfers have costs, and replica eligibility still depends on compatible model configuration. A cache hit is an optimization rather than an authorization decision. Keep tenant access boundaries intact and test scale-out, worker replacement and failure recovery. Routing and cache transfer are separate mechanisms; deploying one does not automatically provide the other.
Related terms
References
- Preble: Efficient Distributed Prompt Scheduling for LLM ServingSrivatsa et al. / UC San Diego · 2024-05-08 · class A
- Mooncake: A KVCache-centric Disaggregated Architecture for LLM ServingQin et al. / Moonshot AI and Tsinghua University · 2024-06-24 · class A
- NVIDIA Dynamo, A Low-Latency Distributed Inference Framework for Scaling Reasoning AI ModelsNVIDIA · 2025-03-18 · class B
Last updated: 2026-10-23
This term is also covered in the Skills Atlas as kv cache optimization skill.
This term is also covered in the Skills Atlas as llm inference serving skill.
Referencing this analysis in documentation, research, or an engineering write-up? Copy a formatted link or citation below:
Show citation text & canonical URL
[KV-Aware Routing — AI Glossary — Skills Intelligence](https://www.skillsintelligence.tools/glossary/term/kv-aware-routing)Skills Intelligence. (2026). KV-Aware Routing — AI Glossary. Skills Intelligence AI Glossary. Retrieved from https://www.skillsintelligence.tools/glossary/term/kv-aware-routing