Prefill-Decode Disaggregation

Early LLM serving engines co-located prefill and decode on the same GPU instances using continuous batching. In January 2024, researchers from Peking University and UCSD introduced DistServe (later published at OSDI 2024), while Microsoft Research and the University of Washington independently presented Splitwise at ISCA 2024. Both teams showed that colocating prefill and decode creates strong interference between time-to-first-token (TTFT) and time-per-output-token (TPOT) service-level objectives. Subsequent production architectures—including Moonshot AI’s Mooncake (FAST 2025), vLLM’s disaggregated serving pipeline, SGLang, and NVIDIA Dynamo—adopted KV-cache transfer layers over NVLink and RDMA to run separate prefill and decode clusters.

Sources: s1, s2, s3

Prefill and decode have fundamentally different hardware profiles: prefill processes all input tokens in parallel and saturates tensor compute, whereas autoregressive decode processes one token per request step and is bottlenecked by HBM bandwidth. When both phases share a batch on the same GPU, a single long-context prefill request delays every active decode stream in that iteration, causing tail-latency spikes in streaming applications. Disaggregating the pools lets operators assign compute-dense GPUs or high tensor-parallelism degrees to prefill while configuring memory-optimized batch sizes and pipeline parallelism for decode, significantly increasing goodput under strict TTFT and TPOT constraints.

Sources: s1, s2

In a production RAG or coding-assistant cluster with 32K-token prompts, an ingress router sends incoming requests first to a prefill worker pool configured with 4-way tensor parallelism. Once the prompt’s key-value tensors are computed, the prefill worker streams the KV cache blocks over RoCEv2 or NVLink to a decode worker pool running large continuous batches at lower tensor parallelism. The decode worker immediately begins streaming output tokens without ever being interrupted by new long-prompt arrivals.

Sources: s1, s3

Disaggregation requires transferring potentially gigabytes of KV cache state per request across network or fabric links. On clusters without high-bandwidth RDMA or NVLink interconnects, KV transmission latency can erase the scheduling gains and stall prefill workers waiting to flush memory. Furthermore, workloads with very short prompts or highly volatile traffic ratios between prefill and decode can suffer from pool underutilization unless paired with elastic reconfiguration.

Sources: s2, s3