Disaggregated Prefill-Decode Serving

In standard colocated LLM serving, the same GPU instance executes both the compute-intensive prefill phase (processing the full input prompt in parallel) and the memory-bandwidth-bound decode phase (generating output tokens autoregressively). Long prompts stall ongoing decode batches, causing spikes in time-to-output-token (TPOT), while decode reservations underutilize tensor compute during prefill. Disaggregated serving places prefill and decode on separately configured and independently scaled node pools. Once prefill computes the initial attention keys and values, the runtime transfers those KV-cache blocks over high-bandwidth interconnects (such as NVLink or RDMA) to a decode worker that continues streaming tokens.

Profile workload prompt-to-output length ratios and goodput service-level objectives (TTFT and TPOT targets) before choosing a prefill-to-decode instance ratio. Configure the serving framework (such as vLLM, SGLang or llm-d) with a low-latency KV-cache transport layer and cache-aware request router so repeated prefixes land on prefill workers that already hold valid cached blocks. Enforce strict cryptographic or namespace isolation on shared KV-cache pools so cross-tenant prefix reuse cannot leak prompt state across organizational boundaries. Benchmark under bursty long-context arrivals and verify fallback behavior when interconnect bandwidth saturates.

Disaggregation adds network transfer overhead and orchestration complexity; for short prompts or low-concurrency workloads over slow interconnects, transferring the KV cache can cost more latency than colocated execution. Failure domains also multiply because a dropped KV transfer mid-request requires coordinated retry or recomputation. Engineers must validate actual interconnect throughput, memory accounting and tenant isolation rather than assuming disaggregation improves every workload.