Disaggregated Prefill and Decode in Large-Scale LLM Serving

Separating prefill and decode in large-scale LLM serving can improve latency and throughput, but it is not a universal upgrade. The decision turns on whether relief from competing GPU workloads outweighs the cost of moving their shared state. Concurrency, prompt length, and interconnect quality determine which side wins. More architecture is not automatically more performance.

Why prefill and decode are fundamentally different workloads

Prefill processes the entire input prompt in one forward pass and builds the key-value cache, usually shortened to KV cache. Dense matrix multiplications occupy the GPU’s compute cores, making this a compute-bound phase whose workload scales with prompt token count. Decode has a different appetite for hardware resources. It generates one token per step, reading the full KV cache at each step. Memory bandwidth, rather than raw arithmetic capacity, limits this work, while available cache memory bounds concurrency.

These phases also shape two different measures of responsiveness. Time-to-first-token, or TTFT, measures the wait before an answer starts. Inter-token latency, or ITL, measures the gaps as it continues. In a monolithic serving process, a long prefill can saturate the cores and stall concurrent decode streams, producing head-of-line blocking. vLLM’s disaggregated serving guide illustrates the effect on two NVIDIA L40S GPUs with long prompts, where collocated p99 ITL reaches roughly six times the separated prefill/decode figure at the same offered load.

The GPU node also hosts less glamorous work. Chat templating, tokenization, detokenization, and tool-call parsing are CPU tasks, even when they run on machines rented for their accelerators. They add another resource mismatch to the serving process. The system is not handling one uniform job, despite presenting one answer to the user.

What disaggregation actually does to the serving stack

Disaggregation assigns prefill and decode to distinct GPU pools, joined by a KV cache transfer layer. A cache-aware router directs requests between them. Prefill pods process prompts and send the resulting state to decode pods, which use it to generate tokens. Separation removes the direct contest between prompt processing and ongoing generation. It also makes the handoff between them an explicit part of the architecture.

A useful text-form picture has four tiers, namely a CPU frontend, a cache-aware router, a prefill pool, and a decode pool. The frontend handles the text-facing work, while GPU engines operate on token IDs. vLLM’s frontend exposes /render for tokenization and /derender for detokenization and parsing. Between the GPU pools, KV connectors provide the transfer mechanism. The prompt’s cached KV matrices must reach the decode worker, where that state is reused throughout generation.

The connection is therefore part of the serving path. Between nodes, transfers typically use remote direct memory access, or RDMA, over InfiniBand or RoCE. Within a node, NVLink or GPU peer-to-peer communication can provide the path. The same principle extends beyond prefill and decode to multimodal encoder disaggregation and attention/FFN splitting for MoE models. In each case, the architectural move is to separate work with different resource needs.

The transfer cost that determines whether disaggregation helps or hurts

Every separated request pays a KV transfer tax on TTFT. That tax frees the phases from contention, but it pays off only when the time saved exceeds the transfer overhead. The cost of network transit grows with prompt length and model size, and depends on the interconnect. Even a smaller model with a mid-length context can require transfers of hundreds of megabytes over a 400 Gb/s line, before overhead.

Long prompts make both sides of this bargain larger. They create more prefill work that could disrupt decode, but also more state to move. vLLM’s guide notes that its two-L40S test box lacked peer-to-peer copy capability, highlighting why GPU specifications alone cannot explain the result. Short decode phases provide little opportunity to amortize the transfer. For small batches of short requests, transfer overhead can likewise cost more than separation saves.

Concurrency is the other major condition for a payoff. At low load, two specialized pools may spend their time waiting between bursts, where one well-tuned engine could handle both phases. llm-d’s deployment guide places the crossover at roughly 50 concurrent requests, with monolithic vLLM potentially winning below roughly 20. The relevant threshold depends on the request mix and transfer path.

The useful measure is goodput, the request rate sustainable while meeting both TTFT and ITL targets.

Hardware matching across prefill and decode pools

Once the transfer economics work, separate pools create a hardware cost lever. Prefill rewards high compute throughput, while decode rewards memory bandwidth and capacity. A monolithic deployment has to meet both needs on the same hardware. Disaggregation lets each pool specialize instead. An expensive arithmetic engine is not necessarily the economical place to spend most of a generation reading memory.

The March 2026 hardware estimates illustrate that distinction. The H100 SXM5 offers 3,958 TFLOPS FP8 with sparsity, and the B200 SXM6 also fits compute-heavy prefill. For decode, the H200 SXM5 with 141 GB HBM3e or the A100 80GB offers a memory-oriented choice. These examples reflect the choices at that time, so what to buy may change. A mixed H100 prefill and A100 decode configuration can reduce per-token cost compared with using H100 SXM5 hardware throughout.

If prefill capacity outruns decode, requests queue at the decode pool. If decode capacity outruns prefill, generation workers wait for prompt processing. Buying the right GPU for each phase is only useful if the pools can keep one another supplied.

How the major frameworks implement disaggregation today

The mid-to-late 2026 framework landscape supports disaggregation through several operational models. vLLM v0.31.0 and later offers more than a dozen KV connectors, including NIXL, LMCache, Mooncake, FlexKV, and AMD’s MoRI-IO. Its MultiConnector chains connectors, while support for hybrid SSM models includes Mamba state transfer. This gives operators control over the handoff mechanism. AMD’s single-node MoRI-IO benchmark on one 8-GPU MI300X node achieved roughly 2.4× the goodput of collocated serving on the same hardware.

Amazon SageMaker HyperPod instead composes the transfer path end-to-end. It places the phases on separate GPU pools connected through Elastic Fabric Adapter, or EFA, with RDMA. Its four-layer transfer stack runs from LMCache PD through NIXL and libfabric to EFA. The architecture still depends on moving state efficiently, but the platform supplies the connected stack.

NVIDIA Dynamo and llm-d differ most usefully in their relationship to Kubernetes. Dynamo operates above Kubernetes, while llm-d places the serving machinery inside it. Both help with the work of getting separated resources to work together.

Carnegie Mellon and NVIDIA’s December 2024 paper, A System for Microserving of LLMs, takes another approach. It exposes fine-grained sub-request APIs and a programmable router that turns user requests into sub-request calls. A unified KV cache interface handles computation, transfer, and reuse. Instead of fixing one coordination pattern behind a coarse request API, this design makes the pattern programmable. That flexibility becomes especially relevant when a request is only one turn in a longer task.

Why agentic and multi-turn workloads force a rethink of the disaggregated design

Agentic workloads repeatedly return to the model. In a multi-turn ReAct loop, the model reasons, emits a structured tool call, suspends, receives tool output, and resumes. Each round adds context to the history already there. The conversation grows, but much of it is old. A naive disaggregated design can nevertheless retransfer or recompute the full prior-turn context at every new turn.

This exposes a weakness in the first-generation design. Choosing to separate a large initial prefill from generation also leaves the choice of whether to keep moving the history as it grows. Measurements on real chat workloads find that prior-turn recomputation accounts for the vast majority of multi-turn prefill cost. The system can spend much of its budget restoring context it has already processed. It is an expensive way to remember a conversation.

Agentic and RAG traffic with large prompts needs more prefill capacity, while chat with long generations needs more decode capacity. Independent scaling helps accommodate those differences, but it does not eliminate repeated-context work. Even when the initial split helps, the handoff can still be costly on later turns. Multi-turn serving therefore calls for a closer look at where the cache sits and whether all prefill belongs in the prefill pool.

Partial prefill-decode and other adaptations for multi-turn and long-context serving

Partial Prefill-Decode, or PPD, relaxes that rule for append-prefill operations. It routes new-turn prompt processing to either prefill or decode nodes according to the user’s service-level objective, estimated workload, and initial node configuration. A short append-prefill can remain on a decode node when the following generation will be long. This avoids a round-trip transfer for that operation. The specialization is useful, but it need not become a border checkpoint for every turn.

SGLang’s Q2 2026 roadmap addresses repeated context more directly. A radix cache on the decode side can reuse shared prefixes and request only delta KV, the incremental new context, from prefill. The old prefix then need not travel again. The roadmap also includes routing enhancements for heterogeneous instances within either pool. These changes treat cache reuse and hardware differences as inputs to placement.

NVIDIA’s agentic optimizations work across Dynamo’s frontend API, KV-aware router, and KV cache manager. Validation with the NVIDIA NeMo Agent Toolkit on NVIDIA Hopper delivered up to 4× lower TTFT and 1.5× higher throughput for agentic workloads. The gains come from work across the serving path as well as from a faster prefill kernel. Meanwhile, the microserving router can switch among data-parallel, P/D disaggregated, and P/D work-balancing patterns without restarting the service. Its paper reports that jobs finish much sooner than they do with static strategies.

Other adaptations address the resources consumed by growing histories. Multi-tier KV caching keeps hot blocks in GPU memory and spills to CPU DRAM or NVMe, extending effective cache size and reducing redundant prefill for repeated prefixes. Separate pools also allow different quantization strategies. On Qwen 3 and Gemma 3, removing activation quantization specifically on decode improves accuracy on decode-heavy tasks without increasing inference cost. Prefill can retain lower-precision arithmetic for fast prompt processing, while decode benefits from compact weights that reduce memory traffic.

Deciding the right prefill-to-decode GPU ratio for a target SLO

The allocation starts with throughput requirements, target TTFT, target time-per-output-token, or TPOT, and the expected distribution of input and output lengths. A prefill pool that benchmarks well can still miss TTFT once requests queue. A decode batch that produces many tokens can still violate the required token pace. The right ratio is the one that meets the combined objectives for the actual request mix.

Kingsoft Cloud’s SLO-Aware Compute Resource Allocation for Prefill-Decode Disaggregated LLM Inference combines modeling with measurement. It models prefill as an M/M/1 queue, deriving effective throughput from benchmarked maximum throughput and the TTFT constraint, with queue utilization as an intermediate step. Decode capacity is measured empirically at batch sizes that meet TPOT. Together, those results show how many P/D instances are needed. This ties the GPU ratio to the workload where it will be used.

Dynamo’s Planner applies allocation decisions at runtime. It monitors GPU capacity metrics and TTFT and ITL objectives, then decides whether to use disaggregation or add GPUs to either pool. Dynamo’s GitHub documentation cites Alibaba APSARA 2025 data showing 80% fewer SLA breaches at a lower total cost of ownership with Planner autoscaling. Automation can make it easier to adjust capacity. It does not remove the underlying operational complexity.

The lesson for operators is to disaggregate in response to measured SLO violations. Adding GPU pools brings more chip types, routing logic, and parts of the scheduling system to run and maintain. Dynamo and llm-d handle much of that machinery, but a separated system remains more complex than a monolithic one. Disaggregation earns its place when the workload pays for that complexity. Otherwise, it is another system to run.