Continuous Batching vs Static Batching in LLM Serving

Continuous batching improves LLM serving throughput by changing the unit of scheduling from the whole request to the individual decode step. That is an architectural change. Static batching leaves capacity stranded when short generations finish before long ones. Continuous batching reuses that capacity immediately, provided the serving system can also manage the growing memory demands of active requests.

Why static batching fails for variable-length generation

High GPU bills and low utilization in nvidia-smi make an uncomfortable pair. In LLM serving, that combination can reflect a structural problem. Decode, the phase that generates output tokens one after another, is memory-bandwidth-bound. At batch size 1, much of the GPU’s compute capacity waits on model-weight reads. The expensive machinery is waiting for delivery.

Batching amortizes those reads across several sequences. A decode step reads the model weights from GPU memory once, whether it serves one sequence or many, so additional sequences can add throughput at little extra weight-reading cost. Memory still limits the batch width. Each active sequence keeps a growing key-value cache, or KV cache, that holds the state needed for attention. Available VRAM therefore constrains how many sequences can run together.

Static batching keeps those sequences in the same group. Consider four requests that generate different numbers of tokens. When a short request finishes, its slot remains unavailable while the batch runs through the longest generation, with padding taking the place of useful work. Waiting requests cannot enter those vacancies until the entire batch drains. An empty chair is apparently still reserved.

Dynamic batching adds a timeout window that limits how long requests wait for a full batch before they can run. It does not change what happens afterward. Once the group starts running, it stays the same until its slowest sequence finishes. That makes dynamic batching a useful improvement to queue formation, but not a repair for the capacity lost during generation.

What continuous batching actually changes about the scheduling loop

Continuous batching lets the scheduler change which sequences are in the active group after every model forward pass. The request remains the thing being served, but it no longer defines an indivisible scheduling interval. At each decode step, the scheduler can remove sequences that have finished and let waiting ones in. The batch is now a changing population. Iteration-level scheduling and in-flight batching name this same mechanism.

The loop starts by identifying sequences that emitted an end-of-sequence token, removing them, and freeing their KV cache blocks. It then admits waiting requests within the available memory and batch-size limits. The active sequences are concatenated into a compound batch, with attention masks preventing one request from attending to another request’s tokens. One forward pass gives each sequence its next token, and the loop starts again. Finished sequences no longer leave padding behind for the rest of the group.

A timeline with slots A, B, C, and D makes the difference concrete. Under static batching, a short request in A leaves a gap stretching to the end of the longest request in the group. Under continuous batching, that gap becomes the start of a waiting request while B, C, and D continue. The other slots refill in the same way as their occupants finish.

A request needing 30 tokens therefore leaves the active batch at step 30, releasing its slot instead of tying it up for the longest concurrent generation. Continuous batching replaces the rule governing when membership can change. Tuning a static scheduler cannot reproduce the mechanism.

Why PagedAttention is the memory prerequisite that makes continuous batching practical

Continuous batching can keep admitting requests only while their KV state fits in VRAM, and that state grows during generation. Without suitable allocation, the scheduler can find itself with waiting work and no room to admit it. Scheduling solves only half the problem. The memory layer must support frequent arrivals, departures, and growth.

Before PagedAttention, serving frameworks reserved a contiguous memory block for each request sized to its maximum possible output length. Most sequences never used the whole reservation. Meanwhile, blocks released by completed requests could be too small or poorly aligned for the next allocation. A scheduler cannot refill a slot with good intentions.

PagedAttention applies operating-system-style paging to the KV cache. It allocates fixed-size pages as a sequence grows rather than reserving its full possible allocation at arrival. vLLM’s default page size is 16 tokens. A block table maps a sequence’s logical pages to physical GPU memory locations, which need not be contiguous.

Physical pages can also be shared through copy-on-write semantics. Beam-search branches and requests with a shared system prompt can reference the same KV pages until their generations diverge, reducing duplicated allocation. For beam search, that sharing reduces memory overhead and improves throughput. The resulting stack has two distinct jobs. Continuous batching decides when requests join and leave the active set, while PagedAttention decides where to keep their state.

How vLLM, SGLang, and TensorRT-LLM instantiate these mechanisms differently

vLLM, SGLang, and NVIDIA’s TensorRT-LLM all implement continuous batching and paged KV caching. Each is built around different assumptions about the workload. An engine comparison is more informative when it asks what work can be reused, what hardware must be supported, and how stable the deployment will be.

vLLM, first developed at UC Berkeley, has continuous batching on by default and supports a wide range of hardware and models with performance that is easy to predict. That makes it a default choice for unknown or heterogeneous workloads. Anyscale measured substantial throughput gains over static batching on a single A100, illustrating the scheduling change’s practical value. The important controls include --max-num-seqs, which limits concurrent sequences, and --max-num-batched-tokens, which limits total tokens per iteration. For vLLM V1/v0.18.0, the sequence defaults are 1024 for H100/MI300x-class GPUs and 256 for A100 and smaller GPUs.

SGLang extends paged KV caching with RadixAttention, which maintains cached state across requests in a radix tree. Requests sharing system prompts, few-shot examples, or RAG context can reuse prefix computation. Its advantage depends on that reuse. Chatbots, RAG pipelines, and agentic workloads with long common prefixes are therefore a strong fit, with prefix cache hit rate providing a useful basis for comparison with vLLM.

TensorRT-LLM calls its mechanism in-flight batching and compiles the model into a TensorRT engine at deployment time. That compilation cost suits a different operating pattern. Its strongest position is a high-volume, single-model deployment on NVIDIA hardware, where the workload is stable and compilation is paid once.

Where continuous batching's gains shrink or reverse

Published throughput comparisons deserve one clear qualification. Different studies use different models, batch sizes, concurrency levels, hardware generations, and input and output length distributions. Unless those variables are specified, their results provide directional comparisons.

Output-length variance is central to the gain. When requests generate roughly the same number of tokens, few slots sit empty while longer requests finish, so continuous batching has little wasted time to cut. For offline inference where output lengths are known and the same, static batching can therefore still compete. Static execution is simpler in that regime. It also avoids the overhead of making admission decisions at every iteration.

Mixed prompt lengths introduce a different problem. Prefill processes the input prompt, and a long prefill sharing a forward pass with ongoing decode can stall those decoders for its duration. This is prefill head-of-line blocking. Throughput and response latency are not interchangeable measures of health.

When the KV cache saturates, pressure spreads through the admission queue, and p99 latency can climb over time even without a load increase, potentially progressing toward unavailability. More occupied slots are not always better. For the separate problem of prefill blocking, chunked prefill provides a targeted fix by breaking long prompts into smaller pieces that interleave with decode.

How agentic and multi-turn workloads stress the scheduling model beyond its original design

Agentic reinforcement learning and multi-turn inference repeatedly introduce new prefill phases during an ongoing rollout or interaction. Prefill is no longer simply the entrance to generation. It keeps returning while other requests decode, making the two phases coexist throughout execution. Long agent contexts amplify the resulting stalls. What might be an occasional interruption in a simpler workload becomes a recurring scheduling conflict.

Chunked prefill addresses that conflict within the continuous batching scheduler. Instead of inserting a whole long prompt into a shared forward pass, it interleaves smaller prefill chunks with decode steps. The chunk size bounds how much of the unchanged prefill work can delay ongoing generation at once.

Prefill-decode disaggregation changes the architecture more substantially by assigning the phases to separate GPU pools. Prefill is compute-bound, while decode is memory-bandwidth-bound, so their resource needs differ fundamentally. Separating them lets each pool be optimized for its own phase rather than making one GPU serve both profiles together. The trade-off is operational complexity. Below roughly 8 GPUs per model, chunked prefill and well-tuned continuous batching capture most of the benefit without a proportionate payoff from disaggregation.

SGLang’s prefix reuse addresses the same conflict from another direction. Agents commonly share long system prompts or tool schemas across concurrent sessions, making RadixAttention especially useful for avoiding repeated prefill computation. Chunking distributes the work more carefully, disaggregation places it separately, and prefix reuse reduces how much work must happen at all. Those mechanisms complement the scheduler.

Operational guardrails for running continuous batching in production without latency surprises

Running continuous batching calls for managing memory and admission together, because turning it on leaves capacity to be managed. Its failure modes are understood well enough to guard against systematically. The risk comes from failing to watch memory headroom or control which requests are let in. A batch that fits now may not fit after its sequences grow.

The 2026 serving stack combines vLLM or SGLang with paged attention, continuous batching, automatic prefix caching, FP8 KV cache, and speculative decoding, with multi-tenant LoRA serving layered on top. These features work at different levels. EAGLE-2, for example, uses speculative decoding to increase tokens per step without replacing the continuous batching scheduler. Model-level techniques such as quantization, pruning, and distillation, and application-level techniques such as context compression and prompt caching, address other parts of the cost.

For vLLM deployments, the practical guardrail is to cap concurrency with --max-num-seqs so admission does not outrun KV cache capacity. Raising it toward 2048 for very high-traffic APIs is appropriate only after confirming memory headroom. The --gpu-memory-utilization setting should reserve explicit headroom, with high KV cache utilization short of full capacity treated as an alert. Mixed long-prompt and concurrent-decode workloads should enable --enable-chunked-prefill. That includes almost all agentic deployments.

The 2026 paper arXiv:2606.15555 identifies why admission needs more than a snapshot of free memory. Locally feasible decisions can create future pressure, leading to eviction cascades and throughput loss as aggregate demand approaches capacity. The KV cache is a shared resource requiring active governance. Rising p99 latency without a corresponding increase in load is an early warning.

The useful mental model is the decode step. It reveals which idle gaps a scheduler can remove, which benchmark workloads offer little room for improvement, and why an apparently feasible admission can become a memory problem later. With that distinction clear, serving efficiency becomes a question of useful work per iteration within sustainable memory limits, rather than simply how many requests can be squeezed onto a GPU.