Speculative Decoding to Accelerate Autoregressive LLM Inference

Speculative decoding accelerates autoregressive LLM inference by exploiting a hardware imbalance. A large model can verify several proposed tokens for roughly the cost of generating one. That makes it a structural answer to slow generation. Whether it pays off in production depends on how often the proposals survive verification, how much memory they consume, and what else the serving system is doing.

Why autoregressive decoding hits a wall at scale

Generating text one token at a time leaves a powerful GPU doing a surprisingly prosaic job. It moves weights. Each forward pass streams the model’s weights from VRAM to its compute units to produce a single token, taking tens of milliseconds for a large frontier model on an H100. At typical single-user generation rates, only a small fraction of the GPU’s arithmetic capacity is used. Memory bandwidth, rather than arithmetic, sets the pace.

Spector & Re’s Staged Speculative Decoding describes single-batch decoding as having an arithmetic intensity of roughly 1. Yet the next token cannot begin until the previous one is emitted, so spare compute cannot simply generate the rest of the sentence in parallel. Batching can put more of the hardware to work across requests, but each token in a sequence still has to wait for the one before it.

The core insight — verification of N tokens costs roughly what generation of one costs

Speculative decoding separates proposing text from approving it. A lightweight draft mechanism quickly proposes several candidate tokens, and the target model checks them in one forward pass. The expensive weight-loading trip now serves several possible output tokens. Verification still takes computation, but when the model is memory-bound, that extra work uses capacity that would otherwise sit idle.

The standard method preserves the target’s output distribution through rejection sampling. If the target assigns a proposed token at least as much probability as the draft does, the token is accepted. Otherwise, acceptance is probabilistic, and rejection triggers sampling from a corrected residual distribution. The guarantee is exact. The draft changes speed while preserving the statistical character of the output.

Acceptance rate, written as α, controls the expected yield. With α = 0.8 and four speculated tokens, the expected number of output tokens per target-model call rises from 1 to roughly 3.36. That does not make the whole system proportionally faster, because drafting and verification have costs of their own. The payoff therefore depends on what determines α across tasks and methods and how much spare capacity the serving load leaves for speculation.

How acceptance rate varies by task type and why that shapes method choice

SQL, code, and structured instruction-following tend to produce higher acceptance rates because the draft and target distributions align more closely. Open-ended creative writing, adversarial prompts, and other high-entropy tasks tend to pull them apart. The same model pair can therefore deliver a large gain on one prompt mix and a modest gain on another. Acceptance is not a permanent specification printed on the box.

Aaryam Sharma’s When Is a Draft Accepted? A Theory of Acceptance in Speculative Decoding gives this intuition a formal foundation for greedy, relaxed, and tree-based acceptance rules. The paper derives certificates and bounds that connect rejection to distributional divergence and ranking margins. These results come from theory and do not measure speed in production. For greedy tree decoding, the theory also characterizes when the target’s preferred token remains covered among the draft’s candidates.

That coverage changes what a useful draft looks like. A tree has branches that cover more than one sequence, so the target’s greedy token only has to appear among the candidates. The practical objective is accepted tokens per cycle. A strategy that covers more useful possibilities can better amortize the target pass, provided constructing and checking those possibilities does not consume the savings.

The five production-grade draft strategies and where each wins

The EAGLE family improves proposals by using the target’s internal feature vectors. Its lightweight trained head gets a richer representation to work from. EAGLE-2 adds dynamic candidate trees whose width changes with confidence, while EAGLE-3 and P-EAGLE extend the family of available approaches. This takes training and integration work to bring the draft and target into closer alignment, though available checkpoints make setup easier for supported targets.

Medusa puts parallel decoding heads on the target itself. It needs fine-tuning or a pre-trained checkpoint, but avoids loading a separate draft model. Medusa-1 achieves meaningful batch-size-1 speedups without compromising generation quality. The tradeoff is flexibility. Its task-specific heads tie the drafting arrangement more closely to the target than a separate draft model would.

DeepSeek’s MTP and DSpark approach makes speculation part of the serving stack. DSpark runs across V4-Flash and V4-Pro, substantially accelerating per-user generation over the MTP-1 production baseline at matched throughput. DeepSeek released DSpark under the MIT license together with DeepSpec, its training and evaluation codebase. That combination lowers the barrier to custom speculators, although it does not remove the work of training them.

Prompt Lookup Decoding takes the opposite approach to setup. It needs no extra model. By matching n-grams from the prompt against the generation in progress, it proposes text that the input already contains. This makes it useful for summarization, document QA, and retrieval-augmented generation when outputs draw heavily from source text. With no training cost, zero VRAM overhead, and lossless verification, it is the simplest starting point for predominantly prompt-grounded workloads.

Staged speculative decoding tackles both candidate coverage and drafting cost. It arranges proposals as a tree and applies speculation to the draft model itself. The speculator gets a speculator, without making the target distribution negotiable. This deals with how quickly the chance of a long linear draft being correct throughout falls, and it cut single-batch latency by a meaningful amount while keeping output quality the same. Across these five approaches, checkpoint availability, training effort, memory use, and useful candidate coverage pull in different directions.

Where production speedups hold and where they collapse

Batch-size-1 benchmarks give speculation unusually good conditions to work in. The GPU has idle compute available for checking extra tokens, so EAGLE can approach its peak speedup. As concurrency rises, ordinary serving consumes that capacity and the system becomes compute-bound. Speculation then competes with ordinary serving for resources. A separate draft model also takes VRAM away from the KV cache used to retain sequence state.

That memory pressure can change the economic result. Under high concurrency and a loose inter-token-latency target, an EAGLE-class speculator may reduce cost per million accepted tokens only modestly. A separate draft model can increase cost at the same load because of additional KV pressure. Long-context deployments can likewise lose the advantage.

A vendor’s code-heavy gpt-oss deployment using vLLM and EAGLE-3 provides an example of a real cost reduction in a favorable workload. Mixture-of-experts models also look promising because sparse activation can keep them memory-bound at moderate batch sizes, which leaves room for speculation. At small or large batches, however, verification overhead can erase or reverse that advantage.

Training belongs in the cost calculation as well. EAGLE and EAGLE-2 commonly use about 68K target-generated training samples, with later iterations requiring several times more. The compute expense and few hours of work are modest, but they are not zero. Target updates also turn that one-time setup into a maintenance obligation.

Agentic pipelines break the standard speculative decoding assumptions

Standard speculative decoding usually has the draft and target share a prefix, and the draft is trained on a workload that stays much the same. Agentic pipelines disturb both assumptions. The two models can end up with different context windows because of tool calls, retrieved documents, and compressed memory. Concurrent agents also bring heterogeneous context states, changing the prompt mix against which the draft must perform.

Tool-schema compliance can degrade when compression or added context changes the distribution at critical positions, while batch-level acceptance can collapse when the draft no longer reflects the live workload. AsymSpec, presented at EMNLP 2026, addresses the identical-window requirement directly and preserves tool-schema compliance under context compression. Anthropic’s use of one model to draft for Sonnet/Opus shows how drafting works within one model family, though it does not reveal the full internal serving architecture.

Pipeline scope is a separate constraint on the payoff. Accelerating the final answer does little for steps still spent selecting tools, retrieving information, or generating intermediate reasoning autoregressively. Those costs can dilute or eliminate the user-visible latency benefit. The system-design question is therefore which steps receive speculation, which draft strategy serves each, and how acceptance behaves when their contexts diverge.

The security asymmetry that lossy variants introduce and how SecureSD addresses it

Lossless verification preserves the target’s safety-relevant output distribution along with everything else. Lossy variants relax acceptance criteria to gain throughput, surrendering that protection where draft and target disagree. Those positions can include refusals, caveats, and policy-compliant completions, where the target has strong preferences that the draft does not share. The tradeoff is not evenly distributed. Jailbreak and prompt-injection success rates can rise much faster than ordinary quality metrics deteriorate.

SecureSD targets the positions responsible for that asymmetry. It uses stricter verification for early draft tokens, where decisions that affect security are made, and keeps verification efficient later in generation. By reusing speculative-decoding intermediates, it adds negligible computation relative to the target forward pass. It is a position-aware adjustment to verification.

Evaluations should include adversarial tests alongside standard quality benchmarks. A modest decline on MMLU may coexist with a much larger decline in red-team performance. Evaluating lossy methods on utility alone leaves a gap in the assessment of their tradeoffs.

Model provenance and artifact governance when deploying paired model systems

A draft-target deployment is a compound artifact with independently trained and versioned parts. Both need a recorded provenance. A draft trained on target v1’s outputs learns how that target behaved at that time, and may accept far fewer tokens against target v2. The endpoint can continue returning valid answers while throughput quietly deteriorates. When the target is updated or fine-tuned, the draft needs to be tested again and may need retraining.

DeepSeek’s release of checkpoints with training tooling illustrates one way to make that maintenance reproducible. vLLM documents another part of the pairing convention by listing EAGLE drafts on the Hugging Face Hub against target versions. Those drafts include third-party publications from yuhuili and RedHatAI. Teams that use these ready-made drafts need to audit their provenance.

Both components must remain versioned and retrievable together when a deployment’s output needs to be reconstructed for compliance purposes. Losslessness describes a distribution. It does not remove the operational need to know which pair served a request.

The emerging frontier — parallel verification, block diffusion, and what changes next

Speculative Speculative Decoding attacks the sequential draft-then-verify loop itself. It runs the draft asynchronously on separate hardware, pre-speculating for anticipated verification outcomes to hide drafting latency. Early results show gains over optimized speculative decoding as well as autoregressive generation. The target pass is no longer the only scheduling opportunity.

DFlash, presented at ICML 2026, changes the drafting cost structure through block diffusion. It reached roughly 9,000 tokens/sec on H100 PCIe hardware at batch sizes 1–4, more than double the EAGLE-3 figure in the same environment. Those conditions belong with the result. Depth Exploration Decoding takes a different direction, improving throughput monotonically as its explored depth set becomes finer, rather than stopping at a fixed draft model’s accuracy limit.

Framework support is advancing, but the boundaries remain important. AgentSpec is a research prototype tested within vLLM and is not built into v0.12.0, while vLLM’s Parallel Draft Models feature supports parallel speculative token generation but no official release has said that the gap in structured-output compatibility has been closed. SGLang offers slightly better performance at moderate concurrency, while both frameworks use Apache 2.0, making the choice a tuning question.

The frontier is moving from better guesses toward cheaper and better-scheduled verification. Packaged speculators and serving-framework support will change how much work deployment takes, and new hardware arrangements will change which costs are greatest. How much useful output survives each target pass, and what memory, compute, and pipeline costs were incurred to obtain it?