Speculative Decoding Explained: How Draft Models Accelerate LLM Inference 3x

Overview #

Speculative decoding is an inference optimization algorithm where a lightweight 'draft' model rapidly proposes candidate tokens and a larger target model verifies them in parallel in a single forward pass, increasing generation speed by 2x to 3x with zero loss in output quality.

The Memory Bandwidth Bottleneck in Auto-Regressive LLMs #

During autoregressive token generation, generating a single token requires reading all weights of a 70B model from GPU High Bandwidth Memory (HBM) into compute cores. GPU compute cores sit idle while waiting for memory transfers (memory-bound). Speculative decoding exploits this by verifying multiple candidate tokens simultaneously in a single compute-bound forward pass.

The Propose-and-Verify Algorithm #

1. Draft Generation: A small, ultra-fast model (e.g., Llama-3-8B) rapidly auto-regressively generates K draft tokens (e.g., K=4) in just a few milliseconds.
2. Parallel Verification: The target frontier model (e.g., Llama-3-70B) executes a single parallel forward pass over the draft sequence.
3. Modified Rejection Sampling: The target model accepts tokens matching its probability distribution. If token 3 is rejected, the target model samples a replacement token and discards remaining draft tokens.
   Mathematically, the output distribution is guaranteed to be 100% identical to running the target model alone.

Self-Speculative Decoding and Medusa Heads #

Modern variations (like Medusa and Eagle) eliminate the need for a separate draft model by adding lightweight multi-head prediction heads directly onto the base model, predicting multiple future tokens simultaneously without loading second model weights into VRAM.

Code Example: Enabling Speculative Decoding in vLLM #

# Launch vLLM with speculative draft model configuration
# Target: 70B model, Draft: 8B model
python3 -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3.3-70B-Instruct \
    --speculative-model meta-llama/Llama-3.1-8B-Instruct \
    --num-speculative-tokens 5 \
    --use-v2-block-manager

Frequently Asked Questions #

Q: Does speculative decoding change the quality of the answer?

No. Speculative decoding uses exact rejection sampling math that guarantees the output tokens are drawn strictly from the target model's probability distribution.

Q: What is the typical speedup from speculative decoding?

Real-world speedups typically range between 1.8x and 2.8x depending on acceptance rate (higher for repetitive code and text, lower for dense mathematical reasoning).

Q: Why must the draft model share the same tokenizer?

The draft model and target model must use identical token vocabulary mappings so that proposed token IDs correspond to identical subwords in both models.