Speculative Decoding Explained: How Draft Models Accelerate LLM Inference 3x

During autoregressive token generation, generating a single token requires reading all weights of a 70B model from GPU High Bandwidth Memory (HBM) into compute cores. GPU compute cores sit idle while waiting for memory transfers (memory-bound). Speculative decoding exploits this by verifying multiple candidate tokens simultaneously in a single compute-bound forward pass.

1. Draft Generation: A small, ultra-fast model (e.g., Llama-3-8B) rapidly auto-regressively generates K draft tokens (e.g., K=4) in just a few milliseconds. 2. Parallel Verification: The target frontier model (e.g., Llama-3-70B) executes a single parallel forward pass over the draft sequence. 3. Modified Rejection Sampling: The target model accepts tokens matching its probability distribution. If token 3 is rejected, the target model samples a replacement token and discards remaining draft tokens. Mathematically, the output distribution is guaranteed to be 100% identical to running the target model alone.

Modern variations (like Medusa and Eagle) eliminate the need for a separate draft model by adding lightweight multi-head prediction heads directly onto the base model, predicting multiple future tokens simultaneously without loading second model weights into VRAM.