Home / Question bank / LLM Fundamentals
What is speculative decoding?
30-second answer
A small, fast draft model proposes several tokens; the large model verifies them in a single forward pass and accepts the longest prefix consistent with its own distribution, correcting the first rejected token. Because verification is parallel, you get several tokens per large-model pass. With the standard acceptance rule the output distribution is unchanged, so it's a pure latency win when the draft agrees often.
Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.
0–5 min Clarify and frame
Framing: "Decoding is memory-bound, so the big model has spare compute each step. Speculative decoding uses that spare compute to verify several guessed tokens at once. I'll explain the algorithm, why the output distribution is unchanged, and when it pays off."
5–15 min Core explanation
- A small, fast draft model proposes the next γ tokens (e.g. 4–8).
- The large target model runs one forward pass over the prompt plus all γ draft tokens, producing its own probabilities at each position.
- Accept draft tokens left to right while they pass the acceptance test. At the first rejection, sample a corrected token from an adjusted distribution and discard the rest.
- Repeat.
If the draft agrees often, each expensive target pass yields several tokens instead of one.
15–25 min Deep dive: why it's lossless
For each draft token x with draft probability q(x) and target probability p(x):
- Accept with probability
min(1, p(x) / q(x)). - If rejected, sample from the residual distribution
norm(max(0, p − q)).
This rejection-sampling scheme provably yields samples exactly from p, the target model's distribution. With greedy decoding it's simpler: accept while the draft token equals the target's argmax.
Expected speedup depends on the acceptance rate α and the draft's relative cost. With high α and a draft that is ~10–20× cheaper, 2–3× speedups are common.
Variants:
| Variant | Draft source |
|---|---|
| Classic | Separate small model with the same tokenizer |
| Medusa | Extra prediction heads on the target model |
| EAGLE | Lightweight head predicting at the feature level |
| Prompt lookup / n-gram | Copies spans from the prompt (great for editing and RAG) |
| Self-speculative | Skip layers of the target model to draft |
25–32 min Trade-offs
- Works best when outputs are predictable: code, structured text, edits, summaries that quote the input.
- Less benefit for highly creative, high-temperature sampling (lower acceptance).
- Costs extra memory for the draft model and its KV cache.
- At large batch sizes the target model becomes compute-bound, shrinking the spare compute speculative decoding relies on.
- Draft and target must share a tokenizer (or need mapping).
32–37 min Production example
A code-assistant service added n-gram prompt-lookup speculation for "rewrite this function" requests, where much of the output copies the input. Acceptance rates were high and tokens/sec roughly doubled for edits, with no change to outputs. For open-ended chat they used a small draft model and saw a smaller but still worthwhile gain at low batch sizes.
37–40 min Likely follow-ups
- Does it change output quality? No, with the correct acceptance rule the distribution is identical.
- How would you pick γ? Tune empirically: too small wastes opportunity, too large wastes draft work after early rejections.
- Why does batch size matter? At high batch the GPU is compute-saturated, so verifying extra tokens isn't free.
What interviewers listen for
- The draft–verify loop and why one target pass can check many tokens.
- The acceptance rule and the losslessness guarantee.
- When it helps and when it doesn't.
Red flags
- Thinking the draft model's output is used without verification.
- Claiming it always gives a fixed speedup.