Home / Question bank / LLM Fundamentals
Describe prefill vs. decode. Which one dominates latency for long answers?
30-second answer
Prefill processes the whole prompt in parallel and is compute-bound; it determines time-to-first-token. Decode generates one token per forward pass and is memory-bandwidth-bound; it determines tokens/second. For long outputs, decode dominates total latency, which is why output length is the first lever for speed.
Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.
0–5 min Clarify and frame
Ask whether the context is an API consumer or someone self-hosting. Framing: "Inference has two phases with different bottlenecks. I'll explain both, show how they map to user-visible metrics, and list what you can tune for each."
5–15 min Core explanation
Prefill: the model processes the entire prompt in one parallel forward pass, filling the KV cache and producing the first output token.
- Lots of tokens processed at once → large matrix multiplies → compute-bound.
- Determines time-to-first-token (TTFT).
- Grows with prompt length (and quadratically for attention at long lengths).
Decode: generate the remaining tokens one at a time.
- Each step processes one token but must read all model weights and the KV cache → memory-bandwidth-bound.
- Determines inter-token latency / tokens per second.
- Total decode time ≈ output tokens × time per token.
For long answers, decode dominates. A 2,000-token prompt might prefill in a few hundred milliseconds, while a 1,000-token answer at 50 tokens/sec takes 20 seconds.
15–25 min Deep dive: why the bottlenecks differ
Arithmetic intensity = FLOPs per byte moved. In decode, a 70B-parameter model in FP16 reads ~140 GB of weights to produce one token per sequence. Unless many sequences are batched, the GPU's compute units sit idle waiting on memory.
Batching helps decode: weights are read once per step and reused for every sequence in the batch. That's why serving systems use continuous batching, admitting new requests into the running batch at each step rather than waiting for a whole batch to finish.
Prefill and decode interfere: a long prefill can stall decode steps for other users, causing latency spikes. Mitigations:
- Chunked prefill: split long prompts into chunks interleaved with decode steps.
- Disaggregated serving: separate GPU pools for prefill and decode, transferring KV cache between them.
25–32 min Levers for each phase
| Goal | Lever |
|---|---|
| Lower TTFT | Shorter prompts, prompt/prefix caching, fewer retrieved chunks, faster prefill hardware |
| Faster tokens/sec | Smaller or quantized model, speculative decoding, GQA models, less batching contention |
| Lower total latency | Shorter outputs (biggest lever), streaming for perceived speed |
| Higher throughput | Larger batches, continuous batching, paged KV cache |
Throughput and latency trade off: bigger batches raise tokens/sec per GPU but slow each individual stream.
32–37 min Production example
An internal assistant felt slow: p50 total latency 9s. Breakdown showed TTFT 0.8s, decode 8s for ~400-token answers. Fixes: instructed the model to answer in ≤150 words with an "expand" option, streamed output, and routed simple questions to a smaller model. Perceived latency dropped to under a second to first text and ~3s to completion, without touching infrastructure.
37–40 min Likely follow-ups
- Why is input cheaper than output in API pricing? Input is processed in parallel during prefill; output tokens each need a memory-bound step.
- How does speculative decoding help? It lets decode verify several tokens per large-model pass.
- What metrics would you dashboard? TTFT, inter-token latency, output tokens, queue time, p95/p99 of each.
What interviewers listen for
- Correct bottleneck for each phase and the reason.
- Mapping phases to TTFT vs. tokens/sec.
- Naming output length as the first practical lever.
Red flags
- Treating latency as one number.
- Suggesting a bigger GPU as the only fix.