LLM FundamentalsSeniorQuestion 5 of 13

What's the difference between MHA, MQA and GQA?

30-second answer

Multi-head attention gives each head its own K and V. Multi-query attention shares a single K/V across all heads, shrinking the KV cache dramatically at some quality cost. Grouped-query attention shares K/V within groups of heads, a middle ground used by many current models to cut memory and boost decode throughput.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Confirm they want both the mechanics and the motivation. Framing: "All three differ only in how many key/value heads exist. That number drives KV cache size and decode speed, so I'll explain each, quantify memory, and discuss quality."

5–15 min Core explanation

In standard multi-head attention (MHA), each of the h heads has its own query, key and value projections.

MHA: h query heads, h KV heads.
MQA (multi-query): h query heads, 1 shared KV head.
GQA (grouped-query): h query heads split into g groups; each group shares one KV head. 1 < g < h.

MQA and MHA are the two extremes of GQA: g = 1 is MQA, g = h is MHA.

15–25 min Deep dive: why it matters

During decoding, every step reads the full KV cache from GPU memory. With MHA the cache is large and decoding becomes memory-bandwidth-bound.

Cache size scales with the number of KV heads:

Config (32 layers, head_dim 128, FP16)	KV heads	Per token	32k context
MHA	32	512 KB	16 GB
GQA	8	128 KB	4 GB
MQA	1	16 KB	0.5 GB

Smaller cache means:

More concurrent sequences per GPU (higher throughput).
Less memory read per decode step (lower latency).
Longer feasible contexts.

Quality: MQA forces every query head to read the same keys and values, which reduces expressiveness and can make training less stable. GQA with 4–8 groups recovers nearly all of MHA's quality while keeping most of the memory savings, which is why many current open models use it.

Implementation note: KV heads are repeated (or broadcast) to match query heads inside the attention kernel; efficient kernels avoid materializing the copies.

25–32 min Trade-offs and alternatives

Uptraining: an existing MHA checkpoint can be converted to GQA by mean-pooling KV heads within each group, then training briefly.
Multi-head latent attention (MLA): compresses keys and values into a small latent vector that is cached and expanded on the fly, cutting cache further while keeping per-head diversity.
Tensor parallelism: with very few KV heads, splitting heads across GPUs gets awkward; KV heads may need replication.

32–37 min Production example

A team serving a 70B model on 8 GPUs found batch size was capped by KV memory, not compute. Switching to a GQA model of similar quality allowed roughly 4× more concurrent requests at the same latency target, which directly cut cost per token. Choosing models, they now look at KV heads and head dimension alongside parameter count.

37–40 min Likely follow-ups

Does GQA reduce training FLOPs much? Slightly (fewer K/V projections), but its main benefit is inference memory.
Why not reduce query heads too? Queries aren't cached, so they don't affect memory; keeping many gives expressiveness.
How does it interact with paged attention? Independently; both reduce memory pressure and are commonly combined.

What interviewers listen for

Clear definitions with MQA and MHA as GQA's extremes.
Connecting KV heads to cache size, bandwidth and batch size with numbers.
Awareness of the quality trade-off.

Red flags

Thinking GQA reduces the number of query heads.
Not linking the change to inference memory.