Home / Question bank / LLM Fundamentals

LLM FundamentalsMidQuestion 8 of 13

How does LoRA work and why is it popular?

30-second answer

LoRA freezes the pretrained weight W and learns a low-rank update ΔW = BA with small rank r. Only A and B are trained, cutting trainable parameters and optimizer memory by orders of magnitude. Adapters are small, can be swapped per task or tenant, and can be merged into W so inference has no extra latency. QLoRA adds a 4-bit quantized base to fit larger models on a single GPU.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Check whether they want the math, practical training details, or serving implications. Framing: "I'll explain the low-rank update, why it's parameter-efficient, the key hyperparameters, QLoRA, and how adapters are served."

5–15 min Core explanation

Full fine-tuning updates every weight and stores optimizer state for each: for Adam, roughly 3–4× the model size in extra memory.

LoRA freezes the pretrained weight W (d_out × d_in) and learns a low-rank update:

W' = W + ΔW,   ΔW = B · A
A: r × d_in    (initialized randomly)
B: d_out × r   (initialized to zero, so training starts from W exactly)
output = W x + (α / r) · B A x

With d = 4096 and r = 16, ΔW has 2 × 4096 × 16 ≈ 131k parameters instead of 16.8M, about 0.8%.

The hypothesis behind it: the change needed to adapt a model to a task has low intrinsic rank.

15–25 min Deep dive: practical details

Hyperparameters:

  • Rank r: 8–64 typical. Higher captures more change but costs more memory and can overfit.
  • Alpha α: scales the update; effective scale is α / r. Often set to r or 2r.
  • Target modules: originally Q and V projections; applying to all linear layers (attention and MLP) usually works better.
  • Learning rate: higher than full fine-tuning (e.g. 1e-4 to 2e-4).

QLoRA:

  • Quantize the frozen base to 4-bit (NF4), keep LoRA adapters in 16-bit.
  • Gradients flow through the dequantized base into the adapters.
  • Paged optimizers handle memory spikes. Enables fine-tuning large models on a single GPU.

Serving options:

  1. Merge BA into W: zero added latency, one model per adapter.
  2. Keep separate and apply at runtime: one base model in memory, many small adapters swapped per request. Multi-LoRA serving batches requests for different adapters together.

25–32 min Trade-offs and alternatives

Method Trainable params Notes
Full fine-tune 100% Best ceiling, most memory, one full copy per task
LoRA ~0.1–1% Near full-FT quality for most adaptation tasks
QLoRA Same as LoRA Lower memory, slightly slower training
Prefix / prompt tuning Tiny Weaker on complex tasks
DoRA and variants ~LoRA Decompose magnitude/direction; sometimes better

LoRA can underperform full fine-tuning when the task requires large changes (e.g. teaching a new language or extensive new skills).

32–37 min Production example

A SaaS company offers per-customer writing styles. One base model runs on the GPU fleet; each customer has a ~50 MB LoRA adapter trained on their approved content. The serving layer loads adapters on demand with an LRU cache and batches requests across adapters. Adding a customer means training an adapter in an hour, not deploying a new model.

37–40 min Likely follow-ups

  • Why initialize B to zero? So ΔW = 0 initially and training starts from the pretrained behaviour.
  • Can you combine adapters? Yes, by summing or weighted merges, though interactions aren't guaranteed.
  • Does LoRA reduce inference cost? Not by itself; it reduces training and storage cost.

What interviewers listen for

  • The BA formulation with shapes and the parameter count.
  • Why memory savings come largely from optimizer state.
  • Serving trade-offs: merge vs. multi-adapter.

Red flags

  • Saying LoRA "compresses the model".
  • Not knowing what rank controls.