LLM FundamentalsSeniorQuestion 9 of 13

Compare RLHF and DPO.

30-second answer

RLHF trains a reward model on human preference comparisons, then optimizes the policy against it with RL (commonly PPO) plus a KL penalty to the reference model. DPO derives a closed-form objective that optimizes the policy directly on preference pairs, with no separate reward model or RL loop. DPO is simpler and more stable; RLHF can be more flexible, for example when rewards come from sources other than pairwise preferences.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Confirm they want the pipeline and the conceptual difference. Framing: "Both learn from human preference pairs. RLHF learns a reward model and runs RL; DPO folds the reward into a direct loss on the policy. I'll explain each, then compare stability, cost and flexibility."

5–15 min Core explanation

Starting point for both: an SFT model and a dataset of preference pairs (prompt, chosen, rejected) from human or AI labellers.

RLHF (classic three-stage pipeline):

SFT on demonstrations.
Train a reward model to score responses so that r(chosen) > r(rejected) (Bradley–Terry loss).
Optimize the policy with RL (usually PPO) to maximize reward minus a KL penalty that keeps it close to the SFT reference model, preventing reward hacking and collapse.

DPO:

Shows that the optimal policy for the KL-regularized objective has a closed form in terms of the reward.
Rearranging lets you express the reward via the policy itself, giving a simple classification-style loss directly on preference pairs:

L = -log σ( β [ log π(y_w|x)/π_ref(y_w|x)  -  log π(y_l|x)/π_ref(y_l|x) ] )

No reward model, no sampling during training, no PPO.

15–25 min Deep dive: comparison

Aspect	RLHF (PPO)	DPO
Components	Policy, reference, reward model, value model	Policy, reference
Online sampling	Yes, generates during training	No, offline pairs
Stability	Sensitive hyperparameters	Stable, like supervised training
Compute	High	Lower
Reward reuse	Reward model reusable for ranking, evaluation, best-of-n	No explicit reward model
Exploration	Can improve beyond data via sampling	Limited to the data distribution

Variants: IPO (addresses overfitting to deterministic preferences), KTO (works with thumbs-up/down instead of pairs), online/iterative DPO (re-sample and re-label to regain exploration), and GRPO-style RL used for reasoning with verifiable rewards.

25–32 min Trade-offs and failure modes

Reward hacking (RLHF): policy finds outputs the reward model overrates (verbosity, flattery). KL penalty and reward-model ensembles help.
Verbosity bias: both methods inherit labeller preference for longer answers; length-normalized losses or length-controlled evals mitigate this.
Data quality dominates: inconsistent labels hurt either method.
Sycophancy: optimizing for approval can make models agree with users too readily; needs targeted data and evals.

32–37 min Production example

A team aligning an internal support model to company tone collected 10k pairwise judgments of draft replies. DPO on top of their SFT model trained in a day on modest hardware and improved preference win-rate in blind tests. Later they trained a reward model from the same data to rank candidate replies at inference (best-of-4), which added a further improvement on hard tickets.

37–40 min Likely follow-ups

What does β control? Strength of the implicit KL constraint; higher keeps the policy closer to the reference.
Why is the KL term essential? Without it the policy drifts into degenerate text that fools the reward model.
What's RLAIF? Using an AI judge guided by written principles to produce preference labels at scale.

What interviewers listen for

Correct pipeline stages and the KL penalty's purpose.
What DPO removes and why that's possible.
Practical failure modes: reward hacking, verbosity, sycophancy.

Red flags

"DPO doesn't need preference data."
No awareness of reward hacking.