Basics
The whole idea first, then the terms the rest of this page uses.
1.1Whole idea
RLHF-PPO trains a reward model, then a policy against it, which proves fragile. DPO skips both: given a prompt and a preferred and rejected answer, it nudges the model to favour the preferred one relative to a frozen copy of itself.
Trains on preference pairs directly, with no separate reward model or PPO loop.
1.2Key concepts
One idea per card, in plain words.
A prompt with a chosen and a rejected reply.
A frozen copy that anchors training, so the model can't drift far.
log π(y) − log π_ref(y): how much more the model likes a reply than its reference does.
Scales the loss. Trainers often use about 0.1.
Question
What exactly you must answer, and what to settle before you start.
2.1Question
You have 50,000 preference pairs from reviewers. Write the loss for one pair, given log-probabilities from the model and the frozen reference.
Detailed answer
How it works, and how to build or explain it.
3.1Step by step
The mechanism, in the order you would explain it.
- Score both replies
Sum token log-probabilities for chosen and rejected under both models.
- Take the log-ratios
Subtract reference from model for each reply.
- Compare them
logits = β × (chosen ratio − rejected ratio).
- Apply logistic loss
−log σ(logits). With no preference it equals ln 2, and it falls as the chosen ratio pulls ahead.
3.2Code
Working code for the core of it.
The DPO loss
It trains on pairs of answers, with no separate reward model.
1export interface PairLogProbs {2 chosen: number;3 rejected: number;4 refChosen: number;5 refRejected: number;6}7 8/** DPO: -log sigmoid(beta * [(chosen - refChosen) - (rejected - refRejected)]). No reward model. */9export function dpoLoss(p: PairLogProbs, beta = 0.1): number {10 const logits = beta * (p.chosen - p.refChosen - (p.rejected - p.refRejected));11 return Math.log1p(Math.exp(-logits));12}p.chosen - p.refChosen- How much the model has moved towards the good answer, compared with the frozen reference.
Math.log1p(Math.exp(-logits))- A numerically stable form of minus log sigmoid.
beta- Controls how far the model may drift from the reference.
Tested by npm run test:modules.
Q&A
What interviewers ask next. Answer out loud, then open.
4.1Interview questions
Why keep a reference model?
DPO or PPO?
What does beta do?
Extra depth
Edge cases, counterpoints and what to read next.
5.1Edge cases and counterpoints
Includes the strongest argument against this approach.
Show detailsHide details
- Sum over tokens
- Compute log-probabilities for reply tokens only, not the prompt.
- Counterpoint
- Preference data alone can make models over-verbose. Check your evals for length drift.