Basics
The whole idea first, then the terms the rest of this page uses.
1.1Whole idea
QLoRA trains a LoRA adapter on top of a base model stored in 4 bits. The base never changes, so its low precision matters less. NF4 picks 4-bit levels that suit bell-shaped weights, and double quantisation compresses the scaling constants too.
NF4 and double quantisation shrink a 7B base to about 3.4 GiB, so training fits a small GPU.
1.2Key concepts
One idea per card, in plain words.
Places 16 levels at the quantiles of a normal distribution, because weights roughly follow one.
Normalises each block of 64 weights by one constant before quantising.
Quantises those constants as well, from 0.5 to about 0.127 bits per weight.
Keeps A and B in higher precision. Gradients flow through the 4-bit base, dequantised on the fly.
Question
What exactly you must answer, and what to settle before you start.
2.1Question
Fine-tune a 7B model on a GPU with 8 GB of memory. Compute the base model's memory, say what else takes space, and name where 4-bit weights hurt.
Detailed answer
How it works, and how to build or explain it.
3.1Step by step
The mechanism, in the order you would explain it.
- Normalise per block
Divide each block of 64 weights by a constant, so values sit near a standard normal.
- Map to 16 levels
Assign each weight to its nearest NF4 level, using equal-probability bins.
- Quantise the constants
32-bit constants cost 32 / 64 = 0.5 bits per weight. Double quantisation stores 8-bit ones at 8 / 64 + 32 / (64 × 256) ≈ 0.127.
- Train with dequantisation
Each forward pass decodes weights for the computation, then discards them.
- Budget the rest
Adapter, optimiser state, activations and gradients still need memory, so use gradient checkpointing.
3.27B base model, weights only
| Format | Bits per weight | Memory |
|---|---|---|
| FP16 | 16 | 13 GiB |
| INT8 | 8 | 6.5 GiB |
| NF4 | 4.5 | 3.7 GiB |
| NF4 + double quantisation | 4.127 | 3.4 GiB |
3.3Code
Working code for the core of it.
What 4-bit loading actually saves
The weights shrink, and the constants needed to decode them cost a little extra.
1/** Bits per weight spent on quantisation constants: one 32-bit constant per block of 64 weights. */2export function constantBits(doubleQuant: boolean): number {3 // Double quantisation stores 8-bit constants, plus one 32-bit constant per 256 of those.4 return doubleQuant ? 8 / 64 + 32 / (64 * 256) : 32 / 64;5}6 7export function baseModelGiB(params: number, weightBits: number, doubleQuant: boolean): number {8 return (params * (weightBits + constantBits(doubleQuant))) / 8 / 1024 ** 3;9}32 / 64- One 32-bit constant per 64 weights.
8 / 64 + 32 / (64 * 256)- Double quantisation also compresses those constants.
Tested by npm run test:modules.
Q&A
What interviewers ask next. Answer out loud, then open.
4.1Interview questions
Why not use plain 4-bit integers?
Does the base's low precision hurt quality?
Can you merge into a 4-bit model?
Extra depth
Edge cases, counterpoints and what to read next.
5.1Edge cases and counterpoints
Includes the strongest argument against this approach.
Show detailsHide details
- Speed
- Dequantising on every pass slows training. On a large GPU, FP16 may finish sooner.
- Counterpoint
- Cloud GPUs cost little per hour compared with engineer time. QLoRA matters most when hardware truly limits you.