QLoRA NormalFloat4 & Double Quantization

NF4 and double quantisation shrink a 7B base to about 3.4 GiB, so training fits a small GPU.

Part 1 of 5

Basics

The whole idea first, then the terms the rest of this page uses.

1.1Whole idea

QLoRA trains a LoRA adapter on top of a base model stored in 4 bits. The base never changes, so its low precision matters less. NF4 picks 4-bit levels that suit bell-shaped weights, and double quantisation compresses the scaling constants too.

Why it matters

NF4 and double quantisation shrink a 7B base to about 3.4 GiB, so training fits a small GPU.

1.2Key concepts

One idea per card, in plain words.

01NF4

Places 16 levels at the quantiles of a normal distribution, because weights roughly follow one.

02Block scaling

Normalises each block of 64 weights by one constant before quantising.

03Double quantisation

Quantises those constants as well, from 0.5 to about 0.127 bits per weight.

04FP32 adapter

Keeps A and B in higher precision. Gradients flow through the 4-bit base, dequantised on the fly.

Part 2 of 5

Question

What exactly you must answer, and what to settle before you start.

2.1Question

Interviewer

Fine-tune a 7B model on a GPU with 8 GB of memory. Compute the base model's memory, say what else takes space, and name where 4-bit weights hurt.

Part 3 of 5

Detailed answer

How it works, and how to build or explain it.

3.1Step by step

The mechanism, in the order you would explain it.

  1. Normalise per block

    Divide each block of 64 weights by a constant, so values sit near a standard normal.

  2. Map to 16 levels

    Assign each weight to its nearest NF4 level, using equal-probability bins.

  3. Quantise the constants

    32-bit constants cost 32 / 64 = 0.5 bits per weight. Double quantisation stores 8-bit ones at 8 / 64 + 32 / (64 × 256) ≈ 0.127.

  4. Train with dequantisation

    Each forward pass decodes weights for the computation, then discards them.

  5. Budget the rest

    Adapter, optimiser state, activations and gradients still need memory, so use gradient checkpointing.

3.27B base model, weights only

FormatBits per weightMemory
FP161613 GiB
INT886.5 GiB
NF44.53.7 GiB
NF4 + double quantisation4.1273.4 GiB

3.3Code

Working code for the core of it.

What 4-bit loading actually saves

The weights shrink, and the constants needed to decode them cost a little extra.

nf4-memory.tsTypeScriptCopy
1/** Bits per weight spent on quantisation constants: one 32-bit constant per block of 64 weights. */2export function constantBits(doubleQuant: boolean): number {3  // Double quantisation stores 8-bit constants, plus one 32-bit constant per 256 of those.4  return doubleQuant ? 8 / 64 + 32 / (64 * 256) : 32 / 64;5}6 7export function baseModelGiB(params: number, weightBits: number, doubleQuant: boolean): number {8  return (params * (weightBits + constantBits(doubleQuant))) / 8 / 1024 ** 3;9}
9 linesUTF-8TypeScript
32 / 64
One 32-bit constant per 64 weights.
8 / 64 + 32 / (64 * 256)
Double quantisation also compresses those constants.

Tested by npm run test:modules.

Part 4 of 5

Q&A

What interviewers ask next. Answer out loud, then open.

4.1Interview questions

Why not use plain 4-bit integers?
Uniform levels waste resolution on rare values. Quantile levels spend it where most weights sit.
Does the base's low precision hurt quality?
The QLoRA paper reports little loss, because the adapter trains in high precision and compensates.
Can you merge into a 4-bit model?
Not directly. Load the base at FP16, merge, then quantise again if needed.
Part 5 of 5

Extra depth

Edge cases, counterpoints and what to read next.

5.1Edge cases and counterpoints

Includes the strongest argument against this approach.

Show detailsHide details
Speed
Dequantising on every pass slows training. On a large GPU, FP16 may finish sooner.
Counterpoint
Cloud GPUs cost little per hour compared with engineer time. QLoRA matters most when hardware truly limits you.