Low-Rank Adaptation (LoRA) Weight Decomposition

Trains two thin matrices instead of every weight, so fine-tuning fits one GPU and merges with no added inference latency.

Part 1 of 5

Basics

The whole idea first, then the terms the rest of this page uses.

1.1Whole idea

Full fine-tuning updates billions of weights. LoRA freezes them and learns a small correction for each chosen matrix: W′ = W + (α / r) · B · A. B and A stay thin, so you train a fraction of a percent of the parameters and save a small adapter file.

Why it matters

Trains two thin matrices instead of every weight, so fine-tuning fits one GPU and merges with no added inference latency.

1.2Key concepts

One idea per card, in plain words.

01Low-rank update

Approximates a large change with the product of two thin matrices, B (d_out × r) and A (r × d_in).

02Rank r

Sets the adapter's capacity. Common values run from 8 to 32.

03Alpha

Scales the update through α / r, so changing r doesn't force a new learning rate.

04Zero start

B starts at zero, so training begins from the unchanged base model.

05Merging

Adds the update into W for serving. The merged model runs as fast as the base.

Part 2 of 5

Question

What exactly you must answer, and what to settle before you start.

2.1Question

Interviewer

Your team wants a domain-specific tone from an 8B model, on one 24 GB GPU. Explain how LoRA makes that possible, how many weights you train, and which layers you target.

2.2Clarify first

Each answer removes a guess.

You askAnswer
Style or facts?Style and format suit fine-tuning. Changing facts suit retrieval.
How much data?A few hundred to a few thousand good examples.
Part 3 of 5

Detailed answer

How it works, and how to build or explain it.

3.1Step by step

The mechanism, in the order you would explain it.

  1. Freeze the base

    Set requires_grad to false for every base weight. Nothing else changes in the forward pass.

  2. Wrap target layers

    Add A and B around chosen linear layers. Attention projections are the common choice.

  3. Train the adapter

    Only A and B receive gradients, so memory use and checkpoint size shrink sharply.

  4. Save only the adapter

    The file holds A and B, at tens of megabytes, and loads onto the base later.

  5. Merge or keep separate

    Merge for one fixed adapter. Keep separate to swap many adapters on one base.

3.2Trainable weights, Llama 3.1 8B

32 layers, wrapping the q, k, v and output projections. Full-rank training would touch all 8 billion weights.

RankTrainableShare of model
86.8 million0.08%
1613.6 million0.17%
3227.3 million0.34%

3.3Code

Working code for the core of it.

How few weights LoRA trains

This is the whole reason LoRA is cheap, in one formula.

lora-params.tsTypeScriptCopy
1/** LoRA trains two thin matrices: B (dOut × r) and A (r × dIn). */2export const loraParams = (dIn: number, dOut: number, r: number) => r * (dIn + dOut);3 4/** Trainable weights when LoRA wraps the given projections in every layer. */5export function loraTrainable(layers: number, projections: [dIn: number, dOut: number][], r: number): number {6  return layers * projections.reduce((sum, [dIn, dOut]) => sum + loraParams(dIn, dOut, r), 0);7}
7 linesUTF-8TypeScript
r * (dIn + dOut)
Two thin matrices, rather than one of dIn by dOut. For 4096 by 4096 at rank 16, that's 131,072 weights instead of about 16.8 million.

Tested by npm run test:modules.

Part 4 of 5

Q&A

What interviewers ask next. Answer out loud, then open.

4.1Interview questions

Why does a low-rank update work at all?
The change needed for a narrow task tends to be much simpler than the full weight matrix, so a few directions capture most of it.
Does LoRA slow inference?
Merged, no: the shape stays the same. Unmerged adds a small matrix multiply per wrapped layer.
How do you pick r?
Start at 16. Raise it if training loss stalls; lower it if the adapter overfits.
LoRA or RAG?
RAG supplies facts that change. LoRA shapes style, format and task behaviour. Many products use both.
Part 5 of 5

Extra depth

Edge cases, counterpoints and what to read next.

5.1Edge cases and counterpoints

Includes the strongest argument against this approach.

Show detailsHide details
Target modules
Wrapping every linear layer rarely repays the extra weights. Start with attention projections.
Serving many adapters
vLLM and others load adapters per request, so one GPU can serve many customers.
Counterpoint
Prompting and retrieval solve most needs at far lower cost. Fine-tune only once evals show they fall short.