Basics
The whole idea first, then the terms the rest of this page uses.
1.1Whole idea
A LoRA layer wraps an existing linear layer, freezes it, and adds a small trainable path alongside. Output equals the base output plus the low-rank correction. Merging folds the correction into the base weight.
Writes a drop-in replacement for nn.Linear with trainable low-rank adapters and dropout.
1.2Key concepts
One idea per card, in plain words.
Keeps the original layer untouched, so weights load unchanged.
Gives A random directions to learn along.
Makes the correction start at exactly zero.
Applies dropout to the adapter path only, which regularises a small adapter.
Question
What exactly you must answer, and what to settle before you start.
2.1Question
Implement LoRALinear(base, r, alpha, dropout). The base stays frozen, output starts identical to the base, and merge() folds the update into the weight.
2.2Clarify first
Each answer removes a guess.
| You ask | Answer |
|---|---|
| Bias? | Keep the base bias frozen. The adapter adds none. |
| Which framework? | PyTorch. The interface must match nn.Linear. |
Detailed answer
How it works, and how to build or explain it.
3.1Step by step
The mechanism, in the order you would explain it.
- Freeze the base
requires_grad_(False) covers the weight and any bias.
- Create A and B
A has shape (r, in_features); B has shape (out_features, r). Their product matches the weight's shape.
- Initialise
Kaiming for A and zeros for B, so the first forward pass equals the base.
- Forward
Base output plus (dropout(x) @ Aᵀ @ Bᵀ) × α / r. Matrix order follows the batch-first input.
- Merge
weight += (α / r) · B @ A, then skip the adapter path. Guard against merging twice.
3.2Code
Working code for the core of it.
A LoRA layer from scratch
The layer behind PEFT, written out in full.
1import math2 3import torch4from torch import nn5 6 7class LoRALinear(nn.Module):8 """Wraps a frozen nn.Linear with a trainable low-rank update: W + (alpha / r) * B @ A."""9 10 def __init__(self, base: nn.Linear, r: int = 16, alpha: float = 32.0, dropout: float = 0.05):11 super().__init__()12 self.base = base13 self.base.requires_grad_(False)14 self.lora_A = nn.Parameter(torch.empty(r, base.in_features))15 self.lora_B = nn.Parameter(torch.zeros(base.out_features, r)) # zero: starts as the base model16 nn.init.kaiming_uniform_(self.lora_A, a=math.sqrt(5))17 self.scale = alpha / r18 self.dropout = nn.Dropout(dropout)19 self.merged = False20 21 def forward(self, x: torch.Tensor) -> torch.Tensor:22 out = self.base(x)23 if self.merged:24 return out25 return out + (self.dropout(x) @ self.lora_A.T @ self.lora_B.T) * self.scale26 27 @torch.no_grad()28 def merge(self) -> None:29 if not self.merged:30 self.base.weight += self.scale * (self.lora_B @ self.lora_A)31 self.merged = Truerequires_grad_(False)- The original weights stay frozen.
torch.zeros(...)- B starts at zero, so training begins from exactly the base model.
alpha / r- Scaling that keeps the learning rate stable when you change the rank.
def merge- Bakes the update into the weights, so inference has no extra cost.
Syntax-checked. The GPU libraries it needs don't run in our tests.
Q&A
What interviewers ask next. Answer out loud, then open.
4.1Interview questions
Why not zero both matrices?
What test proves it correct?
What about model.eval()?
Extra depth
Edge cases, counterpoints and what to read next.
5.1Edge cases and counterpoints
Includes the strongest argument against this approach.
Show detailsHide details
- Replacing layers
- Walk named_modules, match target names, and swap in the wrapper with setattr on the parent.
- Counterpoint
- PEFT already handles this, including quantised bases. Write your own to learn it or for an unusual layer, not by default.