Basics
The whole idea first, then the terms the rest of this page uses.
1.1Whole idea
A 70B model does not fit on one GPU, so tensor parallelism splits each layer's matrices across the GPUs in a server. GPUs work on their share, then exchange results twice per transformer layer. That exchange decides whether scaling pays off.
Every transformer layer pays for two collectives on NVLink, which explains why tensor parallelism stays inside one server.
1.2Key concepts
One idea per card, in plain words.
Gives each GPU some columns of the weight matrix. Each computes a slice of the output, and all-gather joins the slices.
Gives each GPU some rows and a slice of the input. Partial results must add up, so all-reduce sums them.
Leaves every GPU holding the element-wise sum of everyone's vector.
Passes chunks around a ring of GPUs: reduce-scatter first, then all-gather. It suits large messages.
Question
What exactly you must answer, and what to settle before you start.
2.1Question
Llama 3.1 70B needs about 140 GB for FP16 weights. You have servers with 8 × 24 GB GPUs. How do you split it, what crosses the wire at each layer, and why must all 8 GPUs sit in one server?
Detailed answer
How it works, and how to build or explain it.
3.1Step by step
The mechanism, in the order you would explain it.
- Split by columns
The first MLP matrix goes column-parallel. Each GPU computes its slice of the output and keeps it local.
- Split by rows
The second matrix goes row-parallel and takes those slices as input, so no communication happens between the two.
- Sum at the end
Partial outputs need adding, so one all-reduce closes the MLP. Attention follows the same shape, giving two all-reduces per transformer layer.
- Run the ring
Reduce-scatter sends chunks round the ring, summing as they go. All-gather then passes finished chunks round. Together they take 2(p − 1) small steps.
- Count the bytes
Each GPU sends 2(p − 1)/p of the vector, so bandwidth use stays flat as p grows. Latency adds up with the step count.
3.2One decode step, bandwidth only
Hidden size 8,192, batch of 64 tokens and FP16 give a 1 MiB vector. Two all-reduces across 80 layers on 8 GPUs send about 280 MiB per GPU.
| Link | Bandwidth | Time for 280 MiB |
|---|---|---|
| NVLink (H100, one direction) | 450 GB/s | 0.65 ms |
| InfiniBand NDR200 | 25 GB/s | 12 ms |
| 100 Gb Ethernet | 12.5 GB/s | 23 ms |
3.3Code
Working code for the core of it.
Ring all-reduce
Every GPU must end with the sum of all gradients. A ring keeps each link equally busy, which suits large vectors.
1/**2 * Ring all-reduce: reduce-scatter, then all-gather, in 2(p - 1) steps.3 * Every rank sends one chunk per step, so each sends 2(p - 1)/p of the vector in total.4 */5export function ringAllReduce(vectors: number[][]): { result: number[][]; sentPerRank: number } {6 const p = vectors.length;7 const n = vectors[0].length;8 const size = Math.ceil(n / p);9 const chunks = vectors.map((v) => Array.from({ length: p }, (_, c) => v.slice(c * size, (c + 1) * size)));10 const mod = (x: number) => ((x % p) + p) % p;11 let sent = 0;12 13 // Reduce-scatter: each chunk travels round the ring, picking up every rank's values.14 for (let step = 0; step < p - 1; step++) {15 const msgs = chunks.map((rank, r) => ({ to: mod(r + 1), c: mod(r - step), data: [...rank[mod(r - step)]] }));16 sent += msgs[0].data.length;17 for (const m of msgs) m.data.forEach((x, i) => (chunks[m.to][m.c][i] += x));18 }19 // Rank r now holds the complete chunk r + 1. All-gather passes the finished chunks on.20 for (let step = 0; step < p - 1; step++) {21 const msgs = chunks.map((rank, r) => ({ to: mod(r + 1), c: mod(r + 1 - step), data: [...rank[mod(r + 1 - step)]] }));22 sent += msgs[0].data.length;23 for (const m of msgs) chunks[m.to][m.c] = m.data;24 }25 return { result: chunks.map((rank) => rank.flat()), sentPerRank: sent };26}2(p - 1) steps- Reduce-scatter, then all-gather.
chunks[m.to][m.c][i] += x- Each chunk travels the ring, adding each rank's values.
sentPerRank- Each rank sends about twice the vector in total, however many GPUs there are. That's why it scales well.
Tested by npm run test:modules.
Q&A
What interviewers ask next. Answer out loud, then open.
4.1Interview questions
Why not pipeline parallelism for serving?
Why a ring and not a tree?
What limits scaling?
Extra depth
Edge cases, counterpoints and what to read next.
5.1Edge cases and counterpoints
Includes the strongest argument against this approach.
Show detailsHide details
- Across servers
- Bandwidth falls more than tenfold outside the NVLink domain. Keep tensor parallelism inside one server.
- Stragglers
- One thermally throttled GPU slows the whole group. Monitor per-GPU clocks, not just the average.
- Counterpoint
- Quantising the model onto fewer GPUs often beats adding tensor parallelism, because fewer GPUs communicate less.