Basics
The whole idea first, then the terms the rest of this page uses.
1.1Whole idea
Naive softmax runs max, exp, sum and divide as separate GPU operations, and each reads and writes memory. A fused kernel does all four in one pass per row. Memory traffic falls, and speed follows, because GPU compute outpaces GPU memory.
Fusing softmax into one kernel cuts memory reads from 4n to n and writes from 2n to n.
1.2Key concepts
One idea per card, in plain words.
The grid sets how many program instances run. Here, one per row.
tl.arange(0, BLOCK_SIZE) makes one vector of offsets. The compiler maps it onto the GPU's parallel units.
Blocks usually come in powers of two, so a mask keeps reads and writes inside the real data.
Marks a value known at compile time, so Triton specialises the kernel for it.
Question
What exactly you must answer, and what to settle before you start.
2.1Question
A softmax over long rows dominates your profile. The naive version reads 4n elements and writes 2n. Write a Triton kernel that reads n and writes n.
Detailed answer
How it works, and how to build or explain it.
3.1Step by step
The mechanism, in the order you would explain it.
- Run one program per row
tl.program_id(0) gives the row index.
- Load with a mask
Padding slots read as negative infinity, so exp turns them into zero and they vanish from the sum.
- Subtract the maximum
This prevents overflow in exp and happens in the same pass.
- Compute and store
Sum the exponentials, divide, and store with the same mask.
- Launch
Round the block up to a power of two and pick more warps for larger blocks.
3.2Code
Working code for the core of it.
A fused softmax in Triton
You write one block of work, and Triton compiles it for the GPU.
1import torch2import triton3import triton.language as tl4 5 6@triton.jit7def softmax_kernel(output_ptr, input_ptr, n_elements, stride, BLOCK_SIZE: tl.constexpr):8 row = tl.program_id(0) # one program per row9 col = tl.arange(0, BLOCK_SIZE)10 mask = col < n_elements11 12 x = tl.load(input_ptr + row * stride + col, mask=mask, other=-float("inf"))13 x = x - tl.max(x, axis=0) # all in one pass: no extra trips to memory14 numerator = tl.exp(x)15 denominator = tl.sum(numerator, axis=0)16 17 tl.store(output_ptr + row * stride + col, numerator / denominator, mask=mask)18 19 20def softmax(x: torch.Tensor) -> torch.Tensor:21 rows, cols = x.shape22 block = triton.next_power_of_2(cols)23 out = torch.empty_like(x)24 softmax_kernel[(rows,)](out, x, cols, x.stride(0), BLOCK_SIZE=block,25 num_warps=4 if block <= 256 else 8)26 return outtl.program_id(0)- Each parallel program handles one row.
x - tl.max(x, axis=0)- Max, exp and sum run in one pass, with no extra trips to memory. That's the point of fusion.
mask- Handles rows that don't fill a power-of-two block.
Syntax-checked. The GPU libraries it needs don't run in our tests.
Q&A
What interviewers ask next. Answer out loud, then open.
4.1Interview questions
Why does fusion matter so much?
Why use a power-of-two block?
Why do masked reads use negative infinity?
Extra depth
Edge cases, counterpoints and what to read next.
5.1Edge cases and counterpoints
Includes the strongest argument against this approach.
Show detailsHide details
- Autotuning
- triton.autotune tries several block sizes and warp counts and keeps the fastest per input shape.
- Benchmarks
- Compare with torch.softmax on your real shapes. Small inputs may not repay the effort.
- Counterpoint
- torch.compile fuses many patterns for you. Write kernels only where a profiler points.