Fused Flash-Softmax Triton Kernel

Fusing softmax into one kernel cuts memory reads from 4n to n and writes from 2n to n.

Part 1 of 5

Basics

The whole idea first, then the terms the rest of this page uses.

1.1Whole idea

Naive softmax runs max, exp, sum and divide as separate GPU operations, and each reads and writes memory. A fused kernel does all four in one pass per row. Memory traffic falls, and speed follows, because GPU compute outpaces GPU memory.

Why it matters

Fusing softmax into one kernel cuts memory reads from 4n to n and writes from 2n to n.

1.2Key concepts

One idea per card, in plain words.

01Grid and program

The grid sets how many program instances run. Here, one per row.

02Block

tl.arange(0, BLOCK_SIZE) makes one vector of offsets. The compiler maps it onto the GPU's parallel units.

03Mask

Blocks usually come in powers of two, so a mask keeps reads and writes inside the real data.

04constexpr

Marks a value known at compile time, so Triton specialises the kernel for it.

Part 2 of 5

Question

What exactly you must answer, and what to settle before you start.

2.1Question

Interviewer

A softmax over long rows dominates your profile. The naive version reads 4n elements and writes 2n. Write a Triton kernel that reads n and writes n.

Part 3 of 5

Detailed answer

How it works, and how to build or explain it.

3.1Step by step

The mechanism, in the order you would explain it.

  1. Run one program per row

    tl.program_id(0) gives the row index.

  2. Load with a mask

    Padding slots read as negative infinity, so exp turns them into zero and they vanish from the sum.

  3. Subtract the maximum

    This prevents overflow in exp and happens in the same pass.

  4. Compute and store

    Sum the exponentials, divide, and store with the same mask.

  5. Launch

    Round the block up to a power of two and pick more warps for larger blocks.

3.2Code

Working code for the core of it.

A fused softmax in Triton

You write one block of work, and Triton compiles it for the GPU.

triton_softmax.pyPythonCopy
1import torch2import triton3import triton.language as tl4 5 6@triton.jit7def softmax_kernel(output_ptr, input_ptr, n_elements, stride, BLOCK_SIZE: tl.constexpr):8    row = tl.program_id(0)                     # one program per row9    col = tl.arange(0, BLOCK_SIZE)10    mask = col < n_elements11 12    x = tl.load(input_ptr + row * stride + col, mask=mask, other=-float("inf"))13    x = x - tl.max(x, axis=0)                  # all in one pass: no extra trips to memory14    numerator = tl.exp(x)15    denominator = tl.sum(numerator, axis=0)16 17    tl.store(output_ptr + row * stride + col, numerator / denominator, mask=mask)18 19 20def softmax(x: torch.Tensor) -> torch.Tensor:21    rows, cols = x.shape22    block = triton.next_power_of_2(cols)23    out = torch.empty_like(x)24    softmax_kernel[(rows,)](out, x, cols, x.stride(0), BLOCK_SIZE=block,25                            num_warps=4 if block <= 256 else 8)26    return out
26 linesUTF-8Python
tl.program_id(0)
Each parallel program handles one row.
x - tl.max(x, axis=0)
Max, exp and sum run in one pass, with no extra trips to memory. That's the point of fusion.
mask
Handles rows that don't fill a power-of-two block.

Syntax-checked. The GPU libraries it needs don't run in our tests.

Part 4 of 5

Q&A

What interviewers ask next. Answer out loud, then open.

4.1Interview questions

Why does fusion matter so much?
GPU memory bandwidth lags compute. Reducing memory traffic often speeds a kernel more than adding parallelism.
Why use a power-of-two block?
Triton kernels have traditionally used power-of-two blocks. next_power_of_2 rounds up, and the mask handles the extra slots.
Why do masked reads use negative infinity?
exp(−∞) equals 0, so padding adds nothing to the denominator.
Part 5 of 5

Extra depth

Edge cases, counterpoints and what to read next.

5.1Edge cases and counterpoints

Includes the strongest argument against this approach.

Show detailsHide details
Autotuning
triton.autotune tries several block sizes and warp counts and keeps the fastest per input shape.
Benchmarks
Compare with torch.softmax on your real shapes. Small inputs may not repay the effort.
Counterpoint
torch.compile fuses many patterns for you. Write kernels only where a profiler points.