Triton Compiler JIT Codegen Pipeline

Follows one Python function down to GPU binary, which explains caching, specialisation and autotuning.

Part 1 of 5

Basics

The whole idea first, then the terms the rest of this page uses.

1.1Whole idea

@triton.jit doesn't run your function in Python. It parses the function into an intermediate form, lowers it through GPU-aware stages, and produces binary code at first launch. Later launches with the same shape reuse the cache.

Why it matters

Follows one Python function down to GPU binary, which explains caching, specialisation and autotuning.

1.2Key concepts

One idea per card, in plain words.

01JIT

Compiles at first use, when argument types and constants are known.

02Triton IR

Captures your kernel at the level of blocks and tensor operations.

03TritonGPU IR

Adds layout decisions: how data maps onto threads and warps.

04Specialisation

Compiles a separate version per data type and constexpr value.

Part 2 of 5

Question

What exactly you must answer, and what to settle before you start.

2.1Question

Interviewer

A Triton kernel takes seconds on its first call and milliseconds afterwards, then slows again when the block size changes. Explain why.

Part 3 of 5

Detailed answer

How it works, and how to build or explain it.

3.1Step by step

The mechanism, in the order you would explain it.

  1. Parse the Python

    Triton reads the function's source and builds its intermediate representation.

  2. Lower to TritonGPU IR

    The compiler assigns layouts and fuses compatible operations.

  3. Lower to LLVM IR

    Standard compiler optimisations run here.

  4. Emit PTX

    LLVM's NVIDIA backend emits GPU assembly.

  5. Assemble to binary

    ptxas produces the binary, which is cached on disk.

  6. Reuse the cache

    Same source, types and constexprs load instantly. A new block size means a new compile.

3.2Code

A working implementation, then what each part does and why.

A fused softmax in Triton

You write one block of work, and Triton compiles it for the GPU.

triton_softmax.pyPythonCopy
1import torch2import triton3import triton.language as tl4 5 6@triton.jit7def softmax_kernel(output_ptr, input_ptr, n_elements, stride, BLOCK_SIZE: tl.constexpr):8    row = tl.program_id(0)                     # one program per row9    col = tl.arange(0, BLOCK_SIZE)10    mask = col < n_elements11 12    x = tl.load(input_ptr + row * stride + col, mask=mask, other=-float("inf"))13    x = x - tl.max(x, axis=0)                  # all in one pass: no extra trips to memory14    numerator = tl.exp(x)15    denominator = tl.sum(numerator, axis=0)16 17    tl.store(output_ptr + row * stride + col, numerator / denominator, mask=mask)18 19 20def softmax(x: torch.Tensor) -> torch.Tensor:21    rows, cols = x.shape22    block = triton.next_power_of_2(cols)23    out = torch.empty_like(x)24    softmax_kernel[(rows,)](out, x, cols, x.stride(0), BLOCK_SIZE=block,25                            num_warps=4 if block <= 256 else 8)26    return out
26 linesUTF-8Python
tl.program_id(0)
Each parallel program handles one row.
x - tl.max(x, axis=0)
Max, exp and sum run in one pass, with no extra trips to memory. That's the point of fusion.
mask
Handles rows that don't fill a power-of-two block.

Syntax-checked. The GPU libraries it needs don't run in our tests.

Part 4 of 5

Q&A

What interviewers ask next. Answer out loud, then open.

4.1Interview questions

Why is the first call slow?
It runs the whole pipeline. Later calls hit the in-memory or disk cache.
Why do many warp configs slow start-up?
Autotune compiles and runs every candidate on first use.
How do you debug a slow kernel?
Dump the IR and compare block size, warps and memory stalls per configuration.
Part 5 of 5

Extra depth

Edge cases, counterpoints and what to read next.

5.1Edge cases and counterpoints

Includes the strongest argument against this approach.

Show detailsHide details
Cache in containers
Ship ~/.triton/cache in your image, or every new replica pays the compile cost.
Counterpoint
Most engineers never need these stages. Learn them when kernels misbehave.