Two terms on GPU performance checklists, tile quantization and wave quantization, name two rounding steps that a matrix multiply goes through at launch. The output is rounded up to whole tiles, and the tiles are rounded up to whole waves of thread blocks. Each step can leave computed work unused or SMs idle, and together they produce the steps in a plot of matrix-multiply time against one dimension.
The symptom
NVIDIA's Matrix Multiplication Background User's Guide measures a multiply with M = 2304 and K = 4096 on an A100 and varies N. Its wave quantization section uses 256 by 128 tiles. Going from N = 1536 to N = 1544 adds 8 columns, about half a percent more output, yet throughput roughly halves and duration roughly doubles. Similar jumps follow after N = 3072, 4608 and 6144.
These are cited results, not measurements made for this page. The guide reports them for an A100-SXM4-80GB with CUDA 11.2 and cuBLAS 11.4, and the text and captions of the relevant sections do not state the precision. The rest of the page explains the jump with arithmetic that the figures below calculate.
How a multiply is cut into tiles
In the guide's description of the GPU implementation, the M by N output matrix is partitioned into tiles, and each tile is assigned to a thread block. The block steps through K, accumulating the products for its tile. With a tile of Mtile by Ntile, the launch has ceil(M / Mtile) tile rows and ceil(N / Ntile) tile columns, and their product is the tile count. K does not appear in that count. It sets how much work each tile holds, not how many tiles there are.
The guide lists typical tile dimensions in cuBLAS from 256 by 128, the most efficient, down to 64 by 64, the least efficient. A cuBLAS heuristic chooses among them, and the small tiles are used mainly for small multiplies. The Tensor Cores page covers the instructions that compute one fragment of a tile, and the Streaming Multiprocessors page covers where a block runs.
Tile quantization
The guide's tile quantization section defines the first loss this way: tile quantization occurs when matrix dimensions are not divisible by the thread block tile size. The tiles at the edge are computed in full, so each one costs as much as an interior tile while holding fewer useful values. The figure draws the tile cover of an output with M = 2304 and N set by the slider. Set N to 1544 with 256 by 128 tiles, then to 1536.
Calculated from the inputs on screen, not measured. M is fixed at 2304, as in the NVIDIA guide's example. K does not change the tile count, because each block walks K inside its tile. A narrow remainder in the last column is drawn at least 1% of the width so that it stays visible; the readout gives the exact count.
At N = 1536 the output is 9 by 12 = 108 tiles and every tile is full. At N = 1544 it is 9 by 13 = 117 tiles, and the last column of tiles holds 8 useful columns out of 128, so 92.8% of the computed values are output. The guide's own example is more extreme. With M = 27648, K = 4096 and 256 by 128 tiles forced, N from 136 to 256 needs two tile columns, and at N = 136 the second one holds 8 / 128 = 6.25% useful data. In that range the duration stays constant while the tile count is constant. The guide notes that cuBLAS would choose narrower tiles in practice.
Alignment is a different, smaller rule. The Tensor Cores page covers alignment and padding, which concerns multiples of 8, 64 or 128 elements. Tile quantization concerns the thread-block tile, which is 64 to 256 elements on a side in the guide's list.
Wave quantization
The second loss comes from running the tiles. The guide's wave quantization section describes it as the total number of tiles being quantized to the number of SMs. A block runs on one SM, so the number of blocks that run at once is the number of SMs times the number of blocks one SM can hold. That second factor depends on the kernel's registers, shared memory and threads, as the occupancy section of the SM page explains. Call the product the wave size. The guide's example assumes one block per SM, which gives an A100 with 108 SMs a wave size of 108.
The figure draws each wave as a grid of slots. With more than three waves it shows the first two, a note of how many full waves are left out, and the last wave. It opens on the guide's 117 tiles. The RTX 5090 preset uses 170 SMs, the value that the GPU reported to the lab behind the RTX 5090 eight-kernels article.
Wave 1108 of 108 busy
Wave 29 of 108 busy
Calculated from the inputs on screen, not measured. Each tile is one thread block, and the model assumes a wave finishes before the next one starts. Blocks per SM depends on the kernel's registers, shared memory and threads.
At 108 tiles one wave runs them all. At 117 tiles the first wave runs 108 and a second wave runs the other 9, which leaves 99 of 108 slots idle while it runs. The guide gives the same fraction, 9 / 108 = 8.33% of the SMs. Fewer tiles than slots is the extreme case. A launch of 50 tiles on an A100 is a single partial wave, and 58 of the 108 slots stay idle for the whole launch.
Both together
The two losses multiply. The useful share of tile slots is the tile factor times the wave factor, where the tile factor is the useful share of all tiles and the wave factor is the busy share of the slots over all waves. Written out, the product is M x N divided by the waves times the wave size times the tile area, so the tile count cancels. The figure plots all three against N, with blocks per SM fixed at 1.
Share of tile slots doing useful work (calculated)
N, the width of the output (M is 2304)
Calculated, not measured. The model assumes every tile takes the same time and a wave finishes before the next one starts. Blocks per SM is fixed at 1 here; the real value depends on the kernel. Padding costs time only through the tile count, so the steps fall where the tile count crosses a multiple of the wave size. The curve shows where the steps fall, not how large a measured slowdown is.
The solid useful-share line is a ramp with a cliff wherever the tile count crosses a multiple of the wave size, and nothing else. For the A100 with 256 by 128 tiles, N = 1536 makes exactly 108 tiles, and N = 1544 makes 117 and needs a second wave for 9 of them. The wave factor falls to 117 / 216 = 54.2% and stays there while the tile count stays at 117, through N = 1664. The product at N = 1544 is 50.3%, and it reaches 54.2% at N = 1664 as the tile factor returns to 100%.
The dashed amber tile factor shows how much of the computed tiles is padding, and it has a tooth at every multiple of the tile width. Under this model a partly empty tile takes as long as a full one, so padding costs time only through the tile count: the duration steps fall where the tile count crosses a multiple of the wave size. In the guide's Figure 7, with M = 27648 and 256-row tiles, the output has 108 tile rows. Every added tile column then adds exactly 108 tiles, one wave on an A100 with one block per SM, which is why the steps there fall at every column.
The measured jumps in the guide fall at the same places, 1536, 3072, 4608 and 6144, which are multiples of the 108-tile wave at this tile width. The curve does not predict how large a measured slowdown is. It locates the steps but not their size. It assumes that every tile takes the same time, that a wave finishes before the next one starts, and that blocks per SM is an input and not a property of the GPU. Real kernels also differ in how they load and overlap work, and the library may pick another tile.
What reduces the losses
- Dimensions that are multiples of the tile. A dimension that is a multiple of the tile width removes the tile loss. A tile count that is a multiple of the wave size removes the wave loss, and 216 tiles on 108 slots fill both waves.
- Smaller tiles. More tiles give a finer split, which is what the cuBLAS heuristic can choose, but the guide ranks smaller tiles as less efficient per tile.
- Splitting K. Cutting the inner dimension into ranges makes more blocks from the same tiles. Kernel 7 of the eight-kernels article does this, and Filling the chip sets its block count against the SM count. Blocks that hold partial results for the same tile must combine them afterward, a merge into global memory of the kind the privatization page counts.
- Persistent kernels and Stream-K. A persistent kernel launches a fixed-size grid, usually one block per SM, and each block handles several tiles, taking its next tile from a tile scheduler. A persistent kernel alone still loses at wave boundaries: Colfax shows its data-parallel schedule dropping sharply there. Stream-K targets wave quantization: it splits some tiles along K between blocks and combines their partial results through a global memory workspace. Both are described in Colfax Research's CUTLASS tutorial on persistent kernels and Stream-K.
Recognizing it
In a benchmark sweep over one dimension, look for runs where the time is flat, followed by steps. The flat runs last while the wave count is constant, and the steps fall where the tile count passes a multiple of the wave size. A new tile column causes a step only when it pushes the count past one, as every column does in the guide's Figure 7.
From a single launch, the grid size gives the tile count when the kernel uses one block per tile. Divide it by the number of SMs times the blocks per SM and round up for the wave count. The last wave holds the remainder, or a full wave when the remainder is zero, and a small remainder means most slots idle during it. A kernel that splits K, that is persistent, or that is coarsened with one block per C tiles, as on the thread coarsening page, does not launch one block per tile, so its grid size is not the tile count. A coarsened kernel launches the tile count divided by C blocks.
Further reading
- NVIDIA, Matrix Multiplication Background User's Guide, sections on the GPU implementation, typical tile dimensions in cuBLAS, tile quantization and wave quantization (last updated 1 February 2023).
- Colfax Research, CUTLASS tutorial: persistent kernels and Stream-K.
- One multiply, one RTX 5090, eight kernels, for the block count of a launch against the SM count of the chip.
Related concepts
A fine kernel can still profile poorly: warp divergence taxes throughput ~1/N; alone on a long scoreboard wait, SM issue util can hit 0%. Interactive demos.
Master GPU memory hierarchy from registers to global memory, understand coalescing patterns, bank conflicts, and optimization strategies for maximum performance
When many threads add to the same address, the updates queue. Give each block a private copy in shared memory, merge once, and calculate how the longest queue and the merge cost change.
A tiled matrix multiply repeats global loads across neighboring blocks. Let one block compute several output tiles, then calculate the loads it saves and the SM slots it leaves idle.
How page-locked memory changes CPU-to-GPU transfers, the extra staging in pageable paths, and the conditions needed to overlap copies with GPU work.
Deep dive into the CUDA context object: control vs data plane, inventory (memory, modules, streams, events, graphs), push/pop/setCurrent stacks, primary retain/release, flags and limits, isolation, cost, and traps.
