TL;DR
- The paper reports that, for ONNXRuntime on a V100, average compute-core utilization is 51.6% while memory bandwidth utilization reaches 96.7%, which the paper reads as GPU memory being the end-to-end bottleneck of recent models.
- Welder describes a model as a graph of tiles. Choosing an output tile fixes the shape of every input tile behind it, up to how the reduction axis is split, and the global traffic of the resulting tile-graph follows from tile and tensor shapes alone, with no kernel run.
- For a Matmul followed by a Softmax, the worked example gives 840MB of global traffic with a [4 × 128] output tile and 264MB with [16 × 128] (exactly 840 and 264 MiB, as the body shows). Each edge also gets a reuse layer (register, shared memory or global memory) that decides where its intermediate tile lives.
- The headline speedups come with conditions: up to 21.4× over PyTorch is MobileNet with TensorCore, FP16, batch size 1 on a V100, while the average over TensorRT is 1.47× at FP32, batch size 1.
- Welder loses to TensorRT on UNet and VGG16, where TensorRT runs the dominant convolutions with Winograd, and the design targets static models. The traffic model on this page is a teaching calculation and does not predict latency.
Why memory traffic became the bottleneck
Compilers have long treated a neural network as a graph of compute-heavy operators and optimized each operator, or fused simple neighbors. Shi et al. argue that this view no longer fits. In their profiling of ONNXRuntime on a V100 (Figure 1), classical models such as ResNet and BERT reach compute utilization above 80%, but the more recent models in the study mix in memory-intensive operators and use the cores less. The paper also reports a higher ratio of memory stores to loads for the newer models, which it attributes to large activations produced from high-fidelity inputs. Paper, Section 2
If intermediate tensors travel through global memory between operators, the remedy is to keep them on chip. That is hard because each operator wants its own tile shape, and a tile that is best for one operator may not be usable by the next. The question the paper answers is how to choose tile shapes and memory layers for a whole chain of operators rather than one operator at a time. The GPU memory hierarchy concept covers the layers involved, and the streaming multiprocessor page covers where shared memory sits.
Three evidence classes appear on this page and are labeled where they occur. Paper-reported values are measurements or statements from the paper. Calculated here values come from the formulas shown, evaluated by the page's JavaScript and checked against a Python reference. Authored illustrations explain a definition and are not Welder output. The page measures nothing on a GPU.
One pair of operators, two optimal tiles
The paper's motivating example is a Matmul followed by a Softmax. Optimized independently, the best shared-memory tile for the Matmul is [32 × 64] and for the Softmax is [4 × 128]. Those shapes do not match, so the Softmax cannot read the Matmul's output from shared memory, and the pair takes 0.36 ms (Figure 2). Forcing a shape that serves both operators, [16 × 128], lets them fuse through shared memory, and the pair takes 0.29 ms, a 1.26× speedup. Run separately, the two operators each lose some of their stand-alone efficiency with that aligned tile, 15% and 4% in the paper's account, and the fused kernel still wins. Paper, Section 2
The lesson is that the best tile for a fused pair is not the best tile of either member. Finding it for long chains requires a rule for deriving the shared tile and a cost that compares candidates without running them. The paper supplies both.
Three observations that make the search tractable
The design rests on three observations, each of which the later sections use.
- Back-to-front shape inference. Given the tensor expression of each operator, an output tile of a Softmax determines its input tile, and that determines the input tiles of the Matmul behind it. Choosing the last operator's tile fixes the shape of every tile behind it, apart from how the reduction axis is split.
- Traffic is analytic. Once tile shapes and tensor shapes are known, the traffic follows by arithmetic. Traffic ranks tile choices at one memory layer without profiling; Welder profiles only the top K.
- Inter-layer independence. The traffic at one memory layer depends only on the tile configuration at that layer, so each layer can be optimized separately. The only constraint between layers is that a lower-level tile must be larger than the tile above it.
The second observation is the one a reader can check by hand, and the explorer below does that.
Working the traffic calculation
Let M × N be the output tensor D, K the reduction size of the Matmul, q the bytes per element, and [m × n] the output tile. The Softmax reduces over a row, so a legal tile keeps n equal to the row width. A tile-graph loads its input tiles, keeps the reuse tile on chip, and stores its output tile:
The paper's tensors are A [98304 × 64], B [64 × 128] and D [98304 × 128] in FP32. For a [4 × 128] output tile, one tile-graph moves (4·64 + 64·128 + 4·128) × 4 = 35,840 bytes, and 24,576 tiles are needed, so the total is 880,803,840 bytes. The paper reports these as 35KB and 840MB. They agree only if the B tile is reloaded for every output tile and the units are 1,024 and 2^20 bytes, so 880,803,840 bytes is exactly 840 MiB. Calculated here: a [16 × 128] tile moves 45,056 bytes, needs 6,144 tiles, and totals 276,824,064 bytes, which is 264 MiB. The paper states only the 264MB total. Paper, Section 2
Pick an output tile, derive the traffic
Choose how many rows of D one tile computes. Softmax reduces over each row, so a tile always spans all 128 columns. The input tiles follow from the output tile, and global traffic follows from the tile shapes alone.
Tiles propagated back to front for D tile [4 × 128]
1. Output tile D (chosen)
[4 × 128]
2,048 B · global traffic
Written once to global memory.
2. Reuse tile C (Softmax input)
[4 × 128]
2,048 B · stays in shared memory
Softmax needs the same shape. This tile is what fusion keeps on chip.
3. Input tile A
[4 × 64]
1,024 B · global traffic
Matmul needs all 64 reduction values for these rows.
4. Input tile B
[64 × 128]
32,768 B · global traffic
The same for every output tile; the paper’s formula counts it once per output tile.
Global bytes per tile
35,840 B
(4·64 + 64·128 + 4·128) × 4 B
Tiles to cover D
24,576
98,304 ÷ 4 rows
Total global traffic
840 MiB
880,803,840 B
Shared-memory footprint
37 KiB
Fits the assumed 64 KiB budget.
Keeping C on chip avoids writing and rereading 48 MiB each way, but at 4 rows the repeated B loads are 768 MiB of the 840 MiB total. Under the assumed budget 5 of 9 candidates fit; the top 3 by traffic are [16 × 128], [8 × 128], [4 × 128]. The selected tile ranks 3.
| Output tile | Total traffic | Footprint | Status |
|---|---|---|---|
| [1 × 128] | 3144 MiB | 33.25 KiB | Fits, outside top K |
| [2 × 128] | 1608 MiB | 34.5 KiB | Fits, outside top K |
| [4 × 128] | 840 MiB | 37 KiB | Kept, rank 3 |
| [8 × 128] | 456 MiB | 42 KiB | Kept, rank 2 |
| [16 × 128] | 264 MiB | 52 KiB | Kept, rank 1 |
| [32 × 128] | 168 MiB | 72 KiB | Dropped: over budget |
| [64 × 128] | 120 MiB | 112 KiB | Dropped: over budget |
| [128 × 128] | 96 MiB | 192 KiB | Dropped: over budget |
| [256 × 128] | 84 MiB | 352 KiB | Dropped: over budget |
Rearranging the formula for this case shows where the savings come from. The A reads and D writes cost 72 MiB however the output is tiled, and the repeated B loads cost 3,072 MiB divided by the number of rows in the tile. This split is calculated here, not stated by the paper. Larger tiles amortize B better, but a bigger tile set has a bigger footprint. In this model only a capacity limit stops the growth; in the paper, Section 4.1 also penalizes tiles that leave too little parallelism. The paper does not state the capacity or search bound behind its [16 × 128] choice, so the figure leaves the budget to the reader. The 64 KiB default is arbitrary: at that value the model happens to rank [16 × 128] first, which follows from the assumed budget and does not reconstruct the paper's choice.
Counting only global bytes also leaves things out. It does not model coalescing or core utilization, and it ignores any cache effect. Section 4.1 below describes how the paper adds some of these back.
A Matmul and Softmax pair needs only one reuse tile. A real model chains many operators across three memory layers, so the next step is a vocabulary for where each tile lives.
From operator-tiles to a hierarchical tile-graph
An operator-tile computes one output tile of an operator from the input region it depends on. In the paper's Figure 4, a convolution tile reads a [3 × 3] window to produce one output position and a pooling tile reads [2 × 2] to produce one. Two adjacent operator-tiles are connected through a common intermediate tile, the reuse-tile, so the second consumes what the first produced without a full intermediate tensor. Connected operator-tiles form a tile-graph. Paper, Section 3.1
Tile propagation runs from the output nodes back to the inputs using each operator's tensor expression. For irregular access such as Gather or strided convolution, the paper uses a conservative upper bound as the input tile. For a tile-graph, the traffic is the sum of its input and output tile sizes multiplied by the number of tile-graphs needed to cover the output, and the minimum footprint comes from a memory allocation algorithm such as best-fit applied to all tiles in topological order. Input tiles that contain reduction axes can also be split into smaller tiles consumed sequentially.
Mapping onto a memory hierarchy repeats the idea once per layer. Figure 5 labels its layers L0 to L2, and this page maps them to the register, shared-memory and global-memory layers of Section 4. Conv and ReLU are connected at the register layer (L0) and form a tile-graph there. They become a virtual node at the next layer, where the ReLU output is reused by MaxPool through a [2, 2, F] reuse-tile, so all three operators form one tile-graph. At the lowest layer the whole graph is a single tile-graph. The result is a hierarchical tile-graph. Paper, Section 3.2
Where does each intermediate tile live?
Set the reuse layer of the two edges in Conv, ReLU and MaxPool. The figure regroups the operators at each memory layer and propagates a 1 × 1 output tile back through every group.
Conv → ReLU edge is reused in
ReLU → MaxPool edge is reused in
Presets
Conv3x3 → ReLU is reused in register; ReLU → MaxPool2x2 in shared memory. No intermediate tensor is materialized in global memory. Sub-graphs per layer: 2 at register, 1 at shared memory, 1 at global memory.
Register layer (L0 in Figure 5)
Tile-graph: Conv3x3 + ReLU
[3, 3]Conv3x3[1, 1]ReLU[1, 1]Tile-graph: MaxPool2x2
[2, 2]MaxPool2x2[1, 1]
Shared-memory layer (L1)
Tile-graph: Conv3x3 + ReLU + MaxPool2x2
[4, 4, C]Conv3x3 + ReLU[2, 2, F]MaxPool2x2[1, 1, F]
Global-memory layer (L2)
Tile-graph: Conv3x3 + ReLU + MaxPool2x2
[H, W, C]Conv3x3 + ReLU + MaxPool2x2[H/2, W/2, F]
Teal chips are the tiles passed between neighboring nodes of a tile-graph; white chips are the tiles a tile-graph loads from or stores to the layer below it. At the global layer the chips are whole tensors, so a teal chip there is an intermediate tensor in global memory.
The paper's SetConnect call assigns a memory level to an edge, and Propagate infers the tiles of a group from a chosen output tile. Together they update the edges and vertices of the tile-graph. Which reuse layer is best for an edge is not decided by this figure. It is decided by the scheduler described next.
Two-step scheduling
Figure 7 gives the scheduling algorithm in two steps plus an ExtractSubgraph helper (lines 26 to 31), and the interface behind it is small: SetConnect, Propagate, and the two cost functions MemFootprint and MemTraffic (Figure 6).
Graph connecting (lines 1 to 11) visits the operators in topological order. For each output edge it tries every memory level, calls SetConnect, extracts the connected sub-graph, asks the sub-graph scheduler for candidate tile configurations, and profiles them on the device. The level with the best measured latency is kept for that edge.
Sub-graph tiling (lines 12 to 25) searches for tile configurations at one level. It enumerates candidate output tiles by starting from a size of 1 and growing toward shapes that can reduce total traffic and align with hardware features, an approach the paper describes as similar to Roller. Each candidate goes through Propagate. If MemFootprint exceeds the level's capacity the candidate is skipped. Otherwise it is queued with MemTraffic as its priority. The best K candidates are kept, and for each one the algorithm recurses into the sub-graphs of the next level up. Paper, Section 3.2
The device side is described by four interfaces, Allocate, LoadTiles, ComputeTile and StoreTiles, plus MemLevels for querying the hierarchy (Table 1). A recursive routine (Figure 8) walks the hierarchical tile-graph, allocating a workspace per tile-graph, loading input tiles, running the nodes in topological order and storing results. The same routine can emit code or run directly, and Welder currently uses it to emit code. The implementation covers CUDA and ROCm GPUs with three layers, global memory, shared memory and registers, and the Graphcore IPU through the same device interface. It adds host memory as a further layer for very large inputs.
Read the paper beside a runnable model
The rows below connect passages of the paper to the equations and to regions of a standard-library Python reference. The reference models tile propagation for a Matmul and Softmax chain and for the Conv, ReLU and MaxPool chain of Figure 5, the traffic calculation, the grouping by reuse layer, and a one-level version of the Figure 7 search. It is an independently authored teaching model. It is not Welder's code, it runs no GPU kernel, and it does not model the hardware penalties of Section 4.1.
1. Propagate an output tile back to the inputs
Original paper
Section 3.1 describes tile propagation as a chain of shape inferences from the output nodes to the inputs, derived from each operator’s tensor expression.
Math and tensor shapes
The reference gives each operator a propagate method that maps an output tile to its chain input and side inputs. Softmax rejects a tile narrower than its row because the reduction needs the whole row. A window operator uses its stride s and kernel size κ and ignores padding. The paper notes that strided or gathered access gets a conservative upper bound; the reference only handles the regular cases.
Python implementation
@dataclasses.dataclass(frozen=True)
class Matmul:
"""C[m, n] = A[m, k] @ B[k, n], with the reduction axis left untiled."""
k: int
def propagate(self, out_tile):
"""Return (chain input tile, side input tiles) for an output tile."""
rows, cols = out_tile
return (rows, self.k), [(self.k, cols)]
@dataclasses.dataclass(frozen=True)
class RowSoftmax:
"""Softmax over the last axis, so a tile must hold whole rows."""
width: int
def propagate(self, out_tile):
rows, cols = out_tile
if cols != self.width:
raise ValueError("Softmax tiles must span the full row width")
return (rows, cols), []
@dataclasses.dataclass(frozen=True)
class Elementwise:
"""ReLU and other pointwise operators keep the tile shape."""
def propagate(self, out_tile):
return tuple(out_tile), []
@dataclasses.dataclass(frozen=True)
class Window:
"""A sliding window over (height, width, channels); padding is ignored.
Set in_channels for a convolution. Leave it unset for pooling.
"""
kernel: int
stride: int
in_channels: int | None = None
def propagate(self, out_tile):
height, width, channels = out_tile
def extent(size):
return (size - 1) * self.stride + self.kernel
if self.in_channels is None:
return (extent(height), extent(width), channels), []
weights = (self.kernel, self.kernel, self.in_channels, channels)
return (extent(height), extent(width), self.in_channels), [weights]
def propagate_chain(operators, out_tile):
"""Propagate an output tile from the last operator back to the first.
Returns (tiles, side_tiles). tiles[0] feeds operators[0], tiles[i] is the
reuse tile between operators[i - 1] and operators[i], and tiles[-1] is the
output tile. side_tiles holds weights and other non-chain inputs.
"""
tiles = [tuple(out_tile)]
side_tiles = []
for operator in reversed(operators):
chain_tile, sides = operator.propagate(tiles[0])
tiles.insert(0, chain_tile)
side_tiles.extend(sides)
return tiles, side_tiles2. Count the global bytes of a tile-graph
Original paper
Section 2 works one example: a [4×128] output tile moves 35KB, 24,576 tiles move 840MB, and the [16×128] tiling moves 264MB in total.
Math and tensor shapes
The reuse tile C is not counted because Softmax consumes it in shared memory. The self-check reproduces 35,840 B, 24,576 tiles and 880,803,840 B, which is 840 MiB, and then 264 MiB for the 16-row tile. These match the paper only when the B tile is reloaded for every output tile and the paper’s MB means 2^20 bytes. The footprint sums every tile, an upper bound on the paper’s best-fit allocation.
Python implementation
def tile_count(tensor_shape, tile):
"""Count the tile-graphs needed to cover the output tensor exactly."""
if len(tensor_shape) != len(tile):
raise ValueError("Tile and tensor need the same rank")
if any(size % step for size, step in zip(tensor_shape, tile)):
raise ValueError("Tile must divide the output tensor evenly")
return math.prod(size // step for size, step in zip(tensor_shape, tile))
def tile_graph_bytes(tiles, side_tiles, bytes_per_element=FP32_BYTES):
"""Count global bytes for one tile-graph: input, side inputs, output.
Reuse tiles between operators stay on chip and are not counted.
"""
moved = [tiles[0], tiles[-1], *side_tiles]
return sum(math.prod(tile) for tile in moved) * bytes_per_element
def footprint_bytes(tiles, side_tiles, bytes_per_element=FP32_BYTES):
"""Sum every tile; an upper bound, since the paper reuses buffers."""
every = [*tiles, *side_tiles]
return sum(math.prod(tile) for tile in every) * bytes_per_element
def traffic(operators, tensor_shape, out_tile, bytes_per_element=FP32_BYTES):
"""Return per-tile bytes, tile count, total bytes and footprint."""
tiles, side_tiles = propagate_chain(operators, out_tile)
per_tile = tile_graph_bytes(tiles, side_tiles, bytes_per_element)
count = tile_count(tensor_shape, out_tile)
return {
"bytesPerTile": per_tile,
"tileCount": count,
"totalBytes": per_tile * count,
"footprintBytes": footprint_bytes(tiles, side_tiles, bytes_per_element),
}3. Regroup operators from the reuse layer of each edge
Original paper
Figure 5 connects Conv and ReLU at L0 and ReLU and MaxPool at L1, so the three operators form one tile-graph at L1. This page reads L0 and L1 as the register and shared-memory layers of Section 4.
Math and tensor shapes
The paper assigns a memory level to each edge with SetConnect. This reference stores one level per edge and derives the groups at each layer by joining neighbors whose edge is reused at that layer or a faster one. It reproduces the two register-layer groups and the single shared-layer group of Figure 5. The paper’s scheduler also profiles each connection plan on hardware; the reference does not.
Python implementation
def connected_groups(edge_levels, layer):
"""Group a chain of operators at one memory layer.
edge_levels[i] names the layer that holds the tile reused between operators
i and i + 1. Neighbors share a group at `layer` when their edge is reused at
that layer or a faster one.
"""
rank = {name: index for index, name in enumerate(LEVELS)}
if layer not in rank or any(level not in rank for level in edge_levels):
raise ValueError(f"Levels must be one of {LEVELS}")
groups = [[0]]
for index, level in enumerate(edge_levels, start=1):
if rank[level] <= rank[layer]:
groups[-1].append(index)
else:
groups.append([index])
return groups4. Filter by footprint, rank by traffic, keep the top K
Original paper
Lines 12 to 25 of Figure 7: enumerate subtiles, propagate, skip any that exceed the level’s capacity, and push the rest with their traffic as priority.
Math and tensor shapes
This miniature covers one level and omits the recursion into upper-level sub-graphs. Candidates are power-of-two row counts, a simple stand-in for the Roller-style tile expansion the paper describes. The capacity is a parameter; the paper leaves it to the memory description of the target device. The paper’s cost model also adds penalties for uncoalesced access and low parallelism, and an infinite penalty above capacity. Only the last of these is modeled here.
Python implementation
def search_tiles(operators, tensor_shape, candidates, capacity_bytes, top_k,
bytes_per_element=FP32_BYTES):
"""Mimic SubGraphTiling: propagate, drop over capacity, rank by traffic."""
kept = []
for out_tile in candidates:
cost = traffic(operators, tensor_shape, out_tile, bytes_per_element)
if cost["footprintBytes"] > capacity_bytes:
continue
kept.append((cost["totalBytes"], tuple(out_tile), cost))
kept.sort(key=lambda entry: entry[:2])
return [{"outTile": list(tile), **cost} for _, tile, cost in kept[:top_k]]Original excerpts: Yining Shi, Zhi Yang, Jilong Xue, Lingxiao Ma, Yuqing Xia, Ziming Miao, Yuxiao Guo, Fan Yang, Lidong Zhou; OSDI 2023, from the open-access USENIX proceedings. The PDF states open access but no reuse license, so only the passages needed for these rows are shown, unaltered and linked to the source. PDF and crop hashes, page rectangles, and rendering scale are recorded in the source manifest. The Python is an independently authored teaching reference.
Run tile_graph.py with Python 3.10 or later. The README records the scope, and requirements.txt confirms that no external packages are needed. Add --json for the deterministic values that the browser-model tests compare against. The self-check reproduces 35,840 bytes, 24,576 tiles, 840 MiB and 264 MiB exactly, checks that traffic falls and footprint grows with the tile rows, rejects tiles that split a softmax row, and confirms the Figure 5 tile shapes. The official artifact is in the NNFusion repository at the artifact revision; this page does not run it.
What the traffic model leaves out
The cost model in the paper is the traffic model plus corrections for the hardware. Section 4.1 introduces three penalty factors. Uncoalesced access adds the extra transactions it needs, and on CUDA GPUs a contiguous 128 bytes is one transaction. A tile that is too large to leave enough parallel work raises the traffic in proportion to the utilization of the cores. A tile configuration whose footprint exceeds capacity gets an infinite penalty. To avoid enumerating poor candidates, the search tries only the output dimensions that reduce traffic the most and keeps the top K. Paper, Section 4.1
Block size is aligned across operators that share a thread block. On a V100 each SM has 4 warp schedulers and each warp has 32 threads, so the paper requires more than 128 tiles at the register level. It takes the greatest common divisor of the operators' tile numbers as the block size when that is above the hardware parallelism and below the maximum, for example 1,024, and otherwise uses the hardware parallelism. For TensorCore operators, tiles bind to warps, thread counts are multiples of the warp size, and the M, N and K axes of each tile are multiples of the fragment size.
Code generation (Section 4.2) is built on TVM. A register-level connection uses TVM's compute_inline. A shared-memory connection generates standalone kernels for each connected part and composes them into one fused kernel by rewriting global accesses to shared accesses, adding fences and padding for bank conflicts. Shared memory is allocated with a best-fit algorithm over buffer liveness, aligned to 32 bytes where TensorCore needs it. Compilation is accelerated by parallel builds and by a sub-graph signature cache, so that in a 12-layer BERT the first layer's result serves the other eleven. The paper states that Welder takes an ONNX graph, has core mechanisms of 5.2k lines of code, and builds on TVM, Roller and Rammer.
What the paper measured
The evaluation uses 10 models, from MobileNet and BERT to NeRF, NAFNet and Restormer. Together they give 34 compiled test cases. Baselines include PyTorch 1.12, ONNXRuntime 1.12, Ansor, Rammer, TensorRT 8.4, FasterTransformer, BladeDISC and Nimble. Each workload runs for at least 5 seconds after warm-up, only the average speed is reported, and averages across models are geometric means. The hardware is a V100 16 GB in an Azure NC24s_v3 VM (CUDA 11.0), an RTX 3090 (CUDA 11.3), an AMD MI50 16 GB (ROCm 5.2.3) and 16 IPUs (Poplar SDK 3.0). Every number below is paper-reported unless marked as calculated here, and the paper's bar charts (Figures 9 to 15) are used only through the values its text states.
NVIDIA V100, FP32, SIMT cores
At batch size 1, the paper reports geometric-mean speedups of 4.29× over PyTorch and 2.07× over ONNXRuntime. Welder is 1.96× faster than Rammer and 1.44× faster than Ansor on average, and the gap to Ansor shows up in CNNs such as NAFNet (1.70×) and BSRN (1.43×), which are mostly convolutions with relatively small channel counts, and in transformers such as BERT (1.71×) and ViT (1.56×), where the paper says Ansor cannot fuse patterns like LayerNorm or Softmax. Against TensorRT the average is 1.47× and the maximum is 3.09× on NAFNet. On BERT the ratio is 1.02× and on Swin-T it is 0.97×. Paper, Section 5.2
At batch size 64 the averages are 1.83× over PyTorch, 1.90× over ONNXRuntime, 1.47× over Ansor and 1.21× over TensorRT. The paper attributes the smaller gains over libraries to those libraries performing better at large batch sizes. The last three models of its Table 2 cannot be traced in PyTorch at large batch sizes.
NVIDIA V100, TensorCore, FP16
With TensorCore at batch size 1, the paper reports an average of 7.18× over PyTorch, with a maximum of 21.4× on MobileNet, 3.08× over ONNXRuntime with a maximum of 8.72× on Conformer, and 1.53× over TensorRT with a maximum of 2.98× on NAFNet. At batch size 64, on the 7 cases it covers, the averages are 1.98× over PyTorch, 2.13× over ONNXRuntime and 1.16× over TensorRT. NeRF is 2.34× faster than TensorRT on TensorCore against 1.16× on SIMT cores. The paper explains that TensorCore speeds up the compute-bound part and makes the memory-bound remainder matter more. Paper, Section 5.2
Ansor does not support TensorCore, so the paper compares FP16 on SIMT cores in Figure 11, with an average of 1.74× and a maximum of 2.82×. The introduction's 2.8× over Ansor matches this 2.82× maximum, not the average.
Other GPUs and FasterTransformer
Across all 34 cases, the geometric-mean speedup over TensorRT is 1.40× on the RTX 3090, against 1.36× on the V100. Paper, Section 5.2 Against FasterTransformer, Table 3 gives an average of 1.11× with a maximum of 1.73× on ViT. The paper attributes the ViT result at batch size 1 to a single convolution with stride and kernel size 32, where the generated kernel is 4.4× faster. Paper, Section 5.2 On the MI50 the averages at batch size 1 are 2.62× over PyTorch, 1.71× over ONNXRuntime and 1.53× over Ansor. The paper explains the smaller gains by the MI50's weaker peak FLOPS and higher bandwidth, which leave the workload more compute-bound. Paper, Section 5.3
Which fusions were found
Welder discovered about 300 distinct fused sub-graphs, counted by unique operator types across the 34 cases. Of these, 89 contain at least two reduction-based operators, which the simple element-wise fusion rule in Ansor cannot cover. Those 89 patterns range from 2 to 48 operators and average 1.87× over basic fusion, with a maximum of 5.4×. The most common pattern was used 191 times. Table 4 shows two examples, one with 48 operators and one with 13. The paper's sentence that "Figure 4 illustrates two examples of such patterns" refers to that table. Whether Q·K is fused with the following Softmax depends on sequence length: fused in BERT at 128 and not in Conformer at 512, a choice the paper says Welder makes automatically. Paper, Section 5.2
Ablation, sensitivity and compilation time
In an ablation on three models at FP32 and batch size 64, Welder-none disables all inter-operator tile connection and Welder-base connects only at the register layer. Moving from Welder-none to Welder-base cuts latency by an average of 52% (2.08×), kernel launches by 67%, global memory transactions by 52% and intermediate result size by 66%. Adding shared-memory connection on top reduces latency by a further 29% on average, up to 1.82×, launches by 60%, transactions by 25% and intermediate result size by 65%. Transactions fall less than intermediate size because the weights cannot be fused away. Paper, Section 5.2
When input size varies, the fusion gain grows for NAFNet with larger images and shrinks for BERT and Conformer with longer inputs. The paper explains the second effect by the quadratic growth of attention compute, which makes those models less memory-bound.
Table 5 compares compile effort with Ansor. For BERT, Ansor takes 15,285 s over 8,000 trials and Welder takes 244 s over 651. For MobileNet the figures are 45,527 s over 25,600 trials and 561 s over 927. The ratios of those times, calculated here, are about 63× and 81×. The paper attributes the gap to its layered scheduling and analytic cost model, with 20 tuning trials per sub-graph for Welder against 800 per operator for Ansor in its evaluation. Paper, Section 5.2
Larger inputs through host memory
By adding host memory as a layer, Welder runs UNet and VGG16 on large images. In Table 7, UNet at 8k × 8k on a GPU takes 38.2 s for the baseline and 14.5 s for Welder, and VGG16 takes 15.7 s and 8.30 s. On an IPU at 2k × 2k, UNet takes 31.1 s and 8.56 s, and VGG16 takes 4.98 s and 1.61 s. The text gives speedups of 2.63× and 1.89× on the GPU and 3.63× and 3.09× on the IPU, with host transfers reduced by 3.11× and 2.90× on the GPU. The baseline is a variant that disables data reuse at the device memory layer, so these results show the value of the host-memory plan, not a comparison with another system. The paper calls them preliminary. Paper, Section 5.4
Where Welder does not win
The paper reports several cases where the gain is small or negative. Each deserves attention when judging how far the results transfer.
- Convolution-dominated models. In Table 6, UNet takes 9.251 for Welder against 4.429 for TensorRT, and VGG16 takes 4.123 against 2.584. The table prints no unit, and lower is better, consistent with the text. The paper explains that TensorRT uses Winograd for the dominant convolutions, that Welder has no further fusion to exploit, and that Winograd support is left as future work. ResNet50 and ResNet18 are close to TensorRT. Welder also trails Ansor on UNet (9.251 against 8.670) and ResNet18 (1.094 against 1.071) in the same table.
- Popular transformers. On the V100 with FP32 SIMT cores at batch size 1, Welder is at 1.02× on BERT and 0.97× on Swin-T against TensorRT, which already has expert fusion rules for them. In Table 3, BERT at FP16 and batch size 64 takes 24.82 ms for Welder against 22.29 ms for FasterTransformer, and Swin-T at FP32 and batch size 1 takes 2.59 ms against 2.38 ms.
- Compute-dominated MLPs. NeRF improves on Ansor by only 1.09× on SIMT cores because the compute-bound MLP dominates.
- Other hardware. The MI50 speedup is slightly smaller over PyTorch and ONNXRuntime (2.62× and 1.71× against 4.29× and 2.07× at batch size 1), as the paper puts it, for the reason given above.
- Longer sequences. The fusion gain shrinks for BERT and Conformer as input length grows (Figure 14).
- Dynamic models. The design targets static models. The paper suggests turning a dynamic graph into static sub-graphs through JIT compilation, or keeping static tile plans while the number of parallel tasks follows the input shape (Section 6). Paper, Section 6
Two further caveats apply to the whole evaluation. The results are the authors' own measurements on the hardware and library versions listed above, and Welder's compute-heavy kernels are less tuned than vendor libraries. The PDF also carries USENIX artifact-evaluated badges, and its appendix states that the artifact reproduces the main V100 results. This page did not rerun it.
What to carry forward
The reusable idea is the order of operations. Pick the output tile of the last operator in a chain, propagate it back, count the global bytes, discard tiles whose footprint does not fit the layer, and only then profile the survivors. Each step before the profiling narrows the search without running a kernel.
Several related reviews look at the same cost from other angles. Liger Kernel: Fused SwiGLU counts the logical traffic of one pointwise fusion boundary with the same style of analytic byte count, and Welder's contribution is searching over many such boundaries. Making Deep Learning Go Brrrr explains how to tell whether a workload is memory-bound, which is the premise of Welder's motivation. Data Movement Is All You Need studies the same cost for transformers. FlashAttention resolves a Matmul and Softmax tile conflict by hand for attention by tiling through on-chip memory, whereas Welder derives a shared tile from the operators' expressions. Online Normalizer Calculation for Softmax removes the requirement that a Softmax tile hold a whole row by keeping a running normalizer. The FlashAttention and online softmax connections are readings across papers and not claims the papers make.
When a fusion result is quoted, check four things: which layer holds the intermediate, which tile shape was chosen and what footprint it needs, which baseline and precision the speedup uses, and whether it is an average or a maximum. The paper's own ratios run from below 0.5× (UNet against TensorRT in Table 6, a ratio calculated here from the table) to 21.4×, depending on those conditions, and its memory access patterns assumptions, coalesced 128-byte transactions in particular, determine how far the analytic traffic tracks the real cost.
