Skip to main content

Welder: Scheduling Deep Learning Memory Access via Tile-graph

Follow how Welder propagates an output tile back through a chain of operators, counts tile traffic analytically, and chooses where each intermediate tile lives, with each reported speedup tied to its stated condition.

TL;DR

  • The paper reports that, for ONNXRuntime on a V100, average compute-core utilization is 51.6% while memory bandwidth utilization reaches 96.7%, which the paper reads as GPU memory being the end-to-end bottleneck of recent models.
  • Welder describes a model as a graph of tiles. Choosing an output tile fixes the shape of every input tile behind it, up to how the reduction axis is split, and the global traffic of the resulting tile-graph follows from tile and tensor shapes alone, with no kernel run.
  • For a Matmul followed by a Softmax, the worked example gives 840MB of global traffic with a [4 × 128] output tile and 264MB with [16 × 128] (exactly 840 and 264 MiB, as the body shows). Each edge also gets a reuse layer (register, shared memory or global memory) that decides where its intermediate tile lives.
  • The headline speedups come with conditions: up to 21.4× over PyTorch is MobileNet with TensorCore, FP16, batch size 1 on a V100, while the average over TensorRT is 1.47× at FP32, batch size 1.
  • Welder loses to TensorRT on UNet and VGG16, where TensorRT runs the dominant convolutions with Winograd, and the design targets static models. The traffic model on this page is a teaching calculation and does not predict latency.

Why memory traffic became the bottleneck

Compilers have long treated a neural network as a graph of compute-heavy operators and optimized each operator, or fused simple neighbors. Shi et al. argue that this view no longer fits. In their profiling of ONNXRuntime on a V100 (Figure 1), classical models such as ResNet and BERT reach compute utilization above 80%, but the more recent models in the study mix in memory-intensive operators and use the cores less. The paper also reports a higher ratio of memory stores to loads for the newer models, which it attributes to large activations produced from high-fidelity inputs. Paper, Section 2

If intermediate tensors travel through global memory between operators, the remedy is to keep them on chip. That is hard because each operator wants its own tile shape, and a tile that is best for one operator may not be usable by the next. The question the paper answers is how to choose tile shapes and memory layers for a whole chain of operators rather than one operator at a time. The GPU memory hierarchy concept covers the layers involved, and the streaming multiprocessor page covers where shared memory sits.

Three evidence classes appear on this page and are labeled where they occur. Paper-reported values are measurements or statements from the paper. Calculated here values come from the formulas shown, evaluated by the page's JavaScript and checked against a Python reference. Authored illustrations explain a definition and are not Welder output. The page measures nothing on a GPU.

One pair of operators, two optimal tiles

The paper's motivating example is a Matmul followed by a Softmax. Optimized independently, the best shared-memory tile for the Matmul is [32 × 64] and for the Softmax is [4 × 128]. Those shapes do not match, so the Softmax cannot read the Matmul's output from shared memory, and the pair takes 0.36 ms (Figure 2). Forcing a shape that serves both operators, [16 × 128], lets them fuse through shared memory, and the pair takes 0.29 ms, a 1.26× speedup. Run separately, the two operators each lose some of their stand-alone efficiency with that aligned tile, 15% and 4% in the paper's account, and the fused kernel still wins. Paper, Section 2

The lesson is that the best tile for a fused pair is not the best tile of either member. Finding it for long chains requires a rule for deriving the shared tile and a cost that compares candidates without running them. The paper supplies both.

Three observations that make the search tractable

The design rests on three observations, each of which the later sections use.

  1. Back-to-front shape inference. Given the tensor expression of each operator, an output tile of a Softmax determines its input tile, and that determines the input tiles of the Matmul behind it. Choosing the last operator's tile fixes the shape of every tile behind it, apart from how the reduction axis is split.
  2. Traffic is analytic. Once tile shapes and tensor shapes are known, the traffic follows by arithmetic. Traffic ranks tile choices at one memory layer without profiling; Welder profiles only the top K.
  3. Inter-layer independence. The traffic at one memory layer depends only on the tile configuration at that layer, so each layer can be optimized separately. The only constraint between layers is that a lower-level tile must be larger than the tile above it.

The second observation is the one a reader can check by hand, and the explorer below does that.

Working the traffic calculation

Let M × N be the output tensor D, K the reduction size of the Matmul, q the bytes per element, and [m × n] the output tile. The Softmax reduces over a row, so a legal tile keeps n equal to the row width. A tile-graph loads its input tiles, keeps the reuse tile on chip, and stores its output tile:

\begin{aligned} \text{bytes per tile}&=q\,\bigl(m K+K n+m n\bigr) \\ N_{\text{tiles}}&=\frac{M N}{m\,n} \\ \text{total}&=\text{bytes per tile}\times N_{\text{tiles}} \end{aligned}

The paper's tensors are A [98304 × 64], B [64 × 128] and D [98304 × 128] in FP32. For a [4 × 128] output tile, one tile-graph moves (4·64 + 64·128 + 4·128) × 4 = 35,840 bytes, and 24,576 tiles are needed, so the total is 880,803,840 bytes. The paper reports these as 35KB and 840MB. They agree only if the B tile is reloaded for every output tile and the units are 1,024 and 2^20 bytes, so 880,803,840 bytes is exactly 840 MiB. Calculated here: a [16 × 128] tile moves 45,056 bytes, needs 6,144 tiles, and totals 276,824,064 bytes, which is 264 MiB. The paper states only the 264MB total. Paper, Section 2

Pick an output tile, derive the traffic

Choose how many rows of D one tile computes. Softmax reduces over each row, so a tile always spans all 128 columns. The input tiles follow from the output tile, and global traffic follows from the tile shapes alone.

64 KiB
3

Tiles propagated back to front for D tile [4 × 128]

1. Output tile D (chosen)

[4 × 128]

2,048 B · global traffic

Written once to global memory.

2. Reuse tile C (Softmax input)

[4 × 128]

2,048 B · stays in shared memory

Softmax needs the same shape. This tile is what fusion keeps on chip.

3. Input tile A

[4 × 64]

1,024 B · global traffic

Matmul needs all 64 reduction values for these rows.

4. Input tile B

[64 × 128]

32,768 B · global traffic

The same for every output tile; the paper’s formula counts it once per output tile.

Global bytes per tile

35,840 B

(4·64 + 64·128 + 4·128) × 4 B

Tiles to cover D

24,576

98,304 ÷ 4 rows

Total global traffic

840 MiB

880,803,840 B

Shared-memory footprint

37 KiB

Fits the assumed 64 KiB budget.

Paper-statedThe paper states 35KB per tile, 24,576 tiles and 840MB in total for this tile.Reader assumptionThe budget and K are set by the reader. The 64 KiB default is arbitrary; at that value the model ranks [16 × 128] first, which does not reconstruct why the paper chose that tile.

Keeping C on chip avoids writing and rereading 48 MiB each way, but at 4 rows the repeated B loads are 768 MiB of the 840 MiB total. Under the assumed budget 5 of 9 candidates fit; the top 3 by traffic are [16 × 128], [8 × 128], [4 × 128]. The selected tile ranks 3.

Candidate output tiles with traffic, footprint and rank
Output tileTotal trafficFootprintStatus
[1 × 128]
3144 MiB
33.25 KiBFits, outside top K
[2 × 128]
1608 MiB
34.5 KiBFits, outside top K
[4 × 128]
840 MiB
37 KiBKept, rank 3
[8 × 128]
456 MiB
42 KiBKept, rank 2
[16 × 128]
264 MiB
52 KiBKept, rank 1
[32 × 128]
168 MiB
72 KiBDropped: over budget
[64 × 128]
120 MiB
112 KiBDropped: over budget
[128 × 128]
96 MiB
192 KiBDropped: over budget
[256 × 128]
84 MiB
352 KiBDropped: over budget
Evidence classes: the 35KB, 24,576, 840MB and 264MB values come from the paper's Section 2. Every other row is calculated here with the same formula (FP32, the B tile reloaded for every output tile, 1KB = 1,024 B and 1MB = 2^20 B, which is what reproduces the paper figures). The footprint sums every tile without buffer reuse, an upper bound on what a best-fit allocator would need. The budget is set by the reader and is not a hardware specification. In this model only capacity stops tile growth; the paper also penalizes tiles that leave too little parallelism (Section 4.1). This is a traffic model, not a latency predictor: it omits the coalescing and parallelism penalties of Section 4.1 and the profiling that picks the final plan.

Rearranging the formula for this case shows where the savings come from. The A reads and D writes cost 72 MiB however the output is tiled, and the repeated B loads cost 3,072 MiB divided by the number of rows in the tile. This split is calculated here, not stated by the paper. Larger tiles amortize B better, but a bigger tile set has a bigger footprint. In this model only a capacity limit stops the growth; in the paper, Section 4.1 also penalizes tiles that leave too little parallelism. The paper does not state the capacity or search bound behind its [16 × 128] choice, so the figure leaves the budget to the reader. The 64 KiB default is arbitrary: at that value the model happens to rank [16 × 128] first, which follows from the assumed budget and does not reconstruct the paper's choice.

Counting only global bytes also leaves things out. It does not model coalescing or core utilization, and it ignores any cache effect. Section 4.1 below describes how the paper adds some of these back.

A Matmul and Softmax pair needs only one reuse tile. A real model chains many operators across three memory layers, so the next step is a vocabulary for where each tile lives.

From operator-tiles to a hierarchical tile-graph

An operator-tile computes one output tile of an operator from the input region it depends on. In the paper's Figure 4, a convolution tile reads a [3 × 3] window to produce one output position and a pooling tile reads [2 × 2] to produce one. Two adjacent operator-tiles are connected through a common intermediate tile, the reuse-tile, so the second consumes what the first produced without a full intermediate tensor. Connected operator-tiles form a tile-graph. Paper, Section 3.1

Tile propagation runs from the output nodes back to the inputs using each operator's tensor expression. For irregular access such as Gather or strided convolution, the paper uses a conservative upper bound as the input tile. For a tile-graph, the traffic is the sum of its input and output tile sizes multiplied by the number of tile-graphs needed to cover the output, and the minimum footprint comes from a memory allocation algorithm such as best-fit applied to all tiles in topological order. Input tiles that contain reduction axes can also be split into smaller tiles consumed sequentially.

Mapping onto a memory hierarchy repeats the idea once per layer. Figure 5 labels its layers L0 to L2, and this page maps them to the register, shared-memory and global-memory layers of Section 4. Conv and ReLU are connected at the register layer (L0) and form a tile-graph there. They become a virtual node at the next layer, where the ReLU output is reused by MaxPool through a [2, 2, F] reuse-tile, so all three operators form one tile-graph. At the lowest layer the whole graph is a single tile-graph. The result is a hierarchical tile-graph. Paper, Section 3.2

Where does each intermediate tile live?

Set the reuse layer of the two edges in Conv, ReLU and MaxPool. The figure regroups the operators at each memory layer and propagates a 1 × 1 output tile back through every group.

Conv → ReLU edge is reused in

ReLU → MaxPool edge is reused in

Presets

Conv3x3 → ReLU is reused in register; ReLU → MaxPool2x2 in shared memory. No intermediate tensor is materialized in global memory. Sub-graphs per layer: 2 at register, 1 at shared memory, 1 at global memory.

Register layer (L0 in Figure 5)

  • Tile-graph: Conv3x3 + ReLU

    [3, 3]Conv3x3[1, 1]ReLU[1, 1]
  • Tile-graph: MaxPool2x2

    [2, 2]MaxPool2x2[1, 1]

Shared-memory layer (L1)

  • Tile-graph: Conv3x3 + ReLU + MaxPool2x2

    [4, 4, C]Conv3x3 + ReLU[2, 2, F]MaxPool2x2[1, 1, F]

Global-memory layer (L2)

  • Tile-graph: Conv3x3 + ReLU + MaxPool2x2

    [H, W, C]Conv3x3 + ReLU + MaxPool2x2[H/2, W/2, F]

Teal chips are the tiles passed between neighboring nodes of a tile-graph; white chips are the tiles a tile-graph loads from or stores to the layer below it. At the global layer the chips are whole tensors, so a teal chip there is an intermediate tensor in global memory.

Authored illustration An illustration of the definitions in the paper's Section 3.2, not output from Welder. Tile shapes follow from the operators: a 3 × 3 convolution at stride 1 needs out + 2 input rows and columns and a 2 × 2 pool at stride 2 needs 2 × out, ignoring padding. C and F are input and output channel counts. Figure 5 draws register-layer tiles as 2-D shapes and gives channel extents from the next layer down, so this figure does the same; that keeps each register tile smaller than the tile above it, the constraint quoted in Section 3.2. The paper's scheduler enumerates a layer per edge and profiles each plan; this figure does not judge which plan is fastest. Figure 5 labels its layers L0 to L2 without naming them. This page maps them to register, shared memory and global memory, the three GPU layers of Section 4; Section 3.2 and Figure 8 number the lowest level 0. Whole-tensor shapes at the global layer are those drawn in Figures 4 and 5.

The paper's SetConnect call assigns a memory level to an edge, and Propagate infers the tiles of a group from a chosen output tile. Together they update the edges and vertices of the tile-graph. Which reuse layer is best for an edge is not decided by this figure. It is decided by the scheduler described next.

Two-step scheduling

Figure 7 gives the scheduling algorithm in two steps plus an ExtractSubgraph helper (lines 26 to 31), and the interface behind it is small: SetConnect, Propagate, and the two cost functions MemFootprint and MemTraffic (Figure 6).

Graph connecting (lines 1 to 11) visits the operators in topological order. For each output edge it tries every memory level, calls SetConnect, extracts the connected sub-graph, asks the sub-graph scheduler for candidate tile configurations, and profiles them on the device. The level with the best measured latency is kept for that edge.

Sub-graph tiling (lines 12 to 25) searches for tile configurations at one level. It enumerates candidate output tiles by starting from a size of 1 and growing toward shapes that can reduce total traffic and align with hardware features, an approach the paper describes as similar to Roller. Each candidate goes through Propagate. If MemFootprint exceeds the level's capacity the candidate is skipped. Otherwise it is queued with MemTraffic as its priority. The best K candidates are kept, and for each one the algorithm recurses into the sub-graphs of the next level up. Paper, Section 3.2

The device side is described by four interfaces, Allocate, LoadTiles, ComputeTile and StoreTiles, plus MemLevels for querying the hierarchy (Table 1). A recursive routine (Figure 8) walks the hierarchical tile-graph, allocating a workspace per tile-graph, loading input tiles, running the nodes in topological order and storing results. The same routine can emit code or run directly, and Welder currently uses it to emit code. The implementation covers CUDA and ROCm GPUs with three layers, global memory, shared memory and registers, and the Graphcore IPU through the same device interface. It adds host memory as a further layer for very large inputs.

Read the paper beside a runnable model

The rows below connect passages of the paper to the equations and to regions of a standard-library Python reference. The reference models tile propagation for a Matmul and Softmax chain and for the Conv, ReLU and MaxPool chain of Figure 5, the traffic calculation, the grouping by reuse layer, and a one-level version of the Figure 7 search. It is an independently authored teaching model. It is not Welder's code, it runs no GPU kernel, and it does not model the hardware penalties of Section 4.1.

1. Propagate an output tile back to the inputs

Original paper

Section 3.1: tile propagation

Section 3.1 describes tile propagation as a chain of shape inferences from the output nodes to the inputs, derived from each operator’s tensor expression.

Math and tensor shapes

\begin{aligned}\text{Matmul}&:\ [m\times n]\to A[m\times k]\\ &\phantom{:\ [m\times n]}\ \ B[k\times n]\\ \text{Softmax}&:\ [m\times n]\to[m\times n]\\ \text{Window}&:\ \text{in}=(\text{out}-1)\,s+\kappa\end{aligned}

The reference gives each operator a propagate method that maps an output tile to its chain input and side inputs. Softmax rejects a tile narrower than its row because the reduction needs the whole row. A window operator uses its stride s and kernel size κ and ignores padding. The paper notes that strided or gathered access gets a conservative upper bound; the reference only handles the regular cases.

Python implementation

tile_graph.py · propagate
python
@dataclasses.dataclass(frozen=True)
class Matmul:
    """C[m, n] = A[m, k] @ B[k, n], with the reduction axis left untiled."""

    k: int

    def propagate(self, out_tile):
        """Return (chain input tile, side input tiles) for an output tile."""
        rows, cols = out_tile
        return (rows, self.k), [(self.k, cols)]


@dataclasses.dataclass(frozen=True)
class RowSoftmax:
    """Softmax over the last axis, so a tile must hold whole rows."""

    width: int

    def propagate(self, out_tile):
        rows, cols = out_tile
        if cols != self.width:
            raise ValueError("Softmax tiles must span the full row width")
        return (rows, cols), []


@dataclasses.dataclass(frozen=True)
class Elementwise:
    """ReLU and other pointwise operators keep the tile shape."""

    def propagate(self, out_tile):
        return tuple(out_tile), []


@dataclasses.dataclass(frozen=True)
class Window:
    """A sliding window over (height, width, channels); padding is ignored.

    Set in_channels for a convolution. Leave it unset for pooling.
    """

    kernel: int
    stride: int
    in_channels: int | None = None

    def propagate(self, out_tile):
        height, width, channels = out_tile

        def extent(size):
            return (size - 1) * self.stride + self.kernel

        if self.in_channels is None:
            return (extent(height), extent(width), channels), []
        weights = (self.kernel, self.kernel, self.in_channels, channels)
        return (extent(height), extent(width), self.in_channels), [weights]


def propagate_chain(operators, out_tile):
    """Propagate an output tile from the last operator back to the first.

    Returns (tiles, side_tiles). tiles[0] feeds operators[0], tiles[i] is the
    reuse tile between operators[i - 1] and operators[i], and tiles[-1] is the
    output tile. side_tiles holds weights and other non-chain inputs.
    """
    tiles = [tuple(out_tile)]
    side_tiles = []
    for operator in reversed(operators):
        chain_tile, sides = operator.propagate(tiles[0])
        tiles.insert(0, chain_tile)
        side_tiles.extend(sides)
    return tiles, side_tiles

2. Count the global bytes of a tile-graph

Original paper

Section 2: worked traffic calculation

Section 2 works one example: a [4×128] output tile moves 35KB, 24,576 tiles move 840MB, and the [16×128] tiling moves 264MB in total.

Math and tensor shapes

\begin{aligned}\text{bytes per tile}&=q\,(\lvert A\rvert+\lvert B\rvert+\lvert D\rvert)\\ N_{\text{tiles}}&=\frac{M\,N}{m\,n}\\ \text{total}&=\text{bytes per tile}\times N_{\text{tiles}}\end{aligned}

The reuse tile C is not counted because Softmax consumes it in shared memory. The self-check reproduces 35,840 B, 24,576 tiles and 880,803,840 B, which is 840 MiB, and then 264 MiB for the 16-row tile. These match the paper only when the B tile is reloaded for every output tile and the paper’s MB means 2^20 bytes. The footprint sums every tile, an upper bound on the paper’s best-fit allocation.

Python implementation

tile_graph.py · traffic
python
def tile_count(tensor_shape, tile):
    """Count the tile-graphs needed to cover the output tensor exactly."""
    if len(tensor_shape) != len(tile):
        raise ValueError("Tile and tensor need the same rank")
    if any(size % step for size, step in zip(tensor_shape, tile)):
        raise ValueError("Tile must divide the output tensor evenly")
    return math.prod(size // step for size, step in zip(tensor_shape, tile))


def tile_graph_bytes(tiles, side_tiles, bytes_per_element=FP32_BYTES):
    """Count global bytes for one tile-graph: input, side inputs, output.

    Reuse tiles between operators stay on chip and are not counted.
    """
    moved = [tiles[0], tiles[-1], *side_tiles]
    return sum(math.prod(tile) for tile in moved) * bytes_per_element


def footprint_bytes(tiles, side_tiles, bytes_per_element=FP32_BYTES):
    """Sum every tile; an upper bound, since the paper reuses buffers."""
    every = [*tiles, *side_tiles]
    return sum(math.prod(tile) for tile in every) * bytes_per_element


def traffic(operators, tensor_shape, out_tile, bytes_per_element=FP32_BYTES):
    """Return per-tile bytes, tile count, total bytes and footprint."""
    tiles, side_tiles = propagate_chain(operators, out_tile)
    per_tile = tile_graph_bytes(tiles, side_tiles, bytes_per_element)
    count = tile_count(tensor_shape, out_tile)
    return {
        "bytesPerTile": per_tile,
        "tileCount": count,
        "totalBytes": per_tile * count,
        "footprintBytes": footprint_bytes(tiles, side_tiles, bytes_per_element),
    }

3. Regroup operators from the reuse layer of each edge

Original paper

Figure 5: Conv, ReLU and MaxPool on a three-layer hierarchy

Figure 5 connects Conv and ReLU at L0 and ReLU and MaxPool at L1, so the three operators form one tile-graph at L1. This page reads L0 and L1 as the register and shared-memory layers of Section 4.

Math and tensor shapes

\begin{aligned}&\text{rank: register}<\text{shared}<\text{global}\\ &i\sim i{+}1\text{ at layer }\ell\\ &\quad\iff\operatorname{rank}(e_i)\le\operatorname{rank}(\ell)\end{aligned}

The paper assigns a memory level to each edge with SetConnect. This reference stores one level per edge and derives the groups at each layer by joining neighbors whose edge is reused at that layer or a faster one. It reproduces the two register-layer groups and the single shared-layer group of Figure 5. The paper’s scheduler also profiles each connection plan on hardware; the reference does not.

Python implementation

tile_graph.py · connect
python
def connected_groups(edge_levels, layer):
    """Group a chain of operators at one memory layer.

    edge_levels[i] names the layer that holds the tile reused between operators
    i and i + 1. Neighbors share a group at `layer` when their edge is reused at
    that layer or a faster one.
    """
    rank = {name: index for index, name in enumerate(LEVELS)}
    if layer not in rank or any(level not in rank for level in edge_levels):
        raise ValueError(f"Levels must be one of {LEVELS}")
    groups = [[0]]
    for index, level in enumerate(edge_levels, start=1):
        if rank[level] <= rank[layer]:
            groups[-1].append(index)
        else:
            groups.append([index])
    return groups

4. Filter by footprint, rank by traffic, keep the top K

Original paper

Figure 7: SubGraphTiling, lines 12 to 25

Lines 12 to 25 of Figure 7: enumerate subtiles, propagate, skip any that exceed the level’s capacity, and push the rest with their traffic as priority.

Math and tensor shapes

\begin{aligned}\mathcal{C}&=\{t:\operatorname{Footprint}(t)\le\text{capacity}\}\\ \text{kept}&=\operatorname{TopK}_{t\in\mathcal{C}}\ \operatorname{Traffic}(t)\end{aligned}

This miniature covers one level and omits the recursion into upper-level sub-graphs. Candidates are power-of-two row counts, a simple stand-in for the Roller-style tile expansion the paper describes. The capacity is a parameter; the paper leaves it to the memory description of the target device. The paper’s cost model also adds penalties for uncoalesced access and low parallelism, and an infinite penalty above capacity. Only the last of these is modeled here.

Python implementation

tile_graph.py · schedule
python
def search_tiles(operators, tensor_shape, candidates, capacity_bytes, top_k,
                 bytes_per_element=FP32_BYTES):
    """Mimic SubGraphTiling: propagate, drop over capacity, rank by traffic."""
    kept = []
    for out_tile in candidates:
        cost = traffic(operators, tensor_shape, out_tile, bytes_per_element)
        if cost["footprintBytes"] > capacity_bytes:
            continue
        kept.append((cost["totalBytes"], tuple(out_tile), cost))
    kept.sort(key=lambda entry: entry[:2])
    return [{"outTile": list(tile), **cost} for _, tile, cost in kept[:top_k]]

Original excerpts: Yining Shi, Zhi Yang, Jilong Xue, Lingxiao Ma, Yuqing Xia, Ziming Miao, Yuxiao Guo, Fan Yang, Lidong Zhou; OSDI 2023, from the open-access USENIX proceedings. The PDF states open access but no reuse license, so only the passages needed for these rows are shown, unaltered and linked to the source. PDF and crop hashes, page rectangles, and rendering scale are recorded in the source manifest. The Python is an independently authored teaching reference.

Run tile_graph.py with Python 3.10 or later. The README records the scope, and requirements.txt confirms that no external packages are needed. Add --json for the deterministic values that the browser-model tests compare against. The self-check reproduces 35,840 bytes, 24,576 tiles, 840 MiB and 264 MiB exactly, checks that traffic falls and footprint grows with the tile rows, rejects tiles that split a softmax row, and confirms the Figure 5 tile shapes. The official artifact is in the NNFusion repository at the artifact revision; this page does not run it.

What the traffic model leaves out

The cost model in the paper is the traffic model plus corrections for the hardware. Section 4.1 introduces three penalty factors. Uncoalesced access adds the extra transactions it needs, and on CUDA GPUs a contiguous 128 bytes is one transaction. A tile that is too large to leave enough parallel work raises the traffic in proportion to the utilization of the cores. A tile configuration whose footprint exceeds capacity gets an infinite penalty. To avoid enumerating poor candidates, the search tries only the output dimensions that reduce traffic the most and keeps the top K. Paper, Section 4.1

Block size is aligned across operators that share a thread block. On a V100 each SM has 4 warp schedulers and each warp has 32 threads, so the paper requires more than 128 tiles at the register level. It takes the greatest common divisor of the operators' tile numbers as the block size when that is above the hardware parallelism and below the maximum, for example 1,024, and otherwise uses the hardware parallelism. For TensorCore operators, tiles bind to warps, thread counts are multiples of the warp size, and the M, N and K axes of each tile are multiples of the fragment size.

Code generation (Section 4.2) is built on TVM. A register-level connection uses TVM's compute_inline. A shared-memory connection generates standalone kernels for each connected part and composes them into one fused kernel by rewriting global accesses to shared accesses, adding fences and padding for bank conflicts. Shared memory is allocated with a best-fit algorithm over buffer liveness, aligned to 32 bytes where TensorCore needs it. Compilation is accelerated by parallel builds and by a sub-graph signature cache, so that in a 12-layer BERT the first layer's result serves the other eleven. The paper states that Welder takes an ONNX graph, has core mechanisms of 5.2k lines of code, and builds on TVM, Roller and Rammer.

What the paper measured

The evaluation uses 10 models, from MobileNet and BERT to NeRF, NAFNet and Restormer. Together they give 34 compiled test cases. Baselines include PyTorch 1.12, ONNXRuntime 1.12, Ansor, Rammer, TensorRT 8.4, FasterTransformer, BladeDISC and Nimble. Each workload runs for at least 5 seconds after warm-up, only the average speed is reported, and averages across models are geometric means. The hardware is a V100 16 GB in an Azure NC24s_v3 VM (CUDA 11.0), an RTX 3090 (CUDA 11.3), an AMD MI50 16 GB (ROCm 5.2.3) and 16 IPUs (Poplar SDK 3.0). Every number below is paper-reported unless marked as calculated here, and the paper's bar charts (Figures 9 to 15) are used only through the values its text states.

NVIDIA V100, FP32, SIMT cores

At batch size 1, the paper reports geometric-mean speedups of 4.29× over PyTorch and 2.07× over ONNXRuntime. Welder is 1.96× faster than Rammer and 1.44× faster than Ansor on average, and the gap to Ansor shows up in CNNs such as NAFNet (1.70×) and BSRN (1.43×), which are mostly convolutions with relatively small channel counts, and in transformers such as BERT (1.71×) and ViT (1.56×), where the paper says Ansor cannot fuse patterns like LayerNorm or Softmax. Against TensorRT the average is 1.47× and the maximum is 3.09× on NAFNet. On BERT the ratio is 1.02× and on Swin-T it is 0.97×. Paper, Section 5.2

At batch size 64 the averages are 1.83× over PyTorch, 1.90× over ONNXRuntime, 1.47× over Ansor and 1.21× over TensorRT. The paper attributes the smaller gains over libraries to those libraries performing better at large batch sizes. The last three models of its Table 2 cannot be traced in PyTorch at large batch sizes.

NVIDIA V100, TensorCore, FP16

With TensorCore at batch size 1, the paper reports an average of 7.18× over PyTorch, with a maximum of 21.4× on MobileNet, 3.08× over ONNXRuntime with a maximum of 8.72× on Conformer, and 1.53× over TensorRT with a maximum of 2.98× on NAFNet. At batch size 64, on the 7 cases it covers, the averages are 1.98× over PyTorch, 2.13× over ONNXRuntime and 1.16× over TensorRT. NeRF is 2.34× faster than TensorRT on TensorCore against 1.16× on SIMT cores. The paper explains that TensorCore speeds up the compute-bound part and makes the memory-bound remainder matter more. Paper, Section 5.2

Ansor does not support TensorCore, so the paper compares FP16 on SIMT cores in Figure 11, with an average of 1.74× and a maximum of 2.82×. The introduction's 2.8× over Ansor matches this 2.82× maximum, not the average.

Other GPUs and FasterTransformer

Across all 34 cases, the geometric-mean speedup over TensorRT is 1.40× on the RTX 3090, against 1.36× on the V100. Paper, Section 5.2 Against FasterTransformer, Table 3 gives an average of 1.11× with a maximum of 1.73× on ViT. The paper attributes the ViT result at batch size 1 to a single convolution with stride and kernel size 32, where the generated kernel is 4.4× faster. Paper, Section 5.2 On the MI50 the averages at batch size 1 are 2.62× over PyTorch, 1.71× over ONNXRuntime and 1.53× over Ansor. The paper explains the smaller gains by the MI50's weaker peak FLOPS and higher bandwidth, which leave the workload more compute-bound. Paper, Section 5.3

Which fusions were found

Welder discovered about 300 distinct fused sub-graphs, counted by unique operator types across the 34 cases. Of these, 89 contain at least two reduction-based operators, which the simple element-wise fusion rule in Ansor cannot cover. Those 89 patterns range from 2 to 48 operators and average 1.87× over basic fusion, with a maximum of 5.4×. The most common pattern was used 191 times. Table 4 shows two examples, one with 48 operators and one with 13. The paper's sentence that "Figure 4 illustrates two examples of such patterns" refers to that table. Whether Q·K is fused with the following Softmax depends on sequence length: fused in BERT at 128 and not in Conformer at 512, a choice the paper says Welder makes automatically. Paper, Section 5.2

Ablation, sensitivity and compilation time

In an ablation on three models at FP32 and batch size 64, Welder-none disables all inter-operator tile connection and Welder-base connects only at the register layer. Moving from Welder-none to Welder-base cuts latency by an average of 52% (2.08×), kernel launches by 67%, global memory transactions by 52% and intermediate result size by 66%. Adding shared-memory connection on top reduces latency by a further 29% on average, up to 1.82×, launches by 60%, transactions by 25% and intermediate result size by 65%. Transactions fall less than intermediate size because the weights cannot be fused away. Paper, Section 5.2

When input size varies, the fusion gain grows for NAFNet with larger images and shrinks for BERT and Conformer with longer inputs. The paper explains the second effect by the quadratic growth of attention compute, which makes those models less memory-bound.

Table 5 compares compile effort with Ansor. For BERT, Ansor takes 15,285 s over 8,000 trials and Welder takes 244 s over 651. For MobileNet the figures are 45,527 s over 25,600 trials and 561 s over 927. The ratios of those times, calculated here, are about 63× and 81×. The paper attributes the gap to its layered scheduling and analytic cost model, with 20 tuning trials per sub-graph for Welder against 800 per operator for Ansor in its evaluation. Paper, Section 5.2

Larger inputs through host memory

By adding host memory as a layer, Welder runs UNet and VGG16 on large images. In Table 7, UNet at 8k × 8k on a GPU takes 38.2 s for the baseline and 14.5 s for Welder, and VGG16 takes 15.7 s and 8.30 s. On an IPU at 2k × 2k, UNet takes 31.1 s and 8.56 s, and VGG16 takes 4.98 s and 1.61 s. The text gives speedups of 2.63× and 1.89× on the GPU and 3.63× and 3.09× on the IPU, with host transfers reduced by 3.11× and 2.90× on the GPU. The baseline is a variant that disables data reuse at the device memory layer, so these results show the value of the host-memory plan, not a comparison with another system. The paper calls them preliminary. Paper, Section 5.4

Where Welder does not win

The paper reports several cases where the gain is small or negative. Each deserves attention when judging how far the results transfer.

  • Convolution-dominated models. In Table 6, UNet takes 9.251 for Welder against 4.429 for TensorRT, and VGG16 takes 4.123 against 2.584. The table prints no unit, and lower is better, consistent with the text. The paper explains that TensorRT uses Winograd for the dominant convolutions, that Welder has no further fusion to exploit, and that Winograd support is left as future work. ResNet50 and ResNet18 are close to TensorRT. Welder also trails Ansor on UNet (9.251 against 8.670) and ResNet18 (1.094 against 1.071) in the same table.
  • Popular transformers. On the V100 with FP32 SIMT cores at batch size 1, Welder is at 1.02× on BERT and 0.97× on Swin-T against TensorRT, which already has expert fusion rules for them. In Table 3, BERT at FP16 and batch size 64 takes 24.82 ms for Welder against 22.29 ms for FasterTransformer, and Swin-T at FP32 and batch size 1 takes 2.59 ms against 2.38 ms.
  • Compute-dominated MLPs. NeRF improves on Ansor by only 1.09× on SIMT cores because the compute-bound MLP dominates.
  • Other hardware. The MI50 speedup is slightly smaller over PyTorch and ONNXRuntime (2.62× and 1.71× against 4.29× and 2.07× at batch size 1), as the paper puts it, for the reason given above.
  • Longer sequences. The fusion gain shrinks for BERT and Conformer as input length grows (Figure 14).
  • Dynamic models. The design targets static models. The paper suggests turning a dynamic graph into static sub-graphs through JIT compilation, or keeping static tile plans while the number of parallel tasks follows the input shape (Section 6). Paper, Section 6

Two further caveats apply to the whole evaluation. The results are the authors' own measurements on the hardware and library versions listed above, and Welder's compute-heavy kernels are less tuned than vendor libraries. The PDF also carries USENIX artifact-evaluated badges, and its appendix states that the artifact reproduces the main V100 results. This page did not rerun it.

What to carry forward

The reusable idea is the order of operations. Pick the output tile of the last operator in a chain, propagate it back, count the global bytes, discard tiles whose footprint does not fit the layer, and only then profile the survivors. Each step before the profiling narrows the search without running a kernel.

Several related reviews look at the same cost from other angles. Liger Kernel: Fused SwiGLU counts the logical traffic of one pointwise fusion boundary with the same style of analytic byte count, and Welder's contribution is searching over many such boundaries. Making Deep Learning Go Brrrr explains how to tell whether a workload is memory-bound, which is the premise of Welder's motivation. Data Movement Is All You Need studies the same cost for transformers. FlashAttention resolves a Matmul and Softmax tile conflict by hand for attention by tiling through on-chip memory, whereas Welder derives a shared tile from the operators' expressions. Online Normalizer Calculation for Softmax removes the requirement that a Softmax tile hold a whole row by keeping a running normalizer. The FlashAttention and online softmax connections are readings across papers and not claims the papers make.

When a fusion result is quoted, check four things: which layer holds the intermediate, which tile shape was chosen and what footprint it needs, which baseline and precision the speedup uses, and whether it is an average or a maximum. The paper's own ratios run from below 0.5× (UNet against TensorRT in Table 6, a ratio calculated here from the table) to 21.4×, depending on those conditions, and its memory access patterns assumptions, coalesced 128-byte transactions in particular, determine how far the analytic traffic tracks the real cost.

If you found this paper review helpful, consider sharing it with others.

Mastodon