Skip to main content

MPK: A Compiler and Runtime for Mega-Kernelizing Tensor Programs

Follow how MPK turns a model into one persistent GPU kernel by tracking dependencies between tasks on individual SMs, with a task-graph explorer, a cross-task pipelining timeline, and a runnable CPU model of its compiler stages.

TL;DR

  • MPK compiles a model's tensor program into one persistent GPU kernel. Its unit of scheduling is a task on a single SM, and dependencies are events between tasks rather than barriers between kernel launches.
  • The compiler decomposes operators into tasks, derives events from overlapping data regions, fuses redundant events, and orders tasks so each event stores only a first and a last task index. An in-kernel runtime of workers and schedulers then executes the graph event by event.
  • The paper reports 1.0 to 1.7x better single-GPU serving performance than the better of SGLang and vLLM on five models and three GPU generations, and, for Qwen3-1.7B with tensor parallelism on up to eight H100 GPUs, 1.1 to 1.4x by the text (the Figure 11 labels show 1.0 to 1.3x). Part of the gain comes from moving page allocation and request scheduling into the kernel, not from fusion alone.
  • The evidence covers LLM decoding on NVIDIA GPUs with batch sizes up to 16, and gains shrink toward 1.0x for larger models and batches. The Python on this page is a toy model of the compiler stages, not MPK.

Why one kernel per operator leaves performance unused

A tensor program is a graph of operators, and most systems run each operator as its own GPU kernel. The paper names three costs of that choice (Sections 1 and 2.1). First, consecutive kernels on one stream are separated by an implicit barrier: every thread of the earlier kernel must finish before any thread of the next begins. That prevents software pipelining across operators, so pipeline bubbles between tasks leave hardware resources underutilized. Programmatic dependent launch allows partial overlap between kernels on a stream but, in the paper's words, requires significant engineering effort. Second, dependencies are recorded only per operator. A matrix multiplication followed by an AllGather or AllReduce makes every communication thread block wait for the whole multiplication, although each one needs only a slice of its output. Third, one inference iteration can launch hundreds to thousands of kernels. CUDA Graphs reduce launch cost but are largely static: changes to control flow, tensor shapes or data dependencies require re-instantiating or modifying the captured graph.

Fusing kernels removes some of these barriers, and compilers already fuse small groups of adjacent operators. The mega-kernel idea (Section 2.2) goes further: launch one persistent kernel that runs all computation and communication of the model, and coordinate its SMs with synchronization in device memory. Handwritten mega-kernels exist, among them FlashDMoE and a low-latency Llama-1B kernel, but such kernels are handcrafted by GPU experts for specific models. MPK's claim is that a compiler can produce such a kernel from a PyTorch program, and the rest of the paper explains the two components that make that possible: a graph representation and an in-kernel runtime.

A graph with one node per SM task

MPK's SM-level graph, the tGraph (Section 3), has two node kinds that alternate. A task is a unit of computation or communication executed on one SM. An event is a synchronization point: it activates once all tasks that trigger it have finished, and each task that depends on it may start once all of its dependent events have fired. A task therefore has incoming edges from the events it depends on and outgoing edges to the events it triggers. Figure 4 includes a MatMul followed by an AllReduce. Because AllReduce performs element-wise communication and reduction, each of its tasks depends on only one MatMul task, the one that produces its input tile, so a tGraph can put one event between each such pair. Figure 4(c) shows an equivalent but suboptimal tGraph whose events capture only operator-level dependencies, which is what a kernel barrier provides. The paper also notes that tGraphs resemble CUDA Graphs but record dependencies between SM tasks and sub-kernel events, not between whole kernels.

The figure below is an authored illustration of that difference. Mark MatMul tiles as finished and see which AllReduce tiles each kind of event releases, then compare how the same eight tasks dispatch onto four workers.

One event per operator, or one per task pair

Mark MatMul tiles as finished and see which AllReduce tiles an event releases. Switch between the barrier form of Figure 4(c) and the task-level form of Figure 4(b), then compare the dispatch they produce.

MatMul tasks

One event

AllReduce tasks

e 0/4waiting
AR1waits
AR2waits
AR3waits
AR4waits

0 of 4 AllReduce tiles may start with no MatMul tile finished. The single event waits for all four notifications.

6

Dispatch of the same eight tasks on four workers (operator-level events)

w1
w2
w3
w4
0246810

Makespan, this granularity

10 units

Makespan, task-level events

7 units

AllReduce tiles started before the last MatMul tile ends

0 of 4

Overlap pays when upstream tiles finish at different times.

Authored illustration with four MatMul and four AllReduce tiles on four workers, following the structure of Figures 3 and 4. Events, counts and the dispatch schedule come from the browser port of the reference model. Durations are illustrative units, not measured times, and the slow tile is an assumption chosen to show the effect.

In the calculated dispatch with the slow tile at 6 units, operator-level events start no AllReduce tile until all four MatMul tiles are done, while task-level events release three of them while the slow tile is still running. Set the slow tile to 2 units and the two forms match, which is the point of the comparison: finer events help when upstream tasks finish at different times. Many tGraphs describe the same computation, so the next question is how MPK builds a precise one without hand work.

The compiler builds the graph in four steps

Section 4 takes a computation graph and an inference configuration and produces a tGraph specialized to the GPU. Figure 5 follows a small example through every step: query, key and value projections feed attention, whose output feeds both an output projection and an RMSNorm.

  1. Operator decomposition. Each operator's output is partitioned into disjoint pieces, one task each, so tasks of one operator run in parallel across SMs. MPK picks the partitioning that minimizes loads from device memory to shared memory and by default creates a number of tasks proportional to the number of SMs.
  2. Dependency analysis. For two operators that share a tensor, MPK enumerates task pairs and creates an event for a pair only if the producer's output region overlaps the consumer's input region. Figure 5(b) shows 18 such events for the example, plus a start event.
  3. Event fusion. Successor-set fusion merges events with the same set of consumer tasks, and predecessor-set fusion merges events with the same set of producer tasks (Definitions 4.1 and 4.2). In the example the 18 events become 8 and then 5, plus the start event, as Figure 5(c) and 5(d) show.
  4. Normalization and linearization. Normalization inserts empty tasks (and, in the general rewrite, new events; Figure 5(e) reuses existing ones, as the README notes) at forks and joins so every task depends on one event and triggers at most one. Linearization then orders tasks breadth first (Algorithm 1) so all tasks launched by one event are consecutive, and an event stores a first and a last index instead of a list. Figure 5(f) shows the resulting table.

The last two steps exist to bound metadata. Without normalization a task would need space for the maximum number of events it could wait on or trigger, and without linearization an event would need space for the maximum fan-out. Section 4.1 also says normalization costs less than 1% in the evaluation because real models are deep rather than wide. Section 6.7 adds that the three compiled LLM tGraphs contained no fork or join groups at all, because the query, key and value projections are emitted as fused operators. Normalization stays necessary for correctness whenever parallel branches do occur.

Separately (Section 4.2), each compute task is given a CUDA implementation found by the Mirage superoptimizer at the thread-block level, starting from a reference PyTorch implementation. The paper says the generated code includes intra-SM software pipelining, register reuse and layout choices that reduce shared-memory bank conflicts.

The in-kernel runtime executes events

With the graph built, Section 5 describes what runs once the single kernel is launched. The runtime splits the GPU's SMs into workers and schedulers. Each worker is one physical SM with its own task queue, and repeatedly dequeues a task, runs it and notifies the event the task triggers. Schedulers are organized by warp, with up to four on an SM, each polling an event queue and launching the tasks of every newly activated event. The assignment is fixed at kernel launch, so there is no role switching inside the kernel. In the paper's configuration (Table 1) four SMs are reserved for schedulers, with 16 scheduler warps in total: A100 has 108 SMs and 104 workers, H100 has 132 and 128, and B200 has 148 and 144.

Execution starts from a designated start event with no prerequisites. Because communication and computation are tasks in one graph, a compute task launches as soon as the communication tasks producing its inputs trigger their event, whatever links those tasks used. The paper presents this as adapting to the interconnect without an explicit topology profile.

Launching tasks. A scheduler can enqueue a task just in time (JIT), after its event has fired, or the task can be enqueued ahead of time (AOT) on a worker that then waits for the event. Figure 8 compares them: JIT takes two synchronization steps (worker to scheduler, scheduler to worker), AOT takes one. JIT adapts to imbalance, which matters for attention because its duration depends on sequence length. MPK's hybrid strategy marks operators with data-dependent durations, such as attention, as JIT, keeps their downstream operators JIT until a global barrier removes the accumulated imbalance, and labels everything else AOT. Workers keep one queue of each kind and always serve JIT tasks first. AOT tasks are pre-enqueued round-robin before execution, so schedulers handle only events that launch JIT tasks.

Pipelining across tasks. Section 5.3 adds three mechanisms. A paged shared-memory abstraction splits shared memory into fixed-size pages, 32 KB in the evaluation, that a task acquires and releases, and a task that has released a page may not acquire another. Cross-task software pipelining splits each task into a pre-loading phase and a compute phase, and overlaps the next task's pre-loading with the current compute phase when two conditions hold: the current task has issued all of its own data-transfer instructions, and enough pages are free. Finally, each task's 352-byte description is prefetched into shared memory before it is needed, which hides device-memory latency on enqueue and dequeue.

The timeline below isolates the second mechanism. It is an authored illustration with invented units, and its page counter shows why the second condition matters.

Preload the next task while the current one computes

Play four identical tasks on one SM. A kernel boundary makes each task load, compute and finish before the next starts loading. With cross-task pipelining, the next task may load while the current one computes, if shared-memory pages are free.

7
Step 1 / 9
Load
Compute
02468101214161820

t = 0 units: T1 starts loading

Loading: T1. Computing: none. Pages held: 3 of 7.

Finish, kernel boundary

20 units

Finish, cross-task pipelining

14 units

Pages needed to preload T2 during T1

6 of 7

Enough pages for two tasks at once, so the loads overlap compute.

Authored illustration in illustrative time units: each task loads for 2, computes for 3 and holds 3 pages until it finishes. The paper does not give these values. Section 6.2 gives 5, 7 and 7 pages per SM on A100, H100 and B200; this figure makes no claim about which of those allows overlap for real tasks. It is not a measurement.

With the default of 7 pages, T2 can load while T1 computes and the run ends earlier than the kernel-boundary schedule. Slide the page count below 6 and the two schedules become identical, because T2 has no free pages to load into. How much real tasks overlap depends on their actual page footprints, which the paper does not list.

Read the compiler and runtime stages beside the code

With the compiler and runtime described, the rows below connect original excerpts from the paper to the definitions they implement and to a runnable CPU model. The model follows the Figure 5 example, decomposes tasks over index ranges, derives events from region overlap, fuses and normalizes them, linearizes the result and replays event-driven dispatch onto FIFO worker queues. It is an independent teaching model: it generates no CUDA, runs nothing on a GPU, and its counts describe toy graphs, not the paper's models.

1. Turn region overlap into task-level events

Original paper

Figure 5(a) and 5(b): operator decomposition and dependency analysis

Section 4.1 introduces an event for a task pair only when the producer’s output region overlaps the consumer’s input region. Figure 5(b) shows the result for the Q/K/V example.

Math and tensor shapes

\begin{aligned}e_{(t_1,t_2)}\ \text{exists}&\iff \operatorname{out}(t_1)\cap\operatorname{in}(t_2)\neq\varnothing\\ \text{edges}&:\ (t_1,e),\ (e,t_2)\end{aligned}

The reference gives each task a half-open range of its operator’s output and the input ranges it reads. Tasks with no producer wait on a start event. The Figure 5 example yields 18 pair events, the same count as e1 through e18 in the figure. Operator granularity expands every pair to all tasks of both operators, which is the kernel-barrier form of Figure 4(c).

Python implementation

tgraph.py · dependencies
python
def decompose(operators):
    """Split operator outputs into disjoint tasks: (name, op, out, reads)."""
    tasks = []
    for op in operators:
        if op.extent % op.tiles:
            raise ValueError(f"{op.name}: extent must divide into equal tiles")
        width = op.extent // op.tiles
        for tile in range(op.tiles):
            lo, hi = tile * width, (tile + 1) * width
            tasks.append((f"{op.name}{tile + 1}", op.name, (op.name, lo, hi),
                          op.reads(lo, hi)))
    return tasks


def analyze_dependencies(tasks, granularity="task"):
    """One event per overlapping (producer, consumer) pair, after a start."""
    pairs = []
    for name, _, _, reads in tasks:
        for tensor, lo, hi in reads:
            for other, _, (written, w_lo, w_hi), _ in tasks:
                if written == tensor and w_lo < hi and lo < w_hi:
                    pairs.append((other, name))
    if granularity == "operator":  # A kernel barrier: all tasks to all tasks.
        by_op = collections.defaultdict(list)
        for name, op, _, _ in tasks:
            by_op[op].append(name)
        op_of = {name: op for name, op, _, _ in tasks}
        pairs = [(p, c) for p0, c0 in pairs
                 for p in by_op[op_of[p0]] for c in by_op[op_of[c0]]]
    pairs = list(dict.fromkeys(pairs))
    consumers = {c for _, c in pairs}
    start = Event(set(), {n for n, _, _, _ in tasks if n not in consumers})
    return [start] + [Event({p}, {c}) for p, c in pairs]

2. Fuse events that share a consumer set or a producer set

Original paper

Section 4.1: Definitions 4.1 and 4.2, event fusion

Definitions 4.1 and 4.2 give successor-set and predecessor-set fusion. Figure 5(c) and 5(d) apply them in that order.

Math and tensor shapes

\begin{aligned}\operatorname{Out}(e_1)=\operatorname{Out}(e_2)&\Rightarrow \operatorname{In}(e^\prime)=\operatorname{In}(e_1)\cup\operatorname{In}(e_2)\\ \operatorname{In}(e_1)=\operatorname{In}(e_2)&\Rightarrow \operatorname{Out}(e^\prime)=\operatorname{Out}(e_1)\cup\operatorname{Out}(e_2)\end{aligned}

Each rewrite leaves the set of producer-consumer pairs unchanged, and the reference checks that. Running successor-set fusion first reduces the 18 pair events to 8, and predecessor-set fusion then gives 5, plus the start event. Applying the two in the other order is also valid but yields different events, so this model fixes the order Figure 5 uses. After the second pass a further round changes nothing for this example.

Python implementation

tgraph.py · fusion
python
def _fuse(events, key, merge):
    """Group events by `key`, keeping the start event (empty ins) untouched."""
    fused = {}
    for event in events:
        slot = key(event) if event.ins else id(event)
        if slot in fused:
            merge(fused[slot], event)
        else:
            fused[slot] = Event(set(event.ins), set(event.outs))
    return list(fused.values())


def fuse_successor_sets(events):
    """Definition 4.1: events with equal OutTasks merge; InTasks are unioned."""
    return _fuse(events, lambda e: frozenset(e.outs),
                 lambda keep, e: keep.ins.update(e.ins))


def fuse_predecessor_sets(events):
    """Definition 4.2: events with equal InTasks merge; OutTasks are unioned."""
    return _fuse(events, lambda e: frozenset(e.ins),
                 lambda keep, e: keep.outs.update(e.outs))

3. Order tasks so each event stores a first and a last index

Original paper

Section 4.1: Algorithm 1, tGraph linearization

Algorithm 1 runs after normalization. Its output order makes the tasks launched by one event consecutive, as in the First task ID and Last task ID rows of Figure 5(f).

Math and tensor shapes

\begin{aligned}\operatorname{Out}(e)&=\{T[i]: \mathrm{first}(e)\le i\le \mathrm{last}(e)\}\\ B_{\text{list}}&=w\,(S+E)\\ B_{\text{range}}&=2wE\end{aligned}

Here w is the index width, S the number of event-to-task links and E the number of events. The paper does not state w. The reference assumes 4 bytes and one length word per list, which is this page’s assumption: it gives 96 bytes against 48 for the six events of the Figure 5 example. Four of those six events already have adjacent successors in creation order; linearization fixes the other two, e3 and e4, which each launch a real task and an empty task. The first and last index of every event matches Figure 5(f). The printed Counts row there lists 2 for e4 and 1 for e5, while its own task rows give 1 and 2, and the reference reports the latter.

Python implementation

tgraph.py · linearize
python
def linearize(names, events):
    """Algorithm 1: breadth-first order; each event gets a contiguous range."""
    dependent = {t: i for i, e in enumerate(events) for t in e.outs}
    trigger = {t: i for i, e in enumerate(events) for t in e.ins}
    queue = collections.deque(i for i, e in enumerate(events) if not e.ins)
    order, placed, dequeued = [], set(), []
    while queue:
        current = queue.popleft()
        dequeued.append(current)
        for task in names:
            if dependent[task] == current:
                order.append(task)
                placed.add(task)
                after = trigger.get(task)
                if after is not None and events[after].ins <= placed:
                    queue.append(after)
    if len(order) != len(names):
        raise ValueError("Tasks are unreachable from the start event")
    position = {task: index + 1 for index, task in enumerate(order)}
    table = []
    for number, index in enumerate(dequeued):
        spots = [position[t] for t in events[index].outs]
        first, last = min(spots), max(spots)
        if last - first + 1 != len(spots):
            raise ValueError("Successors are not contiguous")
        table.append({"id": number, "counts": len(events[index].ins),
                      "first": first, "last": last})
    number = {index: n for n, index in enumerate(dequeued)}
    links = {t: [number[dependent[t]], number.get(trigger.get(t))]
             for t in order}
    return order, table, links

4. Fire events and dispatch their task ranges to FIFO queues

Original paper

Figure 7: the event-driven execution model

Section 5.1: the start event is enqueued first, a scheduler launches the tasks that depend on an activated event, and each worker notifies the event its finished task triggers.

Math and tensor shapes

\begin{aligned}e\ \text{fires}&\iff \#\{t\in\operatorname{In}(e):t\ \text{done}\}=\operatorname{counts}(e)\\ \text{on fire}&:\ \text{enqueue}\ T[\mathrm{first}(e)\ldots\mathrm{last}(e)]\end{aligned}

The reference models just-in-time dispatch with instant schedulers. A launched task goes to the worker with the fewest queued or running tasks, a choice this model makes because the sections used do not specify it. Ahead-of-time launch, hybrid classification, page management and communication are not modeled. Durations are illustrative units. With the stated durations, task-level events start three AllReduce tiles before the last MatMul tile finishes; operator-level events start none.

Python implementation

tgraph.py · schedule
python
def simulate(plan, durations, workers):
    """Replay event-driven dispatch: schedulers enqueue, workers run FIFO."""
    table, order, links = plan["events"], plan["order"], plan["links"]
    remaining = {row["id"]: row["counts"] for row in table}
    queues = [collections.deque() for _ in range(workers)]
    running, finishing, schedule = [None] * workers, [], []

    def start_idle(now):
        for worker in range(workers):
            if running[worker] is None and queues[worker]:
                task = queues[worker].popleft()
                end = now + durations.get(task, 0)
                running[worker] = task
                schedule.append({"task": task, "worker": worker,
                                 "start": now, "end": end})
                heapq.heappush(finishing, (end, worker, task))

    def activate(event, now):
        for index in range(table[event]["first"], table[event]["last"] + 1):
            load = [len(q) + (running[k] is not None)
                    for k, q in enumerate(queues)]
            queues[load.index(min(load))].append(order[index - 1])
        start_idle(now)

    activate(0, 0)  # The start event has no prerequisites.
    while finishing:
        now, worker, task = heapq.heappop(finishing)
        running[worker] = None
        after = links[task][1]
        if after is not None:
            remaining[after] -= 1
            if remaining[after] == 0:
                activate(after, now)
        start_idle(now)
    return schedule

Original excerpts are reproduced from the open-access USENIX OSDI ’26 proceedings and attributed to their authors and venue: Xinhao Cheng, Zhihao Zhang, Yu Zhou, Jianan Ji, Jinchen Jiang, Zepeng Zhao, Ziruo Xiao, Zihao Ye, Yingyi Huang, Ruihang Lai, Hongyi Jin, Bohan Hou, Mengdi Wu, Yixin Dong, Anthony Yip, Zihao Ye, Songting Wang, Wenqin Yang, Xupeng Miao, Tianqi Chen, Zhihao Jia. The paper lists two different authors named Zihao Ye, affiliated with NVIDIA and the University of Michigan. The landing page states open access and shows no Creative Commons license, so none is claimed. PDF and crop hashes, page rectangles, and rendering scale are recorded in the source manifest. The Python is an independently authored teaching model, not MPK’s code.

Run tgraph.py with Python 3.10 or later. The README lists the choices the model makes where the paper is silent or inconsistent, and requirements.txt confirms that no external packages are needed. Add --json for the deterministic values used by the browser-model checks. The self-check asserts the Figure 5 event counts at each stage, each event's first and last task index, and the trigger counts implied by Figure 5(f)'s task rows; inside the start event's range the task order differs from Figure 5(f), as the README notes. It also asserts that fusion preserves the set of producer-consumer pairs, that normalization leaves every task with one dependent event and at most one triggering event, and that every simulated task starts after its producers finish. The paper's code is public in the Mirage repository. The link is the default-branch head at review time, not a paper-tagged release, and this page does not compare the reference line by line with it.

Dynamic workloads and implementation

The graph is built ahead of time, but serving is dynamic (Section 6.1). MPK is a PyTorch compiler backend: torch.compile(backend=MPK) generates the mega-kernel and returns a callable that launches it once. The implementation is roughly 44K lines of C++, 42K of CUDA and 10K of Python, uses semaphores in device memory to coordinate workers and schedulers, and uses NVSHMEM for in-kernel inter-GPU communication. Task queues are circular buffers in device memory with atomic-add enqueue and dequeue, and each scheduler assigns tasks from local state, so the current runtime has no global coordination. The paper leaves globally coordinated policies to future work.

For serving, the scheduler prepares each decoding iteration when it processes a tGraph's start event: in a single task it removes finished requests, admits new ones and updates per-request KV-cache metadata, which stays in device memory for attention tasks to read. To cope with changing batch sizes, MPK compiles one tGraph for each power-of-two batch size up to the maximum, and the scheduler picks the matching one at runtime. For mixture-of-experts models it adds a hybrid balancer, where compile-time expert tasks refine their work at runtime from a meta-tensor produced by top-k softmax, and a fused gather-GEMM that folds the token gather into the GEMM's data-loading phase. This in-kernel serving logic matters for the evidence that follows, because the baselines run page allocation and request scheduling on the CPU and MPK does not.

What the paper's measurements establish

Three evidence classes appear on this page. Paper-reported values come from the paper's text, tables and figure labels. Calculated here values are arithmetic on those numbers or on the toy model. Authored illustration marks the interactive figures and the Python model.

Setup (paper-reported). The five models are Qwen3-0.6B, Llama-3.2-1B-Instruct, Qwen3-1.7B, Qwen3-8B and Qwen3-30B-A3B, run on A100, H100 and B200 at maximum batch sizes from 1 to 16 in bfloat16. The workload is offline batched inference with a 64-token prompt and 1,024 generated tokens, and every system uses paged attention and continuous batching. The baselines use FlashInfer and FlashAttention, cuBLAS and CUTLASS, and CUDA or Triton kernels. Paper, Section 6.2 and Table 1

Single-GPU serving (paper-reported). The paper states the 1.0 to 1.7x range "for single-batch inference" across models and hardware, with larger gains for smaller models and newer GPU generations, and Figure 9 reports throughput. The paper does not define single-batch. The Figure 9 labels at batch size 1 run from 1.1x to 1.6x, while 1.0 to 1.7x is the span across batch sizes 1 to 16 (calculated here from the labels). Each value in Figure 9 is the MPK speedup over the best-performing baseline for that cell, so the baseline can be SGLang or vLLM. The printed labels, read cell by cell, are below, with batch sizes 1, 2, 4, 8 and 16 in each cell. The largest label is 1.7x for Qwen3-0.6B on H100 at batch size 2. The labels do not rank strictly by GPU generation: the largest B200 label is 1.5x (calculated here from the labels). Paper, Figure 9

ModelB200H100A100
Qwen3-0.6B1.4, 1.4, 1.4, 1.4, 1.31.6, 1.7, 1.6, 1.5, 1.51.5, 1.5, 1.4, 1.4, 1.2
Llama-3.2-1B-Instruct1.5, 1.5, 1.5, 1.4, 1.21.5, 1.4, 1.4, 1.3, 1.31.3, 1.3, 1.3, 1.3, 1.2
Qwen3-1.7B1.3, 1.3, 1.3, 1.3, 1.21.4, 1.4, 1.4, 1.3, 1.21.3, 1.3, 1.2, 1.3, 1.1
Qwen3-8B1.1, 1.1, 1.2, 1.2, 1.11.1, 1.1, 1.2, 1.1, 1.11.2, 1.5, 1.4, 1.1, 1.0
Qwen3-30B-A3B1.1, 1.1, 1.1, 1.2, 1.01.1, 1.1, 1.1, 1.1, 1.11.1, 1.1, 1.1, 1.1, 1.0

The paper gives one latency example, under single-batch decoding without stating the batch size: Qwen3-8B on an A100 drops from 14.5 ms per token on vLLM and SGLang to 12.5 ms on MPK, about 1.16x (calculated here). It compares that with a hardware lower bound of about 10 ms, which is the paper's estimate from loading 16 GB of parameters at 1.6 TB/s. The paper gives three reasons the gain grows as models shrink and GPUs speed up: kernel-switch overhead even with CUDA Graphs, pipeline bubbles at kernel boundaries, and CPU-GPU synchronization from page allocation and request scheduling on the host. The last reason matters for interpretation. MPK moves that work into the kernel, so the comparison measures the whole design, and fusion alone would explain only part of it. Paper, Section 6.3

Other baselines (paper-reported). The paper also says MPK gives more than 10x over native PyTorch, without naming a model, GPU or batch size for that figure. For multiple GPUs it reports up to 10x over PyTorch using hand-optimized kernels, CUDA Graphs and torch.compile. These are different baselines from SGLang and vLLM and should not be combined with the 1.0 to 1.7x range. Paper, Section 6.3 and Section 6.5

Multi-GPU and ablations (paper-reported). On one H100 DGX instance with tensor parallelism, the paper's text reports 1.1 to 1.4x over SGLang and vLLM at up to eight GPUs for Qwen3-1.7B, the only model shown. The printed bar labels in Figure 11 run from 1.0x to 1.3x, so this page quotes both. Enabling fine-grained compute-communication overlap reduces per-iteration latency by 1.1x for Qwen3-1.7B on four H100s. The baseline for that comparison restricts the tGraph to one coarse event per collective, the barrier form that Figure 4(c) draws (the paper's text points to Figure 5c). Cross-task pipelining is reported as 1.2 to 1.3x on the final linear layer of Qwen3-8B on B200, with bar labels from 1.14x to 1.29x. That is one layer, not an end-to-end result. Paper, Figures 11 to 13

Mixture of experts (paper-reported). For Qwen3-30B-A3B on B200, the MoE runtime with MPK's hybrid balancer is 1.07 to 1.18x faster than SGLang's MoE implementation across batch sizes 1 to 16, and the paper says it consistently beats static partitioning. Gather preprocessing is up to 11% of MoE time in SGLang at batch size 1. Paper, Section 6.4 and Figure 10

Launch overhead (paper-reported, checked here). Qwen3-8B issues 293 kernel launches per token under kernel-per-operator execution. On B200 each launch takes 3.8 µs eagerly and 0.8 µs with CUDA Graphs, which the paper totals as 1.1 ms and 0.2 ms per token; 293 launches at those costs give about 1.11 and 0.23 ms (calculated here). The in-kernel scheduler takes 0.28% of total runtime. Paper, Section 6.6

Compiler statistics (paper-reported, B200). Table 2 reports per-stage results on three models:

ModelOperatorsTasks per operatorEventsFusionLinearization
Qwen3-1.7B22935.61,87037x4.4x
Qwen3-8B29347.32,36668x5.9x
Qwen3-30B-A3B53332.21,142118x15.0x

For Qwen3-8B, 293 operators decompose into 13,867 tasks. The final tGraphs hold 1,142 to 2,366 events that encode 69,000 to 162,000 producer-consumer pairs. Linearization shrinks the successor encoding from 110,932 to 18,928 bytes for Qwen3-8B, and the paper attributes the larger MoE gain to high-fan-out expert-routing events. The second figure equals 8 bytes times 2,366 events, which is consistent with two 4-byte indices per event, though the paper does not state the width (calculated here). Paper, Section 6.7 and Table 2

Limits and open questions

The evaluation is LLM decoding on NVIDIA GPUs. The workload is a 64-token prompt followed by 1,024 generated tokens at batch sizes up to 16, so the results concern decode-heavy runs and say little about long prompts. The compiler and runtime are described as model-agnostic, but only five LLMs are evaluated. The gains also taper: the Figure 9 labels fall to 1.0x for Qwen3-8B on A100 at batch size 16 and for Qwen3-30B-A3B at batch size 16 on A100 and B200, and the paper attributes the larger gains to smaller models and faster GPUs, where its three overheads weigh more against per-token computation. The multi-GPU results come from a single H100 DGX instance and one model, so they do not show how the design behaves across nodes.

Section 7 discusses resource footprint and portability. A common concern, in the paper's framing, is that a mega-kernel must reserve resources for the union of what its tasks need; the paper says MPK avoids this by time-multiplexing tasks and paging shared memory, with one exception: per-thread register usage is fixed for the whole kernel at the maximum any task type requires. The paper presents two flexibilities: a hand-tuned kernel can be wrapped as a task's device function, and porting to a new GPU architecture means updating the per-task code generators, which it calls the only hardware-specific component. Section 6.1 notes that scheduling is decentralized and leaves globally coordinated policies for future work. For related work the paper notes that TileRT also decomposes operators into tile-level tasks run as one persistent kernel, and treats that as support for persistent execution.

One more caveat: the reference model on this page checks the logic of the compiler stages on toy graphs; it does not test that MPK's generated tasks, scheduling latency or shared-memory pages behave as described.

What to carry forward

The central move is to change what a dependency names. A kernel barrier says that a whole operator finished. An event says that specific tasks finished, so the scheduler can start any consumer whose inputs exist. The rest of the system follows from making that cheap: fusion and linearization shrink the event metadata, and workers and schedulers keep the cost of acting on an event low.

Several reviews on this site cover the pieces MPK builds on or competes with. NanoFlow attacks the same idle-resource problem inside the kernel-per-operator model, splitting batches and sharing SMs between concurrent kernels, while MPK moves the overlap decision into one task graph. PagedAttention is the KV-cache memory manager that every system in this evaluation uses and that MPK drives from inside the kernel. Megatron-LM is the tensor-parallel layout whose AllReduce points MPK compiles into transfer and reduction tasks. NCCLX and Every µs Matters examine collective communication, which MPK replaces with NVSHMEM-based tasks, and are the place to look for how communication latency is attacked outside a persistent kernel. FlashAttention is the fused attention kernel that kernel-per-operator baselines rely on, and Making Deep Learning Go Brrrr gives the broader framework for deciding whether launch overhead, memory traffic or compute is the bottleneck you actually have.

If you found this paper review helpful, consider sharing it with others.

Mastodon