Liger SwiGLU
Liger Kernel: Fused SwiGLU
Pin-Lun Hsu, Yun Dai, +8
Trace Liger Kernel’s fused SiLU and multiply, derive both SwiGLU gradients, and separate saved activation memory from measured GPU performance.
Explore machine learning papers and reviews related to Transformers. Find insights, analysis, and implementation details.
Liger SwiGLU
Pin-Lun Hsu, Yun Dai, +8
Trace Liger Kernel’s fused SiLU and multiply, derive both SwiGLU gradients, and separate saved activation memory from measured GPU performance.
Yansong Peng, Hebei Li, +4
Explore D-FINE’s edge distributions, iterative refinement, localization loss and self-distillation with interactive figures, Python and benchmark caveats.
Shilong Liu, Zhaoyang Zeng, +10
Trace Grounding DINO through text-image enhancement, language-guided anchors, cross-modal decoding and phrase grounding, with Python and evaluation caveats.
Wenyu Lv, Yian Zhao, +4
RT-DETRv2 explained through per-scale sampling, discrete operators, training schedules, executable Python mechanics and fair latency comparisons.
Xizhou Zhu, Weijie Su, +4
A visual explanation of sparse multi-scale attention, bilinear sampling, point and box references, and optional refinement in Deformable DETR.
Hao Zhang, Feng Li, +6
Explore DINO detector queries, denoising masks and look-forward-twice gradients through interactive diagrams, source equations and runnable Python.
Isaac Robinson, Peter Robicheaux, +3
Explore RF-DETR through recorded detections, architecture search, query budgets and masks, with runnable CPU calculations and source-backed paper results.
Mohammad Shoeybi, Mostofa Patwary, +4
Megatron-LM splits each transformer layer across GPUs with two all-reduces forward and two backward, and trains an 8.3B-parameter GPT-2 on 512 V100s.
Yunge Li, Lanyu Xu
Reordering image tokens along a Hilbert curve makes 2D local attention block-sparse, for about 4x faster window attention and 18x faster slide attention.
Leyla Naz Candogan, Arshia Afzal, +2
VIOLIN multiplies ViT attention by decay masks from eight space-filling curves: 1,296 extra parameters on DeiT-B and up to 8.7 points on VTAB-1K spatial tasks.
Kan Zhu, Yufei Gao, +14
How NanoFlow raises LLM serving throughput by overlapping compute, memory and network work inside a single GPU: it shows serving is compute-bound, splits each batch into nano-batches, and shares SMs between concurrent kernels, reaching 1.91x the throughput of TensorRT-LLM.
Tri Dao, Daniel Y. Fu, +3
How FlashAttention computes exact attention with linear memory by tiling Q, K, and V into SRAM-resident blocks and fusing the softmax, avoiding the quadratic HBM cost of materializing the full attention matrix.
Edward J. Hu, Yelong Shen, +6
How LoRA adapts a frozen large language model by learning a low-rank update ΔW = (α/r)·BA to each weight matrix, training under 1% of the parameters of full fine-tuning and adding zero inference latency once merged.
vLLM / PagedAttention
Woosuk Kwon, Zhuohan Li, +7
How PagedAttention (the memory manager behind vLLM) applies OS-style virtual-memory paging to the KV cache — fixed-size blocks, a block table, and copy-on-write prefix sharing — to reduce final-block waste, remove external block fragmentation, and improve throughput in the paper’s measured serving workloads.
William Fedus, Barret Zoph, +1
How the Switch Transformer scales to trillions of parameters by routing each token to a single expert (top-1 gating), decoupling model capacity from per-token compute while managing expert load with a capacity factor.
Jacob Devlin, Ming-Wei Chang, +2
How BERT pre-trains a deep bidirectional Transformer with masked language modeling and next-sentence prediction, then fine-tunes the same model to state-of-the-art results across a wide range of NLP tasks.
Yanghao Li, Hanzi Mao, +2
Investigating the effectiveness of plain Vision Transformers as backbones for object detection and proposing modifications to improve their performance.
DETR
Nicolas Carion, Francisco Massa, +4
Introducing DETR, a novel end-to-end object detection framework that leverages Transformers to directly predict a set of object bounding boxes.
Alexey Dosovitskiy, Lucas Beyer, +10
Vision Transformer explained with original paper equations, tensor shapes, runnable PyTorch code, and real pretrained image classification.
Krishna Teja Chitty-Venkata, Sparsh Mittal, +3
Survey of transformer inference optimization: pruning, quantization, knowledge distillation, neural architecture search, and hardware acceleration.
Ze Liu, Yutong Lin, +6
Swin Transformer: hierarchical Vision Transformer using shifted windows for efficient image classification, object detection, and segmentation.
Ashish Vaswani, Noam Shazeer, +6
Deep dive into the Transformer architecture that revolutionized NLP. Understand self-attention, multi-head attention, and positional encoding.
Andrei Ivanov, Nikoli Dryden, +3
Analysis of transformer performance bottlenecks caused by data movement. Learn optimization strategies for memory-bound operations on GPUs.