Skip to main content

Transformers

Explore machine learning papers and reviews related to Transformers. Find insights, analysis, and implementation details.

  • Tagged with
  • 23 papers
Back to all papers

Papers Related to Transformers

ECCV 2024

Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection

Shilong Liu, Zhaoyang Zeng, +10

Trace Grounding DINO through text-image enhancement, language-guided anchors, cross-modal decoding and phrase grounding, with Python and evaluation caveats.

Technical report 2024

RT-DETRv2: Improved Baseline with Bag-of-Freebies for Real-Time Detection Transformer

Wenyu Lv, Yian Zhao, +4

RT-DETRv2 explained through per-scale sampling, discrete operators, training schedules, executable Python mechanics and fair latency comparisons.

ICLR 2026

RF-DETR: Neural Architecture Search for Real-Time Detection Transformers

Isaac Robinson, Peter Robicheaux, +3

Explore RF-DETR through recorded detections, architecture search, query budgets and masks, with runnable CPU calculations and source-backed paper results.

arXiv 2019

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

Mohammad Shoeybi, Mostofa Patwary, +4

Megatron-LM splits each transformer layer across GPUs with two all-reduces forward and two backward, and trains an 8.3B-parameter GPT-2 on 512 V100s.

OSDI 2025

NanoFlow: Towards Optimal Large Language Model Serving Throughput

Kan Zhu, Yufei Gao, +14

How NanoFlow raises LLM serving throughput by overlapping compute, memory and network work inside a single GPU: it shows serving is compute-bound, splits each batch into nano-batches, and shares SMs between concurrent kernels, reaching 1.91x the throughput of TensorRT-LLM.

NeurIPS 2022

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

Tri Dao, Daniel Y. Fu, +3

How FlashAttention computes exact attention with linear memory by tiling Q, K, and V into SRAM-resident blocks and fusing the softmax, avoiding the quadratic HBM cost of materializing the full attention matrix.

SOSP 2023

vLLM / PagedAttention

Efficient Memory Management for Large Language Model Serving with PagedAttention

Woosuk Kwon, Zhuohan Li, +7

How PagedAttention (the memory manager behind vLLM) applies OS-style virtual-memory paging to the KV cache — fixed-size blocks, a block table, and copy-on-write prefix sharing — to reduce final-block waste, remove external block fragmentation, and improve throughput in the paper’s measured serving workloads.

JMLR 2022

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

William Fedus, Barret Zoph, +1

How the Switch Transformer scales to trillions of parameters by routing each token to a single expert (top-1 gating), decoupling model capacity from per-token compute while managing expert load with a capacity factor.

2023

A Survey of Techniques for Optimizing Transformer Inference

Krishna Teja Chitty-Venkata, Sparsh Mittal, +3

Survey of transformer inference optimization: pruning, quantization, knowledge distillation, neural architecture search, and hardware acceleration.

Mastodon