Skip to main content

Deep Learning

Explore machine learning papers and reviews related to Deep Learning. Find insights, analysis, and implementation details.

  • Tagged with
  • 47 papers
Back to all papers

Papers Related to Deep Learning

ICML 2015

BatchNorm

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Sergey Ioffe, Christian Szegedy

Trace mini-batch statistics, gradients shared across samples, learned scale and shift, and frozen inference; then inspect the original Inception evidence and the limits of its causal explanation.

OSDI 2020

HiveD: Sharing a GPU Cluster for Deep Learning with Guarantees

Hanyu Zhao, Zhenhua Han, +9

HiveD reserves groups of GPUs that sit together, not GPU counts, so a tenant never queues longer in a shared cluster than in a private one of the same size.

arXiv 2019

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

Mohammad Shoeybi, Mostofa Patwary, +4

Megatron-LM splits each transformer layer across GPUs with two all-reduces forward and two backward, and trains an 8.3B-parameter GPT-2 on 512 V100s.

OSDI 2025

NanoFlow: Towards Optimal Large Language Model Serving Throughput

Kan Zhu, Yufei Gao, +14

How NanoFlow raises LLM serving throughput by overlapping compute, memory and network work inside a single GPU: it shows serving is compute-bound, splits each batch into nano-batches, and shares SMs between concurrent kernels, reaching 1.91x the throughput of TensorRT-LLM.

NeurIPS 2022

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

Tri Dao, Daniel Y. Fu, +3

How FlashAttention computes exact attention with linear memory by tiling Q, K, and V into SRAM-resident blocks and fusing the softmax, avoiding the quadratic HBM cost of materializing the full attention matrix.

SOSP 2023

vLLM / PagedAttention

Efficient Memory Management for Large Language Model Serving with PagedAttention

Woosuk Kwon, Zhuohan Li, +7

How PagedAttention (the memory manager behind vLLM) applies OS-style virtual-memory paging to the KV cache — fixed-size blocks, a block table, and copy-on-write prefix sharing — to reduce final-block waste, remove external block fragmentation, and improve throughput in the paper’s measured serving workloads.

JMLR 2022

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

William Fedus, Barret Zoph, +1

How the Switch Transformer scales to trillions of parameters by routing each token to a single expert (top-1 gating), decoupling model capacity from per-token compute while managing expert load with a capacity factor.

ICML 2021

ALIGN: Scaling Vision-Language Learning with Noisy Text Supervision

Chao Jia, Yinfei Yang, +8

How ALIGN scales vision-language pre-training to 1.8 billion noisy image-alt-text pairs, showing that the sheer scale of a raw, uncurated dataset can outweigh careful data cleaning for a simple dual-encoder contrastive model.

TMLR 2022

CoCa: Contrastive Captioners are Image-Text Foundation Models

Jiahui Yu, Zirui Wang, +4

How CoCa trains a single image-text foundation model with both a contrastive loss and a generative captioning loss by splitting the text decoder into unimodal and multimodal halves, computed in one forward pass.

NeurIPS 2022

Flamingo: a Visual Language Model for Few-Shot Learning

Jean-Baptiste Alayrac, Jeff Donahue, +25

How Flamingo bridges a frozen vision encoder and a frozen language model with a Perceiver Resampler and gated cross-attention layers, enabling few-shot learning from interleaved image-text sequences.

arXiv 2024

Qwen2-VL: Vision-Language Perception at Any Resolution

Peng Wang, Shuai Bai, +17

How Qwen2-VL perceives images and video at any resolution with naive dynamic resolution (variable visual tokens) and M-RoPE, a multimodal rotary position embedding that decomposes position into temporal, height, and width components.

ICCV 2023

SigLIP: Sigmoid Loss for Language Image Pre-Training

Xiaohua Zhai, Basil Mustafa, +2

How SigLIP replaces CLIP’s softmax contrastive loss with a pairwise sigmoid loss, removing the global softmax denominator and enabling chunked all-pairs computation and strong language-image pre-training at smaller batch sizes.

Mastodon