Explore machine learning papers and reviews related to deeplearning. Find insights, analysis, and implementation details.
How FlashAttention computes exact attention with linear memory by tiling Q, K, and V into SRAM-resident blocks and fusing the softmax, avoiding the quadratic HBM cost of materializing the full attention matrix.
How LoRA adapts a frozen large language model by learning a low-rank update ΔW = (α/r)·BA to each weight matrix, training under 1% of the parameters of full fine-tuning and adding zero inference latency once merged.
How PagedAttention (the memory manager behind vLLM) applies OS-style virtual-memory paging to the KV cache — fixed-size blocks, a block table, and copy-on-write prefix sharing — to eliminate fragmentation and dramatically raise LLM serving throughput.
How the Switch Transformer scales to trillions of parameters by routing each token to a single expert (top-1 gating), decoupling model capacity from per-token compute while managing expert load with a capacity factor.
How AlexNet won ImageNet 2012 by a landslide and launched the deep-learning era: a deep convolutional network trained on two GPUs with ReLU activations, dropout regularization, and aggressive data augmentation.
How ALIGN scales vision-language pre-training to 1.8 billion noisy image-alt-text pairs, showing that the sheer scale of a raw, uncurated dataset can outweigh careful data cleaning for a simple dual-encoder contrastive model.
How BERT pre-trains a deep bidirectional Transformer with masked language modeling and next-sentence prediction, then fine-tunes the same model to state-of-the-art results across a wide range of NLP tasks.
How BLIP unifies vision-language understanding and generation in one model, and bootstraps noisy web data with CapFilt — a captioner that synthesizes captions and a filter that removes noisy image-text pairs.
How CoCa trains a single image-text foundation model with both a contrastive loss and a generative captioning loss by splitting the text decoder into unimodal and multimodal halves, computed in one forward pass.
How Flamingo bridges a frozen vision encoder and a frozen language model with a Perceiver Resampler and gated cross-attention layers, enabling few-shot learning from interleaved image-text sequences.
How Florence-2 turns detection, segmentation, grounding, captioning, and OCR into one sequence-to-sequence problem driven by task prompts, expressing spatial outputs as location tokens in the text vocabulary.
How GANs frame generative modeling as a two-player minimax game: a generator turns noise into samples while a discriminator learns to tell real from fake, driving the generator toward the true data distribution.
How Qwen2-VL perceives images and video at any resolution with naive dynamic resolution (variable visual tokens) and M-RoPE, a multimodal rotary position embedding that decomposes position into temporal, height, and width components.
How SAM 2 extends promptable segmentation from images to video with a streaming memory bank and memory attention, propagating masks across frames and recovering objects after occlusion.
How SigLIP replaces CLIP’s softmax contrastive loss with a pairwise sigmoid loss, removing the need for a global all-gather normalization and enabling strong language-image pre-training at small batch sizes.
How the variational autoencoder turns an autoencoder into a generative model: an encoder predicts a Gaussian latent distribution, the reparameterization trick makes sampling differentiable, and the ELBO balances reconstruction against a KL prior.