Explore machine learning papers and reviews related to Multimodal Learning. Find insights, analysis, and implementation details.
Chao Jia, Yinfei Yang, +8
How ALIGN scales vision-language pre-training to 1.8 billion noisy image-alt-text pairs, showing that the sheer scale of a raw, uncurated dataset can outweigh careful data cleaning for a simple dual-encoder contrastive model.
Junnan Li, Dongxu Li, +2
How BLIP unifies vision-language understanding and generation in one model, and bootstraps noisy web data with CapFilt — a captioner that synthesizes captions and a filter that removes noisy image-text pairs.
Jiahui Yu, Zirui Wang, +4
How CoCa trains a single image-text foundation model with both a contrastive loss and a generative captioning loss by splitting the text decoder into unimodal and multimodal halves, computed in one forward pass.
Jean-Baptiste Alayrac, Jeff Donahue, +25
How Flamingo bridges a frozen vision encoder and a frozen language model with a Perceiver Resampler and gated cross-attention layers, enabling few-shot learning from interleaved image-text sequences.
Bin Xiao, Haiping Wu, +7
How Florence-2 turns detection, segmentation, grounding, captioning, and OCR into one sequence-to-sequence problem driven by task prompts, expressing spatial outputs as location tokens in the text vocabulary.
Peng Wang, Shuai Bai, +17
How Qwen2-VL perceives images and video at any resolution with naive dynamic resolution (variable visual tokens) and M-RoPE, a multimodal rotary position embedding that decomposes position into temporal, height, and width components.
Xiaohua Zhai, Basil Mustafa, +2
How SigLIP replaces CLIP’s softmax contrastive loss with a pairwise sigmoid loss, removing the need for a global all-gather normalization and enabling strong language-image pre-training at small batch sizes.
Haotian Liu, Chunyuan Li, +2
LLaVA paper: align LLMs with visual information through instruction tuning on image-text pairs, enabling multimodal understanding and reasoning.
Junnan Li, Dongxu Li, +2
BLIP-2 leverages frozen image encoders and LLMs for efficient vision-language pre-training, achieving state-of-the-art multimodal performance.
Alec Radford, Jong Wook Kim, +10
CLIP explained: contrastive learning on 400M image-text pairs enables zero-shot image classification and powerful vision-language understanding.