Sliding Window Attention
Sliding Window Attention for long sequences: local context windows enable O(n) complexity, used in Mistral and Longformer models.
No direct links0 refs
ML Concepts - Page 16
Clear explanations of core machine learning concepts, from foundational ideas to advanced techniques. Understand attention mechanisms, transformers, skip connections, and more.
174
Total Concepts
329
Topics
Page 16
of 29
Concepts - Page 16
Sparse Attention Patterns
Explore sparse attention mechanisms that reduce quadratic complexity to linear or sub-quadratic, enabling efficient processing of long sequences.
No direct links0 refs
SoA vs AoS: Data Layout Optimization
Master Structure of Arrays (SoA) vs Array of Structures (AoS) data layouts for optimal cache efficiency, SIMD vectorization, and GPU memory coalescing.
No direct links0 refs
He/Kaiming Initialization
Learn He (Kaiming) initialization for ReLU neural networks. This weight initialization technique prevents vanishing gradients in deep learning models.
No direct links0 refs
Xavier/Glorot Initialization
Understand Xavier (Glorot) initialization, the weight initialization technique that maintains signal variance across layers for stable deep network training.
No direct links0 refs
Contrastive Loss Functions
Master contrastive loss functions including InfoNCE, NT-Xent, and Triplet Loss for representation learning and self-supervised training.
No direct links0 refs
Showing 91-96 of 174 articles