Sliding Window Attention
Learn how Sliding Window Attention enables efficient processing of long sequences by limiting attention to local context windows, used in Mistral and Longformer.
No direct links0 refs
ML Concepts - Page 17
Clear explanations of core machine learning concepts, from foundational ideas to advanced techniques. Understand attention mechanisms, transformers, skip connections, and more.
167
Total Concepts
274
Topics
Page 17
of 28
Concepts - Page 17
Sparse Attention Patterns
Explore sparse attention mechanisms that reduce quadratic complexity to linear or sub-quadratic, enabling efficient processing of long sequences.
No direct links0 refs
He/Kaiming Initialization
Master He (Kaiming) initialization, the optimal weight initialization technique for ReLU networks that prevents gradient vanishing in deep neural architectures.
No direct links0 refs
Xavier/Glorot Initialization
Understand Xavier (Glorot) initialization, the weight initialization technique that maintains signal variance across layers for stable deep network training.
No direct links0 refs
MSE and MAE Loss Functions
Understand Mean Squared Error (MSE) and Mean Absolute Error (MAE), the fundamental loss functions for regression tasks with different sensitivity to outliers.
No direct links0 refs
SoA vs AoS: Data Layout Optimization
Master Structure of Arrays (SoA) vs Array of Structures (AoS) data layouts for optimal cache efficiency, SIMD vectorization, and GPU memory coalescing with interactive visualizations.
No direct links0 refs
Showing 97-102 of 167 articles