Mixture of Experts (MoE)
Understanding sparse mixture of experts models - architecture, routing mechanisms, load balancing, and efficient scaling strategies for large language models
Related
12 min read · advanced
ML Concepts - Page 27
Clear explanations of core machine learning concepts, from foundational ideas to advanced techniques. Understand attention mechanisms, transformers, skip connections, and more.
194
Total Concepts
341
Topics
Page 27
of 33
Concepts - Page 27
Browse by categoryInternal Covariate Shift
Understand internal covariate shift: why layer input distributions change during training, how it slows convergence, and how batch norm fixes it.
Related
8 min read
NCCL: How NVIDIA Collective Communication Works
A deep dive into NCCL internals: communicators and channels, how it picks ring/tree/NVLS algorithms and LL/LL128/Simple protocols, reading NCCL_DEBUG logs, and tuning and debugging distributed training.
Related
12 min read · intermediate
C++ Preprocessor Directives
C++ preprocessor visualized: macros, header guards, conditional compilation, and #include directives explained interactively.
Related
5 min read
SoA vs AoS: Data Layout Optimization
Structure of Arrays vs Array of Structures as instruments: cache-line fill, SIMD gather vs contiguous load, GPU coalescing, and AoSoA hybrids — when layout is a 10× decision.
Related
12 min read · intermediate
Speculative Decoding: Draft, Then Verify
A small draft model proposes tokens, the target model checks them all in one pass, and a rejection rule keeps the output distribution exactly the target's.
Related
10 min read · intermediate
Showing 157-162 of 194 concepts