Explore machine learning papers and reviews related to Knowledge Distillation. Find insights, analysis, and implementation details.
Maxime Oquab, Timothée Darcet, +24
How DINOv2 combines DINO self-distillation with iBOT masked prediction at scale on curated data (LVD-142M), producing the strongest open-source frozen visual features across classification, segmentation, depth, and retrieval.
Jean-Bastien Grill, Florian Strub, +12
How self-supervised learning works without negative pairs — a predictor and momentum target network are all you need to prevent representation collapse.
Mathilde Caron, Hugo Touvron, +5
How self-distillation with no labels produces Vision Transformer attention maps that automatically segment objects — without any pixel-level supervision.
Krishna Teja Chitty-Venkata, Sparsh Mittal, +3
Survey of transformer inference optimization: pruning, quantization, knowledge distillation, neural architecture search, and hardware acceleration.