Explore machine learning papers and reviews related to Knowledge Distillation. Find insights, analysis, and implementation details.
Maxime Oquab, Timothée Darcet, +24
How DINOv2 combines DINO self-distillation with iBOT masked prediction on curated LVD-142M data, with source-defined frozen-feature evaluations for classification, segmentation, depth, and retrieval.
Jean-Bastien Grill, Florian Strub, +12
How self-supervised learning works without negative pairs — a predictor and momentum target network enable learning without a contrastive negative-pair loss.
Mathilde Caron, Hugo Touvron, +5
How self-distillation with no labels produces Vision Transformer attention maps that automatically segment objects — without any pixel-level supervision.
Krishna Teja Chitty-Venkata, Sparsh Mittal, +3
Survey of transformer inference optimization: pruning, quantization, knowledge distillation, neural architecture search, and hardware acceleration.