Skip to main content

Knowledge Distillation

Explore machine learning papers and reviews related to Knowledge Distillation. Find insights, analysis, and implementation details.

  • Tagged with
  • 5 papers
Back to all papers

Papers Related to Knowledge Distillation

TMLR 2024

DINOv2: Learning Robust Visual Features without Supervision

Maxime Oquab, Timothée Darcet, +24

How DINOv2 combines DINO self-distillation with iBOT masked prediction on curated LVD-142M data, with source-defined frozen-feature evaluations for classification, segmentation, depth, and retrieval.

2023

A Survey of Techniques for Optimizing Transformer Inference

Krishna Teja Chitty-Venkata, Sparsh Mittal, +3

Survey of transformer inference optimization: pruning, quantization, knowledge distillation, neural architecture search, and hardware acceleration.

Mastodon