Visual representations turn images into features. Self-supervised learning explains one way to learn those features before a particular labeled task is available. Once an encoder has been trained, a practical question follows: what should change when a new task arrives?
Suppose an encoder learned from a broad image collection and now needs to classify three kinds of manufacturing defects. Its existing output categories might be irrelevant, while some internal features might still help. Transfer learning reuses knowledge from an earlier training stage. Fine-tuning adapts some or all of the pretrained parameters through further training. The key choice is how much of the existing representation the downstream task may rewrite.
Carry the encoder into a new task
Write the downstream model as an encoder followed by a task head:
The encoder parameters begin at pretrained values . The head parameters map features to the new task's outputs. For three-class classification, a linear head can have weights and bias for a -coordinate feature vector. Its three logits are ; a classification loss compares them with the downstream label.
If the original head predicts unrelated classes, replace it rather than interpreting its old logits as the new labels. A self-supervised pretraining predictor may also be discarded: the module used to construct a pretraining objective need not be the module used for deployment. The retained encoder is the transferable part in this setup. Pretraining itself can be supervised or self-supervised.
Reusing a checkpoint also requires preserving its input contract: channel order, image normalization, spatial resolution rules, and the feature interface expected by the head. A weight file alone does not specify a complete model pipeline. The official PyTorch transfer-learning tutorial illustrates loading pretrained weights, preparing images, and replacing a classification layer.
Decide which parameters can move
Three common adaptation choices differ in their trainable parameter sets:
| Adaptation | Encoder parameters | Task head | What can change? |
|---|---|---|---|
| Frozen encoder, trained head | All fixed | Trained | The mapping from existing features to outputs |
| Partial fine-tuning | Selected blocks trained | Trained | Selected parts of the features and their output mapping |
| Full fine-tuning | All trainable | Trained | The whole representation and output mapping |
Training a linear head over fixed features is often called linear probing. Here it is an adaptation option; using that procedure to compare representations is a separate evaluation question. A nonlinear head trained on fixed features is still frozen-encoder adaptation, but it is not a linear probe.
For an encoder with an early block and a later block , the feature is . A common partial strategy freezes and trains plus the head. It preserves the early mapping while allowing the later mapping to adapt. Full fine-tuning allows both blocks to change. The PyTorch tutorial demonstrates the head-only and full-model cases.
Parameter selection can be written as a mask. With simple stochastic gradient descent, coordinate updates as:
A zero mask holds that parameter fixed; a one mask allows its gradient update. The mask describes which coordinates may move. The learning rate controls how far a permitted update moves them. This equation omits momentum, weight decay and adaptive optimizer state so that parameter selection remains visible.
Follow one update through the model
Consider a deliberately small regression model with two encoder weights and one head weight:
Start with , , , and . The forward pass gives , , and . Applying the chain rule gives:
Switch from Head only to Last block, First block, and Full model, then inspect Gradients and After update. Both block-specific options are partial fine-tuning. The learning-rate sliders independently control encoder and head updates. Setting the initial head to zero reveals a first-step dependency: the encoder derivatives contain , while the head derivative does not.
Forward. Every block computes its output, including frozen blocks. The head reads the final encoder feature.
x → a × x → b × (a × x) = z → w × z = ŷ
Early encoder block a
Frozena = 1
Last encoder block b
Frozenb = 0.5
Downstream head w
Trainablew = 0.5
Authored scalar regression example, not pretrained-model performance. Controls recompute one step from the same starting weights; they do not continue training. All changes are manual.
With head learning rate , head-only training changes to . Both encoder weights stay fixed, so the feature remains and the new prediction is . With encoder learning rate , partial fine-tuning also changes to : the new feature is and prediction . Full fine-tuning additionally changes to , producing feature and prediction .
All updates use derivatives from the same original forward pass. Recomputing a derivative after changing an earlier weight would be a different update rule. The explorer recomputes outputs afterward to expose what changed. Its lower loss on one constructed example says nothing about which strategy generalizes best on images.
Freezing weights does not remove a block
A frozen encoder still computes features. What stops is its parameter update. A mathematical derivative can exist even when a training framework is instructed not to accumulate it. The explorer shows those analytical derivatives for comparison and labels frozen coordinates separately.
In PyTorch, requires_grad_(False) prevents parameter-gradient accumulation. Give the optimizer the intended trainable parameters and clear stale gradients when changing that set. Zero gradients and exclusion differ: momentum or weight decay can matter. When unfreezing blocks, ensure the optimizer includes them. PyTorch autograd mechanics
For a frozen prefix whose inputs require no gradients, recording its operations is unnecessary for head training. A frozen block between trainable blocks must still transmit derivatives upstream. Wrapping that middle computation in no_grad() cuts the needed path. Freezing parameters and detaching computation are different decisions. PyTorch autograd mechanics
The explorer's First block mode makes this distinction visible: stays at , but its multiplication still carries the gradient back to . The early weight becomes while the head becomes . The prediction is then . Detaching at the middle block would prevent that early-weight update.
BatchNorm has state beyond trainable weights
Parameter freezing need not freeze the encoder's function. Batch normalization has learned affine parameters and plus running mean and variance. With running statistics enabled, training-mode forward passes update those statistics even when affine parameters receive no gradients. Evaluation uses the stored statistics. PyTorch BatchNorm2d documentation
For example, PyTorch's running-mean rule with momentum is:
With stored mean , batch mean , and , the new stored mean is . No optimizer step occurs, yet later evaluation outputs can change. This is calculated arithmetic, not a measured model result. PyTorch BatchNorm2d documentation
For fixed features, freeze encoder parameters and use encoder evaluation mode while training the head. Whole-model train() switches the encoder back; set encoder mode afterward. Adapting statistics can be intentional, but changes the encoder. With track_running_stats=False, BatchNorm uses batch statistics even in evaluation mode. PyTorch BatchNorm2d documentation
Evaluation mode changes BatchNorm and dropout behavior; it does not disable autograd or freeze parameters. Conversely, no_grad() does not set evaluation behavior. PyTorch autograd mechanics
Choose adaptation strength deliberately
Fine-tuning creates flexibility and exposure to the new training set at the same time. A frozen encoder cannot repair features that discard a distinction the downstream task needs. A fully trainable encoder can change those features, but it can also lose useful distinctions or specialize to accidental correlations. More trainable parameters are not a guarantee of better transfer.
The relationship between tasks matters. Yosinski and colleagues studied how transferability changes across network depth and across source and target tasks. Their experiments identified both feature specialization and difficulties caused by splitting co-adapted layers. This supports inspecting the transfer boundary rather than assuming every early block is universally reusable. How transferable are features in deep neural networks?
Use the following as starting hypotheses to test on downstream validation data:
| Situation | Candidate adaptation | Question to check |
|---|---|---|
| Few labels and already useful features | Frozen encoder plus head | Does the existing representation separate the needed outputs? |
| Evidence that later features need adaptation | Selected blocks plus head | Does unfreezing those blocks help beyond fitting the training set? |
| Enough task evidence to justify broader changes | Full fine-tuning with controlled updates | Do the broader changes improve the intended downstream behavior? |
| Shift affects low-level appearance | Consider earlier blocks too | Is freezing the input-side mapping preventing adaptation? |
These are choices, not a universal small-data/large-data recipe. Surgical Fine-Tuning experiments show that the useful subset can depend on the shift type; tuning early layers can help with image corruptions. Domain shift does not automatically imply that every layer should be unfrozen. Surgical Fine-Tuning Improves Adaptation to Distribution Shifts
Separate learning rates provide another control. A pretrained encoder already supplies a structured mapping, while a new head begins without task-specific training. A smaller encoder rate than head rate is a candidate way to limit representation changes; the toy example shows the arithmetic, not an optimal ratio. The appropriate rates also depend on optimizer, feature scale, batch size, schedule and training duration.
One staged option is to train the head first and then unfreeze selected blocks or the whole encoder. Kumar and colleagues found that linear probing followed by full fine-tuning could preserve transfer behavior better than immediate full fine-tuning in their distribution-shift experiments. Their result motivates testing such a schedule; it does not establish that staging always wins. Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution
Separate adapting a model from judging a representation
The adaptation decision specifies a training procedure: which checkpoint is reused, which head is attached, which parameters and buffers change, and how updates are controlled. A downstream result then reflects that whole procedure. Full fine-tuning performance cannot be attributed solely to the original frozen representation, because the representation has changed.
Before comparing two pretrained encoders, decide what access the downstream task is allowed to have. A fixed-feature head asks what a particular head can extract from existing features. Fine-tuning asks what can be achieved after adaptation under a particular training budget. Continue with Evaluating Visual Representations to make that comparison precise without mixing the two questions.
Related concepts
Follow how a learned target encoder supplies feature targets, how stop-gradient separates backward paths, and how exponential moving averages update its parameters.
Evaluate frozen image features with nearest neighbors and linear probes, separate training from validation and test data, and understand what each protocol can establish.
Follow pixels into image features, compare spatial maps with pooled vectors, and see why image encoders expose different outputs for recognition and localization.
Domain adaptation for embeddings: transfer learning to fine-tune retrieval models across domains while preventing catastrophic forgetting.
Understanding complete, dimensional, and cluster collapse — the failure modes that every self-supervised method must prevent. Learn why collapse happens and how contrastive, asymmetric, regularization, and masking approaches solve it.
Understand contrastive loss for representation learning: interactive demos of InfoNCE, triplet loss, and embedding space clustering with temperature tuning.
