Skip to main content

Transfer Learning and Fine-Tuning

Move a pretrained encoder to a downstream task: trace frozen features, head training, partial and full fine-tuning, learning rates, and BatchNorm state.

Visual representations turn images into features. Self-supervised learning explains one way to learn those features before a particular labeled task is available. Once an encoder has been trained, a practical question follows: what should change when a new task arrives?

Suppose an encoder learned from a broad image collection and now needs to classify three kinds of manufacturing defects. Its existing output categories might be irrelevant, while some internal features might still help. Transfer learning reuses knowledge from an earlier training stage. Fine-tuning adapts some or all of the pretrained parameters through further training. The key choice is how much of the existing representation the downstream task may rewrite.

Carry the encoder into a new task

Write the downstream model as an encoder followed by a task head:

z=fθ(x),y^=hϕ(z).z = f_\theta(x), \qquad \hat y = h_\phi(z).

The encoder parameters θ\theta begin at pretrained values θ0\theta_0. The head parameters ϕ\phi map features to the new task's outputs. For three-class classification, a linear head can have weights W∈R3×dW\in\mathbb{R}^{3\times d} and bias b∈R3b\in\mathbb{R}^3 for a dd-coordinate feature vector. Its three logits are Wz+bWz+b; a classification loss compares them with the downstream label.

If the original head predicts unrelated classes, replace it rather than interpreting its old logits as the new labels. A self-supervised pretraining predictor may also be discarded: the module used to construct a pretraining objective need not be the module used for deployment. The retained encoder is the transferable part in this setup. Pretraining itself can be supervised or self-supervised.

Reusing a checkpoint also requires preserving its input contract: channel order, image normalization, spatial resolution rules, and the feature interface expected by the head. A weight file alone does not specify a complete model pipeline. The official PyTorch transfer-learning tutorial illustrates loading pretrained weights, preparing images, and replacing a classification layer.

Decide which parameters can move

Three common adaptation choices differ in their trainable parameter sets:

AdaptationEncoder parametersTask headWhat can change?
Frozen encoder, trained headAll fixedTrainedThe mapping from existing features to outputs
Partial fine-tuningSelected blocks trainedTrainedSelected parts of the features and their output mapping
Full fine-tuningAll trainableTrainedThe whole representation and output mapping

Training a linear head over fixed features is often called linear probing. Here it is an adaptation option; using that procedure to compare representations is a separate evaluation question. A nonlinear head trained on fixed features is still frozen-encoder adaptation, but it is not a linear probe.

For an encoder with an early block faf_a and a later block fbf_b, the feature is z=fb(fa(x))z=f_b(f_a(x)). A common partial strategy freezes aa and trains bb plus the head. It preserves the early mapping while allowing the later mapping to adapt. Full fine-tuning allows both blocks to change. The PyTorch tutorial demonstrates the head-only and full-model cases.

Parameter selection can be written as a mask. With simple stochastic gradient descent, coordinate ii updates as:

θi′=θi−ηimi∂L∂θi,mi∈{0,1}.\theta_i' = \theta_i - \eta_i m_i\frac{\partial L}{\partial\theta_i}, \qquad m_i\in\{0,1\}.

A zero mask holds that parameter fixed; a one mask allows its gradient update. The mask describes which coordinates may move. The learning rate ηi\eta_i controls how far a permitted update moves them. This equation omits momentum, weight decay and adaptive optimizer state so that parameter selection remains visible.

Follow one update through the model

Consider a deliberately small regression model with two encoder weights and one head weight:

h=ax,z=bh,y^=wz,L=12(y^−y)2.h=ax, \qquad z=bh, \qquad \hat y=wz, \qquad L=\frac12(\hat y-y)^2.

Start with x=2x=2, y=2y=2, a=1a=1, b=0.5b=0.5 and w=0.5w=0.5. The forward pass gives h=2h=2, z=1z=1, y^=0.5\hat y=0.5 and L=1.125L=1.125. Applying the chain rule gives:

∂L∂a=(y^−y)wbx=−0.75,∂L∂b=(y^−y)wh=−1.5,∂L∂w=(y^−y)z=−1.5.\frac{\partial L}{\partial a}=(\hat y-y)wbx=-0.75, \quad \frac{\partial L}{\partial b}=(\hat y-y)wh=-1.5, \quad \frac{\partial L}{\partial w}=(\hat y-y)z=-1.5.

Switch from Head only to Last block, First block, and Full model, then inspect Gradients and After update. Both block-specific options are partial fine-tuning. The learning-rate sliders independently control encoder and head updates. Setting the initial head to zero reveals a first-step dependency: the encoder derivatives contain ww, while the head derivative does not.

Which weights can adapt?
Train
Inspect
Initial head w
0.02
0.10

Forward. Every block computes its output, including frozen blocks. The head reads the final encoder feature.

x → a × x → b × (a × x) = z → w × z = ŷ

Early encoder block a

Frozen

a = 1

Last encoder block b

Frozen

b = 0.5

Downstream head w

Trainable

w = 0.5

Authored scalar regression example, not pretrained-model performance. Controls recompute one step from the same starting weights; they do not continue training. All changes are manual.

With head learning rate 0.10.1, head-only training changes ww to 0.650.65. Both encoder weights stay fixed, so the feature remains 11 and the new prediction is 0.650.65. With encoder learning rate 0.020.02, partial fine-tuning also changes bb to 0.530.53: the new feature is 1.061.06 and prediction 0.6890.689. Full fine-tuning additionally changes aa to 1.0151.015, producing feature 1.07591.0759 and prediction 0.6993350.699335.

All updates use derivatives from the same original forward pass. Recomputing a derivative after changing an earlier weight would be a different update rule. The explorer recomputes outputs afterward to expose what changed. Its lower loss on one constructed example says nothing about which strategy generalizes best on images.

Freezing weights does not remove a block

A frozen encoder still computes features. What stops is its parameter update. A mathematical derivative can exist even when a training framework is instructed not to accumulate it. The explorer shows those analytical derivatives for comparison and labels frozen coordinates separately.

In PyTorch, requires_grad_(False) prevents parameter-gradient accumulation. Give the optimizer the intended trainable parameters and clear stale gradients when changing that set. Zero gradients and exclusion differ: momentum or weight decay can matter. When unfreezing blocks, ensure the optimizer includes them. PyTorch autograd mechanics

For a frozen prefix whose inputs require no gradients, recording its operations is unnecessary for head training. A frozen block between trainable blocks must still transmit derivatives upstream. Wrapping that middle computation in no_grad() cuts the needed path. Freezing parameters and detaching computation are different decisions. PyTorch autograd mechanics

The explorer's First block mode makes this distinction visible: bb stays at 0.50.5, but its multiplication still carries the gradient back to aa. The early weight becomes 1.0151.015 while the head becomes 0.650.65. The prediction is then 0.659750.65975. Detaching at the middle block would prevent that early-weight update.

BatchNorm has state beyond trainable weights

Parameter freezing need not freeze the encoder's function. Batch normalization has learned affine parameters γ\gamma and β\beta plus running mean and variance. With running statistics enabled, training-mode forward passes update those statistics even when affine parameters receive no gradients. Evaluation uses the stored statistics. PyTorch BatchNorm2d documentation

For example, PyTorch's running-mean rule with momentum qq is:

μt+1=(1−q)μt+qμbatch.\mu_{t+1}=(1-q)\mu_t+q\mu_{\text{batch}}.

With stored mean 22, batch mean 88, and q=0.1q=0.1, the new stored mean is 2.62.6. No optimizer step occurs, yet later evaluation outputs can change. This is calculated arithmetic, not a measured model result. PyTorch BatchNorm2d documentation

For fixed features, freeze encoder parameters and use encoder evaluation mode while training the head. Whole-model train() switches the encoder back; set encoder mode afterward. Adapting statistics can be intentional, but changes the encoder. With track_running_stats=False, BatchNorm uses batch statistics even in evaluation mode. PyTorch BatchNorm2d documentation

Evaluation mode changes BatchNorm and dropout behavior; it does not disable autograd or freeze parameters. Conversely, no_grad() does not set evaluation behavior. PyTorch autograd mechanics

Choose adaptation strength deliberately

Fine-tuning creates flexibility and exposure to the new training set at the same time. A frozen encoder cannot repair features that discard a distinction the downstream task needs. A fully trainable encoder can change those features, but it can also lose useful distinctions or specialize to accidental correlations. More trainable parameters are not a guarantee of better transfer.

The relationship between tasks matters. Yosinski and colleagues studied how transferability changes across network depth and across source and target tasks. Their experiments identified both feature specialization and difficulties caused by splitting co-adapted layers. This supports inspecting the transfer boundary rather than assuming every early block is universally reusable. How transferable are features in deep neural networks?

Use the following as starting hypotheses to test on downstream validation data:

SituationCandidate adaptationQuestion to check
Few labels and already useful featuresFrozen encoder plus headDoes the existing representation separate the needed outputs?
Evidence that later features need adaptationSelected blocks plus headDoes unfreezing those blocks help beyond fitting the training set?
Enough task evidence to justify broader changesFull fine-tuning with controlled updatesDo the broader changes improve the intended downstream behavior?
Shift affects low-level appearanceConsider earlier blocks tooIs freezing the input-side mapping preventing adaptation?

These are choices, not a universal small-data/large-data recipe. Surgical Fine-Tuning experiments show that the useful subset can depend on the shift type; tuning early layers can help with image corruptions. Domain shift does not automatically imply that every layer should be unfrozen. Surgical Fine-Tuning Improves Adaptation to Distribution Shifts

Separate learning rates provide another control. A pretrained encoder already supplies a structured mapping, while a new head begins without task-specific training. A smaller encoder rate than head rate is a candidate way to limit representation changes; the toy example shows the arithmetic, not an optimal ratio. The appropriate rates also depend on optimizer, feature scale, batch size, schedule and training duration.

One staged option is to train the head first and then unfreeze selected blocks or the whole encoder. Kumar and colleagues found that linear probing followed by full fine-tuning could preserve transfer behavior better than immediate full fine-tuning in their distribution-shift experiments. Their result motivates testing such a schedule; it does not establish that staging always wins. Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution

Separate adapting a model from judging a representation

The adaptation decision specifies a training procedure: which checkpoint is reused, which head is attached, which parameters and buffers change, and how updates are controlled. A downstream result then reflects that whole procedure. Full fine-tuning performance cannot be attributed solely to the original frozen representation, because the representation has changed.

Before comparing two pretrained encoders, decide what access the downstream task is allowed to have. A fixed-feature head asks what a particular head can extract from existing features. Fine-tuning asks what can be achieved after adaptation under a particular training budget. Continue with Evaluating Visual Representations to make that comparison precise without mixing the two questions.

If you found this explanation helpful, consider sharing it with others.

Mastodon