Adapt the model without updating every weight
A vision-language model combines visual features with a language model. For a new task, we can freeze its pretrained weights and train a smaller set of parameters. Parameter-efficient fine-tuning (PEFT) changes which parameters receive optimizer updates; the frozen model still participates in the forward pass, and gradients may need to flow through its operations to reach trainable modules.
Start with one linear projection. LoRA adds a trainable branch beside the frozen projection, then sums the two outputs. The branch can sit in a vision encoder, language model or connector, depending on the architecture and the task. For the connectors involved, see vision-language alignment compared: CLIP's shared space, BLIP-2's Q-Former, LLaVA's projector and Flamingo's gated cross-attention.
Follow a LoRA update
For a column vector x \in \mathbb{R}^{d_{\mathrm{in}}}, the bias-free projection is:
- W_0 \in \mathbb{R}^{d_{\mathrm{out}} \times d_{\mathrm{in}}} contains the frozen base weights.
- A \in \mathbb{R}^{r \times d_{\mathrm{in}}} is the trainable down-projection: it maps the input to an r-dimensional vector.
- B \in \mathbb{R}^{d_{\mathrm{out}} \times r} is the trainable up-projection: it maps that vector to the output width.
- \alpha/r scales the update in ordinary LoRA. This page uses that convention; some LoRA variants use a different scaling rule.
Thus \Delta W = (\alpha/r)BA has rank at most r and the same shape as W₀. The effective weight W_{\mathrm{eff}} = W_0 + \Delta W changes during adaptation even though the stored W₀ remains fixed. This is the distinction between a frozen base parameter and an updated effective projection. LoRA paper, PEFT LoRA documentation.
In the first figure, press Apply SGD step and compare B, the output and the toy loss. At initialization B is zero: A is trainable but has a zero gradient on the first step. After B changes, later gradients can update both matrices. The second figure uses larger shapes to count parameters; its rank and placement controls are independent of the tiny training example.
What the optimizer changes
For the toy's loss \mathcal{L} = \tfrac12\|y-t\|_2^2, write e=y-t, z=Ax and s=\alpha/r. The gradients are:
SGD computes both gradients at the current weights, then applies A \leftarrow A-\eta\nabla_A\mathcal{L} and B \leftarrow B-\eta\nabla_B\mathcal{L}. W₀ is excluded from the optimizer. A zero initial B gives a zero initial adapter output and a zero first gradient for A, not a permanently frozen A.
This PyTorch layer implements the same mechanism for batch-major inputs. It freezes the base weight and bias, initializes B to zero and leaves A/B trainable. The base layer's bias, if present, remains on the frozen path.
# examples/lora_linear.py
import math
import torch
from torch import nn
from torch.nn import functional as F
class LoRALinear(nn.Module):
"""Add a scaled low-rank update to a frozen linear projection."""
def __init__(self, base, rank=16, alpha=32):
super().__init__()
self.base = base.requires_grad_(False)
self.A = nn.Parameter(base.weight.new_empty(rank, base.in_features))
self.B = nn.Parameter(base.weight.new_zeros(base.out_features, rank))
nn.init.kaiming_uniform_(self.A, a=math.sqrt(5))
self.scaling = alpha / rank
def forward(self, x):
update = F.linear(F.linear(x, self.A), self.B)
return self.base(x) + self.scaling * update
def merged_weight(self):
"""Return the effective weight without changing the frozen base."""
return self.base.weight + self.scaling * (self.B @ self.A)
Count trainable parameters before estimating savings
One bias-free projection contributes:
For a 768 × 768 projection at rank 16, A/B contain 24,576 entries, compared with 589,824 frozen base entries. Across 48 such adapted projections, the trainable count is 1,179,648. These counts exclude biases and any separately trained connector or task head. In a real model, sum the counts over the actual target shapes, which need not be equal.
A small trainable fraction does not equal the same fraction of total GPU memory. A hypothetical model with exactly 7 billion weights needs 14 GB in bf16/fp16 or 28 GB in fp32 for weights alone, using decimal GB. Gradients, optimizer states, activations, temporary buffers and additional trainable modules are separate. Freezing reduces gradient and optimizer storage for those parameters; it does not eliminate the base weights or all activation storage. Quantization can reduce base-weight storage further, with its own implementation and accuracy considerations. These arithmetic counts do not establish that a particular training job fits on a consumer GPU.
The storage benefit is easier to isolate: several task-specific adapters can reference the same base checkpoint. Their compatibility still depends on the base revision, target modules and adapter configuration.
Compare what each method trains
LoRA is one way to adapt a frozen network. The following methods alter different parts of the computation; their parameter counts need different assumptions.
| Method | Trainable object | How it affects the forward pass |
|---|---|---|
| LoRA | A/B matrices at selected projections | Adds a scaled linear weight update |
| Bottleneck adapter | Down/up layers, usually with biases | Adds a small nonlinear residual branch |
| Prefix tuning | Per-layer attention prefixes, sometimes produced by an MLP | Adds key/value states for attention to read |
| BitFit | Existing bias parameters in the chosen model scope | Changes offsets while other parameters stay fixed |
Bottleneck adapters: a nonlinear residual branch
A simple bottleneck adapter maps width d down to b, applies a nonlinearity, maps back to d, then adds the residual:
Here D \in \mathbb{R}^{b\times d} and U \in \mathbb{R}^{d\times b}. Including both biases, this branch has 2db+b+d trainable parameters. Placement inside a transformer block is architecture-specific. Unlike ordinary LoRA's linear branch, this nonlinear branch generally cannot be merged into a single base weight matrix. Houlsby et al..
# examples/bottleneck_adapter.py
from torch import nn
from torch.nn import functional as F
class BottleneckAdapter(nn.Module):
"""Residual bottleneck; freeze the surrounding backbone separately."""
def __init__(self, dim, bottleneck_dim=64):
super().__init__()
self.down = nn.Linear(dim, bottleneck_dim)
self.up = nn.Linear(bottleneck_dim, dim)
def forward(self, x):
return x + self.up(F.relu(self.down(x)))
Prefix tuning: attention key/value states
Prefix tuning supplies trainable key and value states at attention layers. Input soft-prompt tuning instead prepends trainable embeddings at the model input; the two are related but not interchangeable. An MLP may generate a prefix during training. Access to the model's internal attention states and gradients is required, so this is not a method for a text-only black-box API. Li and Liang.
This minimal direct prefix has two tensors per layer, each shaped heads × prefix length × head width. Its count is 2L H p d_h. It deliberately omits an MLP and does not assume every architecture uses the same prefix interface.
# examples/attention_prefix.py
import torch
from torch import nn
class AttentionPrefix(nn.Module):
"""Produce per-layer K/V prefixes, not input token embeddings."""
def __init__(self, num_layers, num_heads, num_tokens, head_dim):
super().__init__()
self.states = nn.Parameter(
torch.randn(num_layers, 2, num_heads, num_tokens, head_dim) * 0.02
)
def for_layer(self, layer_index, batch_size):
key, value = self.states[layer_index].unbind(dim=0)
key = key.unsqueeze(0).expand(batch_size, -1, -1, -1)
value = value.unsqueeze(0).expand(batch_size, -1, -1, -1)
return key, value
At the selected layer, concatenate these states before the ordinary K/V along the sequence axis, and extend the attention mask so valid queries can attend to the prefix. Position handling, cache integration and grouped key/value heads must follow the target model. This class only produces the prefix tensors; it is not a complete multimodal integration.
BitFit: update existing biases
BitFit updates existing bias terms in the model scope passed to it. The trainable count is the sum of those bias sizes, not a universal percentage. A model or submodule without biases provides no trainable parameters through this function. Zaken et al..
# examples/bitfit.py
def apply_bitfit(model):
"""Train only parameters whose final name component is 'bias'."""
for name, parameter in model.named_parameters():
parameter.requires_grad_(name.rsplit(".", 1)[-1] == "bias")
return model
Pass the entire model to tune all its biases, or a named submodule to restrict the scope. Combining this policy with LoRA requires an explicit combined trainable-parameter policy; applying the function above after LoRA would freeze its A/B matrices.
Choose placement from the task and the actual model
The parameter ledger's vision/text blocks are a schematic, not the architecture of a specific released model. A multimodal task may benefit from adapting the visual encoder, language projections, the connector, or a combination. Treat placement and rank as experiment choices: inspect the model, record which modules train, and compare on a held-out task split. More rank increases the number of available update parameters; it does not guarantee better accuracy.
- Vision projections: test when the image domain changes or the existing visual features miss the target distinctions.
- Language projections: test when output style, terminology or reasoning behavior needs adaptation.
- Connector or interaction modules: test when visual features reach the language model through an inadequate bridge.
- Partial placement: test upper, lower or alternating blocks when a smaller adapter budget is useful.
Keep an adapter-only run distinct from partial or full fine-tuning. Unfreezing base layers changes the optimizer state, trainable count and memory requirements. If several tasks need adapters, save and select them separately; adding or merging unrelated task updates does not guarantee useful multi-task behavior.
A verified PEFT example
This CPU, no-download API smoke example was executed with PEFT 0.21.2, PyTorch 2.12.0 and Transformers 5.16.1. It adapts a tiny nn.Module, checks freezing and trainable counts, performs one optimizer step and verifies merge equivalence. It does not train LLaVA or measure multimodal accuracy. PEFT 0.21.2 release, custom-model guide.
# examples/peft_0_21_2_smoke.py
import torch
from peft import LoraConfig, TaskType, get_peft_model
from torch import nn
class TinyProjection(nn.Module):
def __init__(self):
super().__init__()
self.proj = nn.Linear(3, 2, bias=False)
def forward(self, x):
return self.proj(x)
torch.manual_seed(7)
base = TinyProjection()
with torch.no_grad():
base.proj.weight.copy_(torch.tensor([[1.0, 0.0, -0.5], [0.0, 1.0, 0.5]]))
config = LoraConfig(
r=1,
lora_alpha=1,
target_modules=["proj"],
lora_dropout=0.0,
bias="none",
# Omit task_type for the generic nn.Module wrapper used here.
)
model = get_peft_model(base, config)
projection = model.get_base_model().proj
with torch.no_grad():
# Fixed A makes this example reproduce the interactive toy exactly.
projection.lora_A["default"].weight.copy_(torch.tensor([[0.5, -0.5, 1.0]]))
trainable = [
parameter for parameter in model.parameters() if parameter.requires_grad
]
assert sum(parameter.numel() for parameter in trainable) == 5
frozen_before = projection.base_layer.weight.detach().clone()
x = torch.tensor([[1.0, 2.0, -1.0]])
target = torch.tensor([[2.0, 1.0]])
optimizer = torch.optim.SGD(trainable, lr=0.1)
optimizer.zero_grad()
loss = 0.5 * (model(x) - target).square().sum()
loss.backward()
assert projection.base_layer.weight.grad is None
optimizer.step()
assert torch.equal(projection.base_layer.weight, frozen_before)
model.eval()
with torch.no_grad():
adapted_output = model(x)
torch.testing.assert_close(adapted_output, torch.tensor([[1.6125, 1.3875]]))
merged = model.merge_and_unload()
with torch.no_grad():
torch.testing.assert_close(merged(x), adapted_output)
PEFT's TaskType describes wrapper behavior such as causal language modeling or feature extraction. TaskType.MULTIMODAL does not exist in the tested version. For a real vision-language model, choose a wrapper and training integration that support its forward signature; an image input does not by itself define a new PEFT task enum. Documented task types.
Likewise, target_modules=["q_proj", "v_proj"] matches exact module names or suffixes, not shell glob patterns. A list entry containing * does not mean “every layer.” A single string can instead specify a regular expression. Inspect model.named_modules() and check the selected modules against the specific model revision, including which belong to the vision tower or language model. LoraConfig reference.
Evaluate and deploy the adaptation
A useful experiment report names the base checkpoint/revision, target modules and shapes, adapter method/rank/scaling, trainable parameters, precision, dataset splits, optimizer, training budget and evaluation metric. Report measured task scores and peak memory with that configuration. The figures here supply exact arithmetic and a toy gradient trace; they supply no VQA scores, GPU-memory measurements or empirical rank-to-quality scaling law.
For an ordinary unquantized LoRA projection in evaluation mode, merging computes W_{\mathrm{merged}} = W_0 + (\alpha/r)BA. The tested example uses PEFT's merge_and_unload() and checks the output, instead of manually editing .data or assuming every module exposes the same adapter representation. Other methods and quantized configurations need their own supported deployment path.
Separate adapters can share a base checkpoint, but selecting an adapter and merging one into base weights are different serving operations. Keep the original base, adapter configuration and evaluation results so each task's behavior is reproducible. The next step is to understand vision-language alignment, then decide which part of that architecture the new task needs to change.
References
- Hu et al. LoRA: Low-Rank Adaptation of Large Language Models.
- Houlsby et al. Parameter-Efficient Transfer Learning for NLP.
- Li and Liang. Prefix-Tuning: Optimizing Continuous Prompts for Generation.
- Zaken et al. BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models.
- Liu et al. Visual Instruction Tuning.
- Hugging Face. PEFT LoRA reference and task types.
Related concepts
Trace one image-caption pair through cosine scores, contrastive positives and negatives, a trainable projection, and retrieval evaluation.
Measure the centroid gap between normalized image and text embeddings, then check why a smaller gap alone does not guarantee correct retrieval.
Explore a fixed compute budget with Chinchilla’s text-only loss fit, then see why multimodal scaling needs its own data, objectives, and evaluation metrics.
CLIP, BLIP-2, LLaVA and Flamingo align images with text by different routes. Compare their objectives, bridges, data, and which parts train or stay frozen.
Trace a target query through source keys, attention weights, and a weighted value mix. Understand cross-attention origins, shapes, masks, and applications.
Move a pretrained encoder to a downstream task: trace frozen features, head training, partial and full fine-tuning, learning rates, and BatchNorm state.
