Deep Learning
Topic 14 · Deep Learning · ⏱ about 8 h

Transfer Learning, Fine-Tuning & Embeddings

Reuse what a pretrained model already knows: feature extraction vs fine-tuning, choosing a strategy, practical fine-tuning, LoRA and other parameter-efficient methods, and embeddings.

Show:
In this topic
Beginner

Why pretrained models transfer

Training a large network from scratch needs millions of labelled examples and a lot of compute. Most teams have neither. Fortunately, what a network learns on a big general dataset (ImageNet photos, web text) is largely reusable: early CNN layers detect edges and textures that appear in X-rays and scanned forms too, and a language model's understanding of grammar and meaning helps with clinic messages. Transfer learning starts from such a pretrained model and adapts it to your task with far less data.

Two basic ways to adapt it:

  • Feature extraction (linear probing): freeze the pretrained network, use its penultimate-layer outputs as features, and train only a new small head (a linear layer or logistic regression). Fast, needs very little data, hard to overfit.
  • Fine-tuning: replace the head and continue training the whole network (or its top layers) on your data with a small learning rate. Usually more accurate when you have enough data, but easier to overfit and more expensive.
Beginner

Watch it: Transfer learning: freeze, replace the head, then fine-tune

A short narrated animation. Press ▶ Play, or step through it with ⏮ ⏭. Tick 🔊 narrate to hear each step.

Interactive: this part runs in your browser.

Beginner

Choosing a strategy

Your data is similar to the pretraining dataYour data is quite different (e.g. medical scans vs photos)
Few labels (hundreds)Feature extraction + linear headFeature extraction from earlier layers; consider a domain-specific pretrained model; augment heavily
Many labels (tens of thousands+)Fine-tune the whole networkFine-tune the whole network (pretraining still helps optimisation); domain pretraining if available

Before any of this, check whether a domain-specific pretrained model exists (for example models pretrained on medical images or clinical text): it usually beats a general one. And always compare with a simple baseline trained on the same split.

Try it

Try it: which strategy, how many labels?

Set how many labelled examples you have and how close your data is to the pretraining data. The chart shows typical (illustrative) learning curves for each strategy.

Interactive: this part runs in your browser.

Beginner

Fine-tuning well

  • Small learning rate: typically 10 to 100 times smaller than training from scratch (e.g. 1e-5 to 1e-4 with AdamW), with warm-up. Large rates destroy the pretrained features ("catastrophic forgetting").
  • Train the new head first, then unfreeze: a randomly initialised head sends large, noisy gradients into the pretrained layers. Train the head for an epoch with the backbone frozen, then unfreeze (gradually, top layers first, if data is small).
  • Discriminative learning rates: smaller rates for early layers (general features), larger for later ones.
  • Use the same preprocessing as in pretraining: image size, normalisation mean/std, tokenizer. Mismatches quietly cost accuracy.
  • BatchNorm with small batches: keep it in eval mode (frozen statistics) during fine-tuning.
  • Check the licence and model card of the pretrained model: allowed uses, training data, known limitations.
Beginner

Parameter-efficient fine-tuning: LoRA

Fully fine-tuning a model with billions of parameters needs memory for all their gradients and optimiser states, and a full copy of the model per task. Parameter-efficient fine-tuning (PEFT) freezes the original weights and trains a small number of new ones.

LoRA (low-rank adaptation) is the most popular. For a frozen weight matrix W (d × d), it learns a low-rank update made of two thin matrices, B (d × r) and A (r × d), with r much smaller than d (often 8 to 64):

W′=W+αr BA,trainable parameters 2dr≪d2.W' = W + \tfrac{\alpha}{r}\,BA,\qquad \text{trainable parameters } 2dr \ll d^2.

B starts at zero, so training begins exactly at the pretrained model. With d = 4,096 and r = 16, that is 131,072 trainable parameters instead of 16.8 million per matrix (0.8%). Adapters for many tasks can share one base model, and after training BA can be merged into W, so inference costs nothing extra. QLoRA additionally stores the frozen base model in 4-bit precision, so large models can be fine-tuned on a single GPU. Other PEFT methods include adapter layers and prompt tuning.

Beginner

Embeddings: a pretrained model as a feature extractor

The vector a network produces just before its final layer is an embedding: a compact representation in which similar inputs are close together. Embeddings from pretrained models power semantic search and retrieval-augmented generation (see the LLM topic), duplicate detection, clustering, recommendation and few-shot classification with a simple classifier on top.

Compare embeddings with cosine similarity (usually after normalising them to length 1), and use a vector index (FAISS, a vector database) when there are millions. An embedding model should be evaluated on your data: retrieve examples, label whether the neighbours are truly similar, and compare models.

Try it

Code example: few labels, pretrained features

A worked example: read it step by step (every part is explained), then run it and experiment.

Interactive: this part runs in your browser.

Code

Code: fine-tune a pretrained ResNet, and LoRA with PEFT

The two most common fine-tuning set-ups.

Python · torchvision
import torch, torch.nn as nn
from torchvision.models import resnet50, ResNet50_Weights
w = ResNet50_Weights.DEFAULT
model = resnet50(weights=w)
preprocess = w.transforms()                         # SAME preprocessing as pretraining
for p in model.parameters(): p.requires_grad = False
model.fc = nn.Linear(model.fc.in_features, 4)       # new head: 4 document types
# stage 1: train only model.fc for an epoch or two (lr 1e-3)
# stage 2: unfreeze, smaller lr for the backbone
for p in model.parameters(): p.requires_grad = True
opt = torch.optim.AdamW([{"params": model.fc.parameters(), "lr": 1e-3},
                         {"params": [p for n, p in model.named_parameters() if not n.startswith("fc")], "lr": 1e-5}],
                        weight_decay=0.01)
This code needs PyTorch, which cannot run inside a web page. To run it, press Copy, then paste it into a new notebook in Google Colab ↗ (free; Runtime → Change runtime type → GPU for deep learning).
Python · Hugging Face PEFT
from transformers import AutoModelForSequenceClassification
from peft import LoraConfig, get_peft_model
base = AutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased", num_labels=3)
cfg = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.1, target_modules=["q_lin", "v_lin"])
model = get_peft_model(base, cfg)
model.print_trainable_parameters()     # roughly 1% of parameters are trainable
This code needs Hugging Face transformers, which cannot run inside a web page. To run it, press Copy, then paste it into a new notebook in Google Colab ↗ (free; Runtime → Change runtime type → GPU for deep learning).
Check yourself

Quiz

1. You have 300 labelled X-ray images and a general ImageNet model. A sensible first approach is…

2. Why use a much smaller learning rate when fine-tuning?

3. Why train the new head before unfreezing the backbone?

4. With LoRA rank r = 8 on a 1,024 × 1,024 matrix, how many trainable parameters are added?

5. Your model was pretrained with images normalised by ImageNet mean/std at 224 px. In production you feed 0–255 pixels at 512 px. Expect…

6. How should two embeddings usually be compared?

Go deeper

📚 Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready