Normalisation & Regularisation
Batch, layer and RMS normalisation, dropout, weight decay, data augmentation, early stopping and label smoothing: what each does and when to use it.
Big networks, small data
Modern networks have far more parameters than training examples and could memorise the training set, labels and all. They generalise anyway thanks to a combination of good architecture, SGD's implicit preferences, and regularisation: anything that makes the model prefer simpler, more stable solutions. The toolbox:
- More (and augmented) data: the most effective regulariser of all.
- Weight decay: keep weights small (L2 / AdamW).
- Dropout: randomly switch units off during training.
- Early stopping: stop when validation loss stops improving.
- Label smoothing, mixup: discourage over-confident outputs.
Normalisation layers are mainly there to make optimisation easier and faster, with a mild regularising side effect.
Watch it: Dropout: a different thinned network every step
A short narrated animation. Press โถ Play, or step through it with โฎ โญ. Tick ๐ narrate to hear each step.
Interactive: this part runs in your browser.
Dropout
During training, dropout sets each activation to zero with probability p (0.1 to 0.5), choosing a new random mask for every example and step. No unit can rely on a particular other unit being present, so the network learns redundant, robust features; it behaves like training a large ensemble of thinned networks that share weights.
At test time nothing is dropped, so the expected activation must be the same in both modes. Inverted dropout, which every framework uses, scales the kept activations by during training, so evaluation needs no change:
This is why model.train() and model.eval() matter: forgetting eval() leaves dropout on at evaluation, which gives noisy, worse predictions. Dropout is common in dense layers and transformers (p โ 0.1); CNNs rely more on augmentation and weight decay.
Batch normalisation
BatchNorm standardises each feature (each channel, for images) using the mean and variance of the current batch, then applies a learned scale ฮณ and shift ฮฒ:
Keeping every layer's inputs in a stable range allows much higher learning rates and makes deep CNNs train far more reliably. During training BatchNorm also keeps a running average of the mean and variance; in eval() mode it uses those instead of the batch statistics, so a single example gives a deterministic output.
Its weaknesses all come from depending on the batch: it works poorly with very small batches (statistics too noisy), behaves differently in training and evaluation (a classic source of "it worked in training" bugs), and is awkward for variable-length sequences. In fine-tuning with small batches, people often freeze BatchNorm statistics.
Layer, group and RMS normalisation
LayerNorm normalises across the features of each example instead of across the batch, so it behaves identically in training and evaluation and for any batch size or sequence length. It is the standard in transformers and RNNs. RMSNorm skips the mean subtraction and divides by the root mean square only; it is a little cheaper and used in many recent large language models. GroupNorm normalises groups of channels per example: a good choice for CNNs trained with small batches (detection, segmentation, medical images).
| Normalise overโฆ | Name | Typical use |
|---|---|---|
| the batch, per feature/channel | BatchNorm | CNNs with decent batch sizes |
| the features, per example | LayerNorm / RMSNorm | Transformers, RNNs |
| groups of channels, per example | GroupNorm | CNNs with small batches |
Try it: regularise a network on small data
A small network learns from only 40 noisy points. Train it, then try weight decay and dropout, and watch the decision boundary and the gap between training and validation loss.
Interactive: this part runs in your browser.
Data augmentation, early stopping and smoothing
Data augmentation creates new training examples by transformations that do not change the label: flips, crops, small rotations, colour jitter and noise for images; synonym replacement or back-translation for text; time shifts and noise for signals. Choose only label-preserving transforms (flipping a "6" makes a "9"; flipping a chest X-ray swaps left and right anatomy), and apply them to training data only.
Early stopping evaluates on the validation set every epoch and keeps the weights from the best epoch; training stops after a "patience" of several epochs without improvement. It is free and should be on by default.
Label smoothing (e.g. 0.1) and mixup (training on blends of two examples and their labels) discourage the network from becoming over-confident, which also improves calibration.
Code example: batch norm and dropout
A worked example: read it step by step (every part is explained), then run it and experiment.
Interactive: this part runs in your browser.
Code: the PyTorch building blocks
The same tools as layers and arguments.
import torch.nn as nn
block = nn.Sequential(
nn.Conv2d(64, 128, 3, padding=1, bias=False), # bias is redundant before BatchNorm
nn.BatchNorm2d(128), nn.ReLU(),
nn.Dropout(p=0.1))
ln = nn.LayerNorm(512) # transformers: normalise each token's 512 features
loss_fn = nn.CrossEntropyLoss(label_smoothing=0.1)
model.train() # dropout on, BatchNorm uses batch stats and updates running stats
model.eval() # dropout off, BatchNorm uses running stats: ALWAYS before validation/inferenceQuiz
1. With inverted dropout (p = 0.2), what happens at test time?
2. Your model scores well on validation inside the training loop but badly in production. You notice model.eval() is never called. Why does that matter?
3. Why do transformers use LayerNorm rather than BatchNorm?
4. You train a segmentation CNN with batch size 2. Which normalisation is a better choice than BatchNorm?
5. Which augmentation is NOT label-preserving for handwritten digits?
6. Early stopping should monitorโฆ
๐ Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- Dive into Deep Learning (free book with code) โ Interactive textbook in PyTorch, by Zhang, Lipton, Li & Smola
- Understanding Deep Learning (Prince) โ Modern, clear and free textbook
- Deep Learning (Goodfellow, Bengio & Courville) โ The classic textbook, free online