Deep Learning
Topic 7 ยท Deep Learning ยท โฑ about 6 h

Normalisation & Regularisation

Batch, layer and RMS normalisation, dropout, weight decay, data augmentation, early stopping and label smoothing: what each does and when to use it.

Show:
In this topic
Beginner

Big networks, small data

Modern networks have far more parameters than training examples and could memorise the training set, labels and all. They generalise anyway thanks to a combination of good architecture, SGD's implicit preferences, and regularisation: anything that makes the model prefer simpler, more stable solutions. The toolbox:

  • More (and augmented) data: the most effective regulariser of all.
  • Weight decay: keep weights small (L2 / AdamW).
  • Dropout: randomly switch units off during training.
  • Early stopping: stop when validation loss stops improving.
  • Label smoothing, mixup: discourage over-confident outputs.

Normalisation layers are mainly there to make optimisation easier and faster, with a mild regularising side effect.

Beginner

Watch it: Dropout: a different thinned network every step

A short narrated animation. Press โ–ถ Play, or step through it with โฎ โญ. Tick ๐Ÿ”Š narrate to hear each step.

Interactive: this part runs in your browser.

Beginner

Dropout

During training, dropout sets each activation to zero with probability p (0.1 to 0.5), choosing a new random mask for every example and step. No unit can rely on a particular other unit being present, so the network learns redundant, robust features; it behaves like training a large ensemble of thinned networks that share weights.

At test time nothing is dropped, so the expected activation must be the same in both modes. Inverted dropout, which every framework uses, scales the kept activations by 1/(1โˆ’p)1/(1-p) during training, so evaluation needs no change:

h~=mโŠ™h1โˆ’p,miโˆผBernoulli(1โˆ’p).\tilde h = \frac{m\odot h}{1-p},\qquad m_i\sim\text{Bernoulli}(1-p).

This is why model.train() and model.eval() matter: forgetting eval() leaves dropout on at evaluation, which gives noisy, worse predictions. Dropout is common in dense layers and transformers (p โ‰ˆ 0.1); CNNs rely more on augmentation and weight decay.

Beginner

Batch normalisation

BatchNorm standardises each feature (each channel, for images) using the mean and variance of the current batch, then applies a learned scale ฮณ and shift ฮฒ:

x^=xโˆ’ฮผbatchฯƒbatch2+ฯต,y=ฮณx^+ฮฒ.\hat x=\frac{x-\mu_\text{batch}}{\sqrt{\sigma^2_\text{batch}+\epsilon}},\qquad y=\gamma\hat x+\beta.

Keeping every layer's inputs in a stable range allows much higher learning rates and makes deep CNNs train far more reliably. During training BatchNorm also keeps a running average of the mean and variance; in eval() mode it uses those instead of the batch statistics, so a single example gives a deterministic output.

Its weaknesses all come from depending on the batch: it works poorly with very small batches (statistics too noisy), behaves differently in training and evaluation (a classic source of "it worked in training" bugs), and is awkward for variable-length sequences. In fine-tuning with small batches, people often freeze BatchNorm statistics.

Beginner

Layer, group and RMS normalisation

LayerNorm normalises across the features of each example instead of across the batch, so it behaves identically in training and evaluation and for any batch size or sequence length. It is the standard in transformers and RNNs. RMSNorm skips the mean subtraction and divides by the root mean square only; it is a little cheaper and used in many recent large language models. GroupNorm normalises groups of channels per example: a good choice for CNNs trained with small batches (detection, segmentation, medical images).

Normalise overโ€ฆNameTypical use
the batch, per feature/channelBatchNormCNNs with decent batch sizes
the features, per exampleLayerNorm / RMSNormTransformers, RNNs
groups of channels, per exampleGroupNormCNNs with small batches
Try it

Try it: regularise a network on small data

A small network learns from only 40 noisy points. Train it, then try weight decay and dropout, and watch the decision boundary and the gap between training and validation loss.

Interactive: this part runs in your browser.

Beginner

Data augmentation, early stopping and smoothing

Data augmentation creates new training examples by transformations that do not change the label: flips, crops, small rotations, colour jitter and noise for images; synonym replacement or back-translation for text; time shifts and noise for signals. Choose only label-preserving transforms (flipping a "6" makes a "9"; flipping a chest X-ray swaps left and right anatomy), and apply them to training data only.

Early stopping evaluates on the validation set every epoch and keeps the weights from the best epoch; training stops after a "patience" of several epochs without improvement. It is free and should be on by default.

Label smoothing (e.g. 0.1) and mixup (training on blends of two examples and their labels) discourage the network from becoming over-confident, which also improves calibration.

Try it

Code example: batch norm and dropout

A worked example: read it step by step (every part is explained), then run it and experiment.

Interactive: this part runs in your browser.

Code

Code: the PyTorch building blocks

The same tools as layers and arguments.

Python ยท PyTorch
import torch.nn as nn
block = nn.Sequential(
    nn.Conv2d(64, 128, 3, padding=1, bias=False),   # bias is redundant before BatchNorm
    nn.BatchNorm2d(128), nn.ReLU(),
    nn.Dropout(p=0.1))
ln = nn.LayerNorm(512)                             # transformers: normalise each token's 512 features
loss_fn = nn.CrossEntropyLoss(label_smoothing=0.1)
model.train()   # dropout on, BatchNorm uses batch stats and updates running stats
model.eval()    # dropout off, BatchNorm uses running stats: ALWAYS before validation/inference
This code needs PyTorch, which cannot run inside a web page. To run it, press Copy, then paste it into a new notebook in Google Colab โ†— (free; Runtime โ†’ Change runtime type โ†’ GPU for deep learning).
Check yourself

Quiz

1. With inverted dropout (p = 0.2), what happens at test time?

2. Your model scores well on validation inside the training loop but badly in production. You notice model.eval() is never called. Why does that matter?

3. Why do transformers use LayerNorm rather than BatchNorm?

4. You train a segmentation CNN with batch size 2. Which normalisation is a better choice than BatchNorm?

5. Which augmentation is NOT label-preserving for handwritten digits?

6. Early stopping should monitorโ€ฆ

Go deeper

๐Ÿ“š Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready