Building Networks: Layers, Activations, Initialisation & Losses
Assemble a multi-layer network piece by piece: dense layers, activation functions, why initialisation matters, output layers and stable losses, then train it from scratch.
Layers and parameters
A dense (fully connected, Linear) layer computes : every output is a weighted sum of every input plus a bias. With in inputs and out outputs it has parameters. A multi-layer perceptron (MLP) stacks them with an activation function after each hidden layer:
Without the nonlinearity , stacking would be pointless: a product of matrices is just one matrix, so the whole network would be a linear model. With it, even one wide hidden layer can approximate any reasonable function (the universal approximation theorem); in practice depth learns useful features with far fewer parameters than width.
Counting parameters is a basic skill: an MLP 784 → 256 → 128 → 10 has 784·256+256 + 256·128+128 + 128·10+10 = 235,146 parameters.
Activation functions
| Activation | Formula | Notes |
|---|---|---|
| Sigmoid | Outputs (0, 1). Saturates: for large |x| its gradient is nearly 0, so deep sigmoid networks learn slowly. Use only for binary outputs and gates. | |
| tanh | Outputs (−1, 1), zero-centred, still saturates. Used in RNN/LSTM cells. | |
| ReLU | Cheap, does not saturate for x > 0: the default for years. Units whose input is always negative get zero gradient and stop learning ("dead ReLUs"), often after a too-high learning rate. | |
| Leaky ReLU | A small slope for x < 0 keeps every unit alive. | |
| GELU / SiLU (Swish) | / | Smooth versions of ReLU; standard in transformers and modern CNNs. |
A reasonable default: ReLU (or GELU) in hidden layers; no activation on the final layer, because the loss applies the right output function (see below).
Initialisation: why the starting weights matter
Each layer multiplies its input by a weight matrix. If the weights are too small, the signal shrinks at every layer and after 10 layers the activations (and gradients) are essentially zero; if too large, they grow exponentially and explode. Training cannot start in either case.
The fix is to choose the random initial weights so that the variance of the signal stays roughly constant from layer to layer. For a layer with inputs:
- Xavier / Glorot (tanh, sigmoid, linear): (often simplified to ).
- He / Kaiming (ReLU family): . The factor 2 compensates for ReLU zeroing half of its inputs.
Biases start at zero. PyTorch layers already use a sensible default (a Kaiming-uniform variant), but when you write layers yourself, or a network won't train, initialisation is one of the first things to check. Normalisation layers and residual connections (later topics) make deep networks much less sensitive to it.
Try it: activations through 10 layers
Push a batch through a 10-layer network at initialisation. Each bar shows the spread of activations in one layer. Try a too-small and too-large scale, then the right scheme for each activation.
Interactive: this part runs in your browser.
Output layers and losses
The last layer and the loss go together, and should match the task:
| Task | Last layer | Loss | PyTorch |
|---|---|---|---|
| Regression | Linear, 1 output per target | MSE, or Huber if there are outliers | MSELoss, HuberLoss |
| Binary / multi-label | Linear, 1 logit per label | Binary cross-entropy on sigmoid(logit) | BCEWithLogitsLoss |
| Multi-class (one label) | Linear, C logits | Cross-entropy on softmax(logits) | CrossEntropyLoss (targets are class ids) |
Softmax turns logits into probabilities: . Cross-entropy is , averaged over the batch. Computed stably, , and its gradient with respect to the logits is the famously simple .
A sanity check worth memorising: at initialisation, with C balanced classes, the network's probabilities are roughly uniform, so the first loss should be about (2.30 for 10 classes). A starting loss far above that means overconfident initial outputs or a bug. Label smoothing (targets of 0.9 / 0.1/(C−1) instead of 1 / 0) is a common trick that reduces overconfidence.
Code example: forward and backward for an MLP
A worked example: read it step by step (every part is explained), then run it and experiment.
Interactive: this part runs in your browser.
Code example: train it on the spiral
A worked example: read it step by step (every part is explained), then run it and experiment.
Interactive: this part runs in your browser.
Code: the same MLP in PyTorch
Everything you wrote by hand, in a few lines. The training loop is the one from the PyTorch topic.
import torch, torch.nn as nn
class MLP(nn.Module):
def __init__(self, d_in=2, d_hidden=64, n_classes=3):
super().__init__()
self.net = nn.Sequential(
nn.Linear(d_in, d_hidden), nn.ReLU(),
nn.Linear(d_hidden, d_hidden), nn.ReLU(),
nn.Linear(d_hidden, n_classes)) # logits: no softmax here
for m in self.net:
if isinstance(m, nn.Linear):
nn.init.kaiming_normal_(m.weight, nonlinearity="relu"); nn.init.zeros_(m.bias)
def forward(self, x): # x: (B, d_in)
return self.net(x) # (B, n_classes)
model = MLP()
print(sum(p.numel() for p in model.parameters()), "parameters")
loss_fn = nn.CrossEntropyLoss() # softmax + log + NLL, stably
logits = model(torch.randn(8, 2))
print(loss_fn(logits, torch.randint(0, 3, (8,))).item(), "≈ ln 3 =", 1.0986)Quiz
1. Why does a deep network need nonlinear activations?
2. How many parameters does a Linear(100, 20) layer have?
3. For a ReLU network, which initialisation keeps the activation variance stable?
4. Why not initialise all weights to zero?
5. A 10-class classifier's first training loss is 47. What should you suspect?
6. Multi-class classification in PyTorch: the last layer and loss should be…
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- Dive into Deep Learning (free book with code) ↗ Interactive textbook in PyTorch, by Zhang, Lipton, Li & Smola
- Neural Networks: Zero to Hero (Karpathy) ↗ Video course building backprop, MLPs and GPT from scratch
- Deep Learning (Goodfellow, Bengio & Courville) ↗ The classic textbook, free online