Machine Learning
Topic 45 · Tools & libraries · ⏱ about 6 h

PyTorch Essentials

The library step by step: tensors and devices, autograd, nn.Module, Dataset and DataLoader, the training loop, evaluation mode, saving and loading, and the errors everyone hits.

Show:
In this topic
Beginner

What PyTorch gives you

PyTorch is NumPy with three extra powers: tensors can live on a GPU, operations are recorded so gradients are computed automatically (autograd), and it ships building blocks for networks, optimisers and data loading. Prerequisites: NumPy and the ideas in Neural Networks and Training Neural Networks.

PyTorch cannot run inside this web page, so the browser exercise below writes the same pattern in NumPy, and notebook 13 (at the end of this topic) runs real PyTorch in Google Colab with a free GPU.

Beginner

Tensors: shape, dtype, device

import torch
x = torch.tensor([[1., 2.], [3., 4.]])        # from data
z = torch.zeros(3, 4); r = torch.randn(64, 10) # common constructors
x.shape, x.dtype, x.device                    # check these constantly
x.mean(dim=0), x.sum(dim=1, keepdim=True)     # 'dim' is NumPy's 'axis'
x @ x.T, x.view(4), x.unsqueeze(0).shape      # matmul, reshape, add an axis -> (1, 2, 2)
device = "cuda" if torch.cuda.is_available() else "cpu"
x = x.to(device)                              # move to the GPU
x.cpu().numpy(), torch.from_numpy(a)          # to / from NumPy (CPU only)

Rules that prevent most errors: model and data must be on the same device; class labels for cross_entropy are torch.long; inputs are usually float32.

Beginner

Autograd and nn.Module

Set requires_grad=True (parameters have it already) and call loss.backward(): every parameter's .grad receives d(loss)/d(parameter). Gradients accumulate, which is why each step starts with zero_grad().

from torch import nn
class MLP(nn.Module):
    def __init__(self, d_in, hidden, n_out):
        super().__init__()                              # always call this first
        self.net = nn.Sequential(nn.Linear(d_in, hidden), nn.ReLU(), nn.Dropout(0.1),
                                 nn.Linear(hidden, n_out))
    def forward(self, x):
        return self.net(x)                              # return logits; the loss applies softmax
model = MLP(20, 64, 3)
sum(p.numel() for p in model.parameters())            # parameter count

Layers assigned as attributes in __init__ are registered, so model.parameters(), model.to(device) and model.state_dict() find them. A plain Python list of layers is not registered: use nn.ModuleList.

Try it

Try it: build a network and track the shapes

Stack layers and see the output shape and parameter count of each one, as torchinfo.summary would show.

Interactive: this part runs in your browser.

Beginner

Dataset, DataLoader and the training loop

from torch.utils.data import Dataset, DataLoader
class TableData(Dataset):
    def __init__(self, X, y): self.X = torch.tensor(X, dtype=torch.float32); self.y = torch.tensor(y)
    def __len__(self): return len(self.y)
    def __getitem__(self, i): return self.X[i], self.y[i]
train_dl = DataLoader(TableData(X_tr, y_tr), batch_size=64, shuffle=True)   # shuffle training only
val_dl = DataLoader(TableData(X_va, y_va), batch_size=256)

opt = torch.optim.AdamW(model.parameters(), lr=1e-3)
for epoch in range(20):
    model.train()                                   # dropout / batch norm in training mode
    for xb, yb in train_dl:
        xb, yb = xb.to(device), yb.to(device)
        loss = nn.functional.cross_entropy(model(xb), yb)
        opt.zero_grad(); loss.backward(); opt.step()
    model.eval()                                    # evaluation mode
    with torch.no_grad():                           # no graph: faster, less memory
        val_loss = sum(nn.functional.cross_entropy(model(x.to(device)), y.to(device), reduction="sum").item()
                       for x, y in val_dl) / len(val_dl.dataset)
Try it

Code example: the PyTorch pattern in NumPy

A worked example: read it step by step (every part is explained), then run it and experiment.

Interactive: this part runs in your browser.

Beginner

Saving, loading and predicting

torch.save(model.state_dict(), "model.pt")                    # save the weights, not the object
model = MLP(20, 64, 3); model.load_state_dict(torch.load("model.pt", map_location="cpu")); model.eval()
with torch.inference_mode():
    probs = torch.softmax(model(x_new), dim=1)
# to resume training, also save opt.state_dict() and the epoch number
Beginner

The errors everyone hits

Error message (shortened)Usual causeFix
Expected all tensors to be on the same devicemodel on GPU, batch on CPU (or the reverse)xb.to(device) inside the loop; model.to(device) once
mat1 and mat2 shapes cannot be multiplied (64x400 and 256x10)wrong in_features after a reshape or convprint x.shape before the layer; fix nn.Linear(400, …)
expected scalar type Long but found Floatclass labels stored as floatsy.long()
CUDA out of memorybatch too big, or keeping graphs alive (losses.append(loss))smaller batch; loss.item(); mixed precision; torch.no_grad() in evaluation
Trying to backward through the graph a second timereusing a graph across stepsrecompute the forward pass each step; .detach() values you keep
(no error) loss stays flat or validation numbers change between runsmissing zero_grad, wrong learning rate, or forgot model.eval()see Debugging Training
Try it

Hands-on: PyTorch in Colab (notebook 13)

Tensors, autograd, a Dataset and DataLoader, an MLP, a training loop with validation curves, and saving and reloading, with exercises at the end. Copy the code into a new Colab notebook or upload the .ipynb.

Interactive: this part runs in your browser.

Try it

Cheat sheet: PyTorch

The commands you will use most, on one screen. Come back to it while you work.

Interactive: this part runs in your browser.

Check yourself

Quiz

1. Why call opt.zero_grad() every step?

2. What does model.eval() change?

3. Your labels are float tensors and cross_entropy fails. Fix?

4. What should you save to reload a model later?

5. Layers stored in a plain Python list inside a Module are…

Go deeper

📚 Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready