Sequence Models: RNNs, LSTMs & GRUs
Networks with memory: recurrent networks, backpropagation through time, vanishing gradients, LSTM and GRU gates, sequence tasks, padding and masking, and when RNNs still make sense.
A network with memory
Text, speech, sensor streams and patient vital signs are sequences: order matters, and lengths vary. A recurrent neural network (RNN) reads one element at a time and keeps a hidden state h that summarises everything seen so far:
The same weights are used at every time step (weight sharing over time, as a CNN shares over space), so an RNN handles any length. To train it, the network is unrolled over the sequence into a deep feed-forward graph with one layer per time step, and ordinary backpropagation runs through it: backpropagation through time (BPTT). For long sequences BPTT is usually truncated to the last k steps.
Watch it: Inside an LSTM cell: forget, write, reveal
A short narrated animation. Press ▶ Play, or step through it with ⏮ ⏭. Tick 🔊 narrate to hear each step.
Interactive: this part runs in your browser.
The vanishing-gradient problem through time
The gradient that connects the loss at step T to an input at step t passes through copies of the recurrent step, each multiplying it by roughly times the tanh derivative (at most 1). Over 50 or 100 steps that product shrinks exponentially when the factors are below 1 (the network cannot learn long-range dependencies), or explodes when they are above 1 (loss spikes; clipping helps). In practice plain RNNs remember only around 10 to 20 steps.
Try it: how far back can the gradient reach?
Plot the size of the gradient reaching earlier time steps for a plain RNN with different recurrent weight scales, and for an LSTM whose forget gate stays open.
Interactive: this part runs in your browser.
LSTM and GRU: learning what to remember
The LSTM (long short-term memory, 1997) adds a cell state c, a conveyor belt that information can travel along with only small, controlled changes. Three gates, each a sigmoid between 0 and 1, decide what happens to it:
where each (·) is a linear function of . The forget gate f decides what to keep, the input gate i what to write, the output gate o what to reveal. Because depends on through a multiplication by f (close to 1 when remembering) and an addition, gradients can flow across hundreds of steps: the same idea as a residual connection. A common trick is to initialise the forget-gate bias to 1 so the network starts by remembering.
The GRU merges the gates into two (update and reset) and has no separate cell state: fewer parameters and often similar accuracy.
Sequence tasks, padding and masking
- Many-to-one: read a whole sequence, predict one label from the last hidden state (or a pooled one): sentiment, "will this patient deteriorate in the next 6 hours?".
- Many-to-many, aligned: one output per step: tagging each word, detecting events in a signal.
- Sequence-to-sequence: an encoder reads the input and a decoder generates a different-length output: translation, summarisation. Decoders are trained with teacher forcing (feeding the true previous token).
- Bidirectional RNNs read the sequence both ways: useful when the whole sequence is available (not for real-time forecasting).
Batches need equal lengths, so shorter sequences are padded with zeros. The padding must then be masked: excluded from the loss and from pooling, and skipped by the RNN (PyTorch's pack_padded_sequence). Forgetting the mask makes the model learn from padding and is a classic bug.
Where RNNs stand today
For text, transformers (next topic) have replaced RNNs almost everywhere: they train in parallel over the whole sequence and handle long-range dependencies directly. RNNs and their relatives remain useful for streaming data that must be processed one step at a time with constant memory (sensors, on-device audio), small datasets, and as strong baselines for time series, along with 1-D CNNs and temporal convolutional networks. Recent state-space models (S4, Mamba) bring back the recurrent idea with transformer-level quality on long sequences.
Code example: an RNN and an LSTM cell
A worked example: read it step by step (every part is explained), then run it and experiment.
Interactive: this part runs in your browser.
Code: an LSTM classifier in PyTorch
Many-to-one classification of variable-length sequences, with packing so padding is ignored.
import torch, torch.nn as nn
from torch.nn.utils.rnn import pack_padded_sequence
class SeqClassifier(nn.Module):
def __init__(self, d_in=4, d_h=64, n_classes=2):
super().__init__()
self.lstm = nn.LSTM(d_in, d_h, num_layers=2, batch_first=True, dropout=0.2)
self.head = nn.Linear(d_h, n_classes)
def forward(self, x, lengths): # x: (B, T_max, d_in), lengths: (B,)
packed = pack_padded_sequence(x, lengths.cpu(), batch_first=True, enforce_sorted=False)
_, (h, c) = self.lstm(packed) # h: (num_layers, B, d_h), last REAL step
return self.head(h[-1]) # (B, n_classes)
m = SeqClassifier()
x = torch.randn(3, 48, 4); lengths = torch.tensor([48, 30, 12])
print(m(x, lengths).shape)Quiz
1. What does the hidden state of an RNN represent?
2. Why do plain RNNs struggle with long-range dependencies?
3. In an LSTM, what does the forget gate control?
4. Why initialise the LSTM forget-gate bias to about 1?
5. You pad sequences to length 100 but forget to mask the padding in a mean-pooled loss. What happens?
6. A bedside monitor must update a deterioration score every second with constant memory. A good fit is…
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- Understanding LSTM Networks (Olah) ↗ The classic visual explanation
- Dive into Deep Learning (free book with code) ↗ Interactive textbook in PyTorch, by Zhang, Lipton, Li & Smola
- Deep Learning (Goodfellow, Bengio & Courville) ↗ The classic textbook, free online