Deep Learning
Topic 13 ยท Deep Learning ยท โฑ about 10 h

Transformers from Scratch

Scaled dot-product attention, masks, multi-head attention, positional encodings, the transformer block, encoder vs decoder models, and the cost of attention.

Show:
In this topic
Beginner

Attention: every token looks at every other

An RNN passes information along the sequence one step at a time. Attention lets every position look directly at every other position in a single step. Each token produces three vectors with learned linear layers: a query q ("what am I looking for?"), a key k ("what do I contain?") and a value v ("what will I pass on?"). A token's output is a weighted average of all values, weighted by how well its query matches each key:

Attention(Q,K,V)=softmaxโกโ€‰โฃ(QKโŠคdk+M)V\text{Attention}(Q,K,V)=\operatorname{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}+M\right)V

with Q, K, V of shape (T, d_k) for a sequence of T tokens. QKโŠคQK^\top is the (T ร— T) table of match scores; softmax turns each row into weights that sum to 1. Dividing by dk\sqrt{d_k} keeps the scores from growing with the vector size: without it, softmax saturates into a near one-hot, and gradients vanish.

M is a mask added before the softmax: โˆ’โˆž-\infty where attention is forbidden, so those weights become exactly 0. A causal mask hides future tokens (for models that generate text left to right); a padding mask hides padding.

Beginner

Watch it: Data flowing through a transformer block

A short narrated animation. Press โ–ถ Play, or step through it with โฎ โญ. Tick ๐Ÿ”Š narrate to hear each step.

Interactive: this part runs in your browser.

Try it

Try it: an attention matrix

Each row is one token's query; each column a key. Toggle the causal mask and the โˆšd scaling, and compare two heads that have learned different relationships.

Interactive: this part runs in your browser.

Beginner

Multi-head attention and positions

One attention pattern can only express one kind of relationship at a time. Multi-head attention runs h attention operations in parallel, each on its own d/h-dimensional slice (its own learned Q, K, V projections), then concatenates the results and mixes them with an output projection. Different heads learn different relationships: the previous token, the subject of the verb, a matching bracket.

Attention itself ignores order: shuffle the tokens and each output just moves with its token. So position information is added. The original transformer adds fixed sinusoidal positional encodings to the embeddings: PE(pos,2i)=sinโก(pos/100002i/d)PE_{(pos,2i)}=\sin(pos/10000^{2i/d}), PE(pos,2i+1)=cosโก(โ‹…)PE_{(pos,2i+1)}=\cos(\cdot). Many models learn a position embedding instead, and most recent language models use rotary embeddings (RoPE), which rotate q and k by position-dependent angles so the score depends on the relative distance.

Beginner

The transformer block

A transformer is a stack of identical blocks. The modern ("pre-norm") block, for an input X of shape (B, T, d):

X = X + MultiHeadAttention(LayerNorm(X)) # tokens exchange information X = X + MLP(LayerNorm(X)) # each token is processed on its own # MLP: Linear(d, 4d) โ†’ GELU โ†’ Linear(4d, d)

Two residual connections carry the signal (and gradients) through every block; LayerNorm keeps it in range. Attention is the only place where tokens talk to each other; the MLP, which holds about two thirds of the parameters, works on each token separately. Input tokens are first turned into vectors by an embedding table; a final linear layer maps each vector to logits over the vocabulary or to class scores.

Beginner

Encoders, decoders and how they are trained

TypeAttentionPre-training objectiveExamples ยท good at
Encoder-onlyBidirectional (sees all tokens)Masked language modelling: predict hidden wordsBERT, RoBERTa ยท classification, search embeddings, tagging
Decoder-onlyCausal (left to right)Next-token predictionGPT, Llama, Claude ยท generation, chat, almost everything at scale
Encoder-decoderEncoder bidirectional; decoder causal + cross-attention to the encoderSpan corruption / translationT5, original transformer ยท translation, summarisation

The same block also works on images split into patches (Vision Transformers) and on audio. The LLM topic covers how decoder models are instruction-tuned and used; the NLP track goes further.

Beginner

The cost of attention

The score matrix is T ร— T per head, so attention's compute and memory grow with the square of the sequence length: doubling the context makes attention four times more expensive. Three engineering ideas matter in practice:

  • FlashAttention computes exact attention in tiles without storing the full T ร— T matrix in GPU memory: much faster, and the default in PyTorch's scaled_dot_product_attention.
  • When generating, a decoder reuses the keys and values of earlier tokens: the KV cache. It makes each new token cheap but uses a lot of memory for long contexts.
  • Sparse, sliding-window and linear attention variants trade some quality for lower cost on very long inputs.
Try it

Code example: attention with masks and heads

A worked example: read it step by step (every part is explained), then run it and experiment.

Interactive: this part runs in your browser.

Try it

Code example: a whole transformer block

A worked example: read it step by step (every part is explained), then run it and experiment.

Interactive: this part runs in your browser.

Code

Code: a transformer block in PyTorch

Using PyTorch's fused attention (FlashAttention when available).

Python ยท PyTorch
import torch, torch.nn as nn, torch.nn.functional as F

class Block(nn.Module):
    def __init__(self, d=256, h=8, p=0.1):
        super().__init__()
        self.h, self.ln1, self.ln2 = h, nn.LayerNorm(d), nn.LayerNorm(d)
        self.qkv, self.proj = nn.Linear(d, 3 * d), nn.Linear(d, d)
        self.mlp = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d), nn.Dropout(p))
    def forward(self, x, causal=True):                     # x: (B, T, d)
        B, T, d = x.shape
        q, k, v = self.qkv(self.ln1(x)).split(d, dim=-1)
        q, k, v = (t.view(B, T, self.h, d // self.h).transpose(1, 2) for t in (q, k, v))  # (B, h, T, d/h)
        a = F.scaled_dot_product_attention(q, k, v, is_causal=causal)
        x = x + self.proj(a.transpose(1, 2).reshape(B, T, d))
        return x + self.mlp(self.ln2(x))

print(Block()(torch.randn(2, 16, 256)).shape)            # torch.Size([2, 16, 256])
This code needs PyTorch, which cannot run inside a web page. To run it, press Copy, then paste it into a new notebook in Google Colab โ†— (free; Runtime โ†’ Change runtime type โ†’ GPU for deep learning).
Check yourself

Quiz

1. Why divide attention scores by โˆšd_k?

2. What does a causal mask do?

3. Why do transformers need positional information?

4. In a transformer block, where do tokens exchange information?

5. Which model type is BERT?

6. Doubling the context length makes standard attentionโ€ฆ

Go deeper

๐Ÿ“š Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready