Tensors, Shapes & Broadcasting
The data structure of deep learning: shapes and layout conventions, broadcasting rules and their silent bugs, reshaping, einsum and numerical precision.
A tensor is an n-dimensional array with a shape
A tensor is what NumPy calls an ndarray: numbers arranged in a grid with any number of dimensions (also called axes). Its shape lists the size of each axis, its dtype says how each number is stored (float32, int64โฆ), and in PyTorch its device says where it lives (CPU or GPU).
Deep learning uses a few standard layouts. Learn them by heart; most bugs are shape bugs.
| Data | Typical shape | Meaning |
|---|---|---|
| Tabular batch | (B, F) | B examples, F features |
| Images (PyTorch) | (B, C, H, W) | batch, channels (3 for RGB), height, width ("NCHW"). TensorFlow uses (B, H, W, C). |
| Sequences / text | (B, T, D) | batch, time steps or tokens, features per step (embedding size) |
| Attention scores | (B, h, T, T) | batch, heads, queries, keys |
| Dense layer weights | (out, in) in PyTorch | nn.Linear(in, out) stores W as (out, in) and computes x Wแต + b |
The first axis is almost always the batch. Writing the expected shape as a comment next to each line (# (B, T, D)) is a habit used in many professional codebases.
Broadcasting: powerful, and the source of silent bugs
When two arrays of different shapes are combined (+ โ * /, comparisons), NumPy and PyTorch broadcast them. The rule: line the shapes up from the right; each pair of sizes must be equal, or one of them must be 1 (or missing). Size-1 axes are stretched to match.
The last line is the classic silent bug. y_pred has shape (N, 1) because a layer outputs one column, y has shape (N,), and ((y_pred - y) ** 2).mean() happily computes the mean over an N ร N matrix. The loss looks plausible, the model trains badly, and no error is raised. PyTorch even warns about this in MSELoss. The fix is to make shapes match explicitly (y_pred.squeeze(1) or y[:, None]) and to assert shapes in your code.
Try it: what shape comes out?
Pick two shapes. The demo lines them up from the right, explains each axis, and shows the result shape or the error you would get.
Interactive: this part runs in your browser.
Reshaping, transposing and adding axes
reshape (PyTorch also has view) changes the shape without changing the order of the numbers in memory: a (B, 28, 28) batch of images becomes (B, 784) to feed a dense layer, and -1 means "work this axis out". transpose / permute reorders axes: NHWC โ NCHW is x.permute(0, 3, 1, 2).
Confusing the two is a common bug: to split a (B, T, D) tensor into h heads you reshape to (B, T, h, D/h) and then transpose to (B, h, T, D/h). Reshaping straight to (B, h, T, D/h) mixes tokens and heads together without any error.
x[:, None] (NumPy) or x.unsqueeze(1) (PyTorch) adds a size-1 axis, and squeeze removes one; this is how you set up broadcasting deliberately. After a transpose, PyTorch tensors may be non-contiguous in memory: view then fails and reshape (or .contiguous().view) is needed.
Matrix products and einsum
A @ B multiplies the last two axes and broadcasts the rest, so a whole batch is processed at once: (B, T, D) @ (D, H) โ (B, T, H). einsum writes any product of axes as a small formula: each input gets letters for its axes, repeated letters are multiplied and summed over, and the output lists the letters you keep.
einsum makes the intended shapes explicit, which makes it easier to read and review. It is equally available in NumPy, PyTorch and JAX.
Numbers in a computer: dtypes and stability
Neural networks normally train in float32 (about 7 significant digits). Modern GPUs also use bfloat16 (same range as float32, fewer digits) and float16 (more digits than bfloat16, but overflows above 65,504) to halve memory and double speed; see Efficiency & scale. Integers (int64) hold class labels and token ids.
Some formulas are fine on paper and fail in floating point. np.exp(1000) overflows to inf, and np.log(0) is -inf. The cure is to rearrange the formula, for example with the log-sum-exp trick: subtract the maximum before exponentiating.
This is why frameworks combine softmax with the cross-entropy loss (CrossEntropyLoss takes raw logits) and sigmoid with binary cross-entropy (BCEWithLogitsLoss): the combined versions are computed stably.
Code example: shape-safe tensor code
A worked example: read it step by step (every part is explained), then run it and experiment.
Interactive: this part runs in your browser.
Vectorise: no Python loops over data
A Python loop over a million numbers takes about a second; the same operation as one NumPy or PyTorch call takes milliseconds, because the work happens in compiled code (and on a GPU, in parallel). So deep-learning code operates on whole batches with broadcasting, matrix products and reductions, and loops only over epochs, batches and layers. If you find yourself looping over examples or pixels, look for the tensor operation that does it at once. In-place operations (x += 1, PyTorch's add_) save memory but can break autograd if the original value was needed for the backward pass.
Quiz
1. A PyTorch batch of 32 RGB images of 224 ร 224 pixels has shapeโฆ
2. What is the shape of a + b with a: (8, 1, 5) and b: (3, 1)?
3. Predictions have shape (N, 1) and targets (N,). ((pred - y)**2).mean()โฆ
4. To split (B, T, D) into h heads of size D/h you shouldโฆ
5. np.einsum("bqd,bkd->bqk", Q, K) computesโฆ
6. Why does PyTorch's CrossEntropyLoss take logits rather than probabilities?
๐ Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- Dive into Deep Learning (free book with code) โ Interactive textbook in PyTorch, by Zhang, Lipton, Li & Smola
- NumPy: the absolute basics for beginners โ Official NumPy guide
- PyTorch tutorials โ Official tutorials