Images as Data, Preprocessing & Augmentation
Pixels, channels and colour spaces, resizing and normalisation, file formats including medical images, and data augmentation that keeps labels correct.
An image is an array
A greyscale image is a 2-D array of brightness values; a colour image adds a channel axis, usually red, green and blue. Files store 8-bit integers from 0 to 255; models want floats. Watch the layout: PIL and OpenCV give (H, W, C), PyTorch wants (C, H, W), and OpenCV loads colours as BGR, not RGB, a classic bug that silently swaps red and blue.
Other colour spaces are sometimes useful: HSV separates colour (hue) from brightness, which helps when lighting varies; LAB approximates human perception. Medical images are different again: X-rays and CT come as DICOM files with 12โ16-bit values and metadata (patient, scanner, pixel spacing); CT values are in Hounsfield units and are "windowed" (clipped to a range such as soft tissue or bone) before training. Microscopy may have many channels.
Resizing and normalisation
Networks expect a fixed input size (224 ร 224 is common for classification). Resizing choices matter: squashing changes aspect ratios (a round lesion becomes an oval); resize the short side then centre-crop, or letterbox (pad to a square), keeps shapes. Small objects can vanish when a large image is downscaled, so detection on high-resolution scans often uses tiles.
Normalise pixels the same way the pretrained model was trained: typically scale to [0, 1], then subtract the ImageNet mean (0.485, 0.456, 0.406) and divide by the std (0.229, 0.224, 0.225) per channel. Use exactly the same preprocessing at inference; a mismatch is one of the commonest production bugs in vision.
Data augmentation
Augmentation creates new training images with transformations that do not change the label. It is the most effective regulariser in vision. Common families:
- Geometric: random resized crops, flips, small rotations, scaling, perspective.
- Photometric: brightness, contrast, saturation, hue, blur, noise, JPEG compression.
- Occlusion and mixing: Cutout / random erasing, MixUp and CutMix (blend two images and their labels).
- Automatic policies: RandAugment, TrivialAugment (random choices from a list of transforms).
Two rules. First, transforms must be valid for your task: horizontal flips are fine for photos of clothes but swap left and right anatomy in a chest X-ray; colour jitter can destroy the signal in skin images where colour is diagnostic. Second, for detection and segmentation, the boxes and masks must be transformed together with the image (Albumentations does this). Augment training data only; validation and test use the deterministic resize and normalise.
Try it: augmentation playground
Apply augmentations to a small image and its bounding box. Notice how the box must move with the image, and which transforms would change the meaning of a medical image.
Interactive: this part runs in your browser.
Code example: augmentation that moves the boxes too
A worked example: read it step by step (every part is explained), then run it and experiment.
Interactive: this part runs in your browser.
Code: torchvision and Albumentations
Training and evaluation transforms for classification, and a box-aware pipeline for detection.
from torchvision import transforms as T
train_tf = T.Compose([T.RandomResizedCrop(224, scale=(0.7, 1.0)), T.RandomHorizontalFlip(),
T.ColorJitter(0.2, 0.2, 0.1), T.ToTensor(),
T.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])
eval_tf = T.Compose([T.Resize(256), T.CenterCrop(224), T.ToTensor(),
T.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])
import albumentations as A # boxes and masks move with the image
det_tf = A.Compose([A.RandomSizedBBoxSafeCrop(512, 512), A.HorizontalFlip(), A.RandomBrightnessContrast()],
bbox_params=A.BboxParams(format="pascal_voc", label_fields=["labels"]))
out = det_tf(image=img, bboxes=boxes, labels=labels)
import cv2
img = cv2.cvtColor(cv2.imread("scan.jpg"), cv2.COLOR_BGR2RGB) # OpenCV loads BGR!Quiz
1. You load an image with OpenCV and feed it to an ImageNet model without conversion. What is wrong?
2. Which augmentation is risky for chest X-rays?
3. For object detection, a random crop mustโฆ
4. At inference, images should be normalisedโฆ
5. Why is resizing a 4000 ร 3000 slide straight to 224 ร 224 a problem for detecting small cells?
๐ Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- Albumentations documentation โ Augmentation for classification, detection and segmentation
- CS231n: Deep Learning for Computer Vision (Stanford) โ Notes on backprop, CNNs and training