Computer Vision
Topic 7 · Computer Vision · ⏱ about 7 h

Vision Transformers, Self-Supervision & CLIP

Images as sequences of patches, how ViTs compare with CNNs, self-supervised pretraining (DINO, masked autoencoders), and CLIP for zero-shot classification and search.

Show:
In this topic
Beginner

An image is worth 16 × 16 words

A Vision Transformer (ViT) cuts the image into fixed-size patches (e.g. 16 × 16 pixels: a 224 × 224 image gives 14 × 14 = 196 patches), flattens each patch and maps it with a linear layer to a vector: a "token". A learned position embedding is added, a special [CLS] token is prepended, and the sequence goes through ordinary transformer blocks. The [CLS] token's final vector (or the average of all tokens) feeds the classifier.

Compared with CNNs, ViTs have weaker built-in assumptions (no locality or weight sharing over space), so they need more data or strong pretraining, but they scale very well and attention can relate distant parts of the image from the first layer. In practice: with a good pretrained checkpoint, ViTs and modern CNNs (ConvNeXt) perform similarly; CNNs are often cheaper at high resolution and on small devices. Hybrid and hierarchical designs (Swin) bring locality back.

Beginner

Watch it: CLIP: pulling matching images and captions together

A short narrated animation. Press ▶ Play, or step through it with ⏮ ⏭. Tick 🔊 narrate to hear each step.

Interactive: this part runs in your browser.

Try it

Try it: patches and zero-shot labels

See how an image becomes a sequence of patch tokens, then classify images zero-shot by comparing an image embedding with text embeddings of label prompts, as CLIP does.

Interactive: this part runs in your browser.

Beginner

Self-supervised pretraining

Labels are the bottleneck in vision, especially in medicine. Self-supervised learning pretrains on unlabelled images with a task made from the data itself:

  • Contrastive (SimCLR, MoCo): two augmented views of the same image should have similar embeddings, and views of different images dissimilar ones.
  • Self-distillation (DINO, DINOv2): a student network matches a slowly-updated teacher's output on different views; its features are excellent for retrieval, clustering and linear probes.
  • Masked autoencoders (MAE): hide 75% of the patches and reconstruct them.

A hospital with 100,000 unlabelled scans and 1,000 labelled ones can pretrain (or continue pretraining) on the unlabelled set, then fine-tune on the labelled one: often a large gain over ImageNet initialisation alone.

Beginner

CLIP: images and text in one space

CLIP trains an image encoder and a text encoder together on hundreds of millions of image–caption pairs, so that each image's embedding is close (high cosine similarity) to its own caption's and far from the others. The result is a shared space for images and language, which enables:

  • Zero-shot classification: embed prompts such as "a photo of a {label}" for each class and pick the one most similar to the image, with no training.
  • Text-to-image search and image-to-image search over large collections.
  • Vision encoders for multimodal LLMs, and conditioning for image generators.

Zero-shot accuracy depends heavily on the prompt wording and on whether the domain was well covered in the web training data: it is often poor on specialised medical images. Measure it on your data, and fall back to a linear probe on CLIP embeddings with a few labelled examples, which is usually much better.

Try it

Code example: patchify and zero-shot with embeddings

A worked example: read it step by step (every part is explained), then run it and experiment.

Interactive: this part runs in your browser.

Code

Code: zero-shot classification with CLIP

Hugging Face transformers.

Python · transformers
from transformers import CLIPModel, CLIPProcessor
from PIL import Image
import torch
model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32"); proc = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
labels = ["a referral letter", "a pathology report", "a prescription", "a handwritten note"]
inputs = proc(text=[f"a scanned image of {l}" for l in labels], images=Image.open("scan.png"), return_tensors="pt", padding=True)
with torch.no_grad():
    probs = model(**inputs).logits_per_image.softmax(dim=-1)[0]
print({l: round(p.item(), 3) for l, p in zip(labels, probs)})
# better with labels: embed images with model.get_image_features(...) and train a logistic regression on them
This code needs Hugging Face transformers, which cannot run inside a web page. To run it, press Copy, then paste it into a new notebook in Google Colab ↗ (free; Runtime → Change runtime type → GPU for deep learning).
Check yourself

Quiz

1. A 224 × 224 image with 16 × 16 patches gives how many patch tokens?

2. Compared with CNNs, ViTs…

3. What does a masked autoencoder learn from?

4. How does CLIP classify zero-shot?

5. CLIP zero-shot accuracy on your dermatology photos is poor. Next step?

Go deeper

📚 Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready