Multimodal Models & Choosing an Approach
Models that read images and documents, transcribe and speak audio, and generate images and video; then the big design decision for any project: prompting, RAG, fine-tuning, or not using an LLM at all.
Beyond text
- Vision-language models accept images alongside text: read a receipt, describe a chart, check a form, answer questions about a photo or a scanned PDF. They build on ideas from Vision Transformers and CLIP.
- Speech: speech-to-text models (for example the open Whisper family) transcribe audio in many languages; text-to-speech models read answers aloud in natural voices; and some models handle live spoken conversation directly.
- Image and video generation: mostly diffusion models that start from random noise and remove it step by step, guided by your text prompt (see Generative Models).
The same app-building skills apply: prompts, structured output, evaluation and safety. Check image inputs for prompt injection too: text inside an image can carry instructions.
Prompting, RAG or fine-tuning?
| Approach | Use it when | Cost and effort |
|---|---|---|
| Prompting (with examples) | always try first; general tasks, format and tone | minutes; no training |
| RAG | answers depend on knowledge the model lacks: your documents, recent facts | days; needs an index kept up to date |
| Fine-tuning | you need a consistent behaviour or style that prompting cannot reach, or a small cheap model to match a big one on a narrow task, and you have hundreds to thousands of good examples | weeks; needs data, evaluation and retraining when things change |
| Not an LLM | a rule, a SQL query or a classic ML model does the job reliably | often the cheapest and most accurate |
Fine-tuning teaches behaviour, not reliably new facts; for facts, use RAG. The NLP course shows how fine-tuning works with LoRA.
Try it: which approach fits?
Choose the best starting approach for each project.
Interactive: this part runs in your browser.
Quiz
1. A chatbot must answer questions about a company's 2,000-page policy manual, which changes monthly. Best approach?
2. What does fine-tuning mainly change?
3. Most image-generation models today are…
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- Hugging Face LLM Course ↗ Free course on transformers and LLMs
- Anthropic: Building effective agents ↗ workflows vs agents and the common patterns, from real projects