Generative AI & LLM Apps
Topic 11 · Making it reliable · ⏱ about 2 h

Multimodal Models & Choosing an Approach

Models that read images and documents, transcribe and speak audio, and generate images and video; then the big design decision for any project: prompting, RAG, fine-tuning, or not using an LLM at all.

Show:
In this topic
Beginner

Beyond text

  • Vision-language models accept images alongside text: read a receipt, describe a chart, check a form, answer questions about a photo or a scanned PDF. They build on ideas from Vision Transformers and CLIP.
  • Speech: speech-to-text models (for example the open Whisper family) transcribe audio in many languages; text-to-speech models read answers aloud in natural voices; and some models handle live spoken conversation directly.
  • Image and video generation: mostly diffusion models that start from random noise and remove it step by step, guided by your text prompt (see Generative Models).

The same app-building skills apply: prompts, structured output, evaluation and safety. Check image inputs for prompt injection too: text inside an image can carry instructions.

Beginner

Prompting, RAG or fine-tuning?

ApproachUse it whenCost and effort
Prompting (with examples)always try first; general tasks, format and toneminutes; no training
RAGanswers depend on knowledge the model lacks: your documents, recent factsdays; needs an index kept up to date
Fine-tuningyou need a consistent behaviour or style that prompting cannot reach, or a small cheap model to match a big one on a narrow task, and you have hundreds to thousands of good examplesweeks; needs data, evaluation and retraining when things change
Not an LLMa rule, a SQL query or a classic ML model does the job reliablyoften the cheapest and most accurate

Fine-tuning teaches behaviour, not reliably new facts; for facts, use RAG. The NLP course shows how fine-tuning works with LoRA.

Try it

Try it: which approach fits?

Choose the best starting approach for each project.

Interactive: this part runs in your browser.

Check yourself

Quiz

1. A chatbot must answer questions about a company's 2,000-page policy manual, which changes monthly. Best approach?

2. What does fine-tuning mainly change?

3. Most image-generation models today are…

Go deeper

📚 Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready