How LLMs Work: Tokens, Sampling & Training
Tokens and context windows, how a model picks the next token (temperature and top-p), the three stages of training (pretraining, instruction tuning and preference tuning), reasoning models, and why models hallucinate.
Text becomes tokens
An LLM does not read letters or whole words. Text is split into tokens: common words are one token, rarer words are split into pieces ("tokenisation" might become "token" + "isation"). Each token is a number, and the model works with those numbers. The NLP course explains how tokenisers are built.
- A rough rule for English: one token ≈ 4 characters ≈ ¾ of a word, so 1,000 words is about 1,300 tokens. Many other languages, including Nepali and Hindi, use more tokens for the same meaning, which makes them slower and more expensive.
- The context window is the most tokens the model can look at in one go: your instructions, the conversation so far, any documents you include, and the answer it writes. Today's models range from a few thousand tokens for small local models to hundreds of thousands or more for large hosted ones.
- Providers charge per token, separately for input and output, so tokens are also your unit of cost.
Predicting the next token, again and again
Given the tokens so far, the model outputs a score (logit) for every token in its vocabulary, often 50,000 to 200,000 of them. A softmax turns the scores into probabilities (see information theory). The app then picks one token, adds it to the text, and asks again. Generating a 300-token answer means running the model 300 times; this loop is called autoregressive generation, and it is why answers appear word by word when they are streamed.
How the next token is picked is called decoding or sampling:
- Greedy: always take the most likely token. Predictable, but can be dull and repetitive.
- Temperature T divides the scores before the softmax. T below 1 sharpens the distribution (safer, more repeatable); T above 1 flattens it (more varied, more mistakes). T close to 0 is almost greedy.
- Top-p (nucleus sampling) keeps only the most likely tokens whose probabilities add up to p (say 0.9) and samples among them, cutting off the long tail of odd choices.
For extraction, classification and code, use a low temperature. For brainstorming and creative writing, a higher one. Even at temperature 0, outputs are not always exactly identical between calls, so never rely on that for correctness.
Try it: temperature and top-p
The model has read "Take this medicine with a glass of …". Move the sliders and press Sample 20 times to see how temperature and top-p change which words come out.
Interactive: this part runs in your browser.
How a model becomes an assistant: three stages of training
- Pretraining. Next-token prediction on trillions of tokens of web pages, books, code and more. The result, a base model, knows a lot about language and the world but simply continues text: ask it a question and it may reply with more questions.
- Instruction tuning (supervised fine-tuning, SFT). Further training on many examples of instructions and good answers, so it learns to respond helpfully in a chat format.
- Preference tuning. People (and increasingly other models) compare pairs of answers; the model is trained to prefer the better ones, with methods such as RLHF (reinforcement learning from human feedback) or DPO. This shapes helpfulness, honesty, tone and refusing harmful requests.
Reasoning models add a further stage: reinforcement learning on problems with checkable answers (maths, code), which teaches the model to write out a long chain of thought before answering. They are much better at multi-step problems, but slower and use more tokens. The NLP course covers these stages in depth, and Transformers from Scratch shows the network inside.
Why models hallucinate
The model is trained to produce likely-sounding text, not to check facts. When it does not know something, the most likely-sounding continuation is often a confident, plausible answer, not "I don't know". Its knowledge also stops at a training cutoff date, and it has no memory between conversations unless your app supplies it.
The practical defences, all covered in this course: give the model the facts it needs in the prompt (RAG), let it use tools for lookups and arithmetic (tool use), ask it to cite sources and to say when it is unsure, and measure how often it is wrong.
Quiz
1. Roughly how many tokens is 750 English words?
2. You are extracting dates from invoices. Which temperature makes most sense?
3. What does top-p = 0.9 do?
4. Which training stage turns a base model that just continues text into a helpful chat assistant?
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- The Illustrated Transformer (Jay Alammar) ↗ a famous visual explanation of the architecture inside LLMs
- Hugging Face LLM Course ↗ Free course on transformers and LLMs