Language Models, Decoding & Evaluating Generated Text
What a language model computes, perplexity, decoding strategies (greedy, beam, temperature, top-k, top-p), summarisation and translation, and how to evaluate generated text: overlap metrics, embedding metrics, LLM judges and humans.
A language model predicts the next token
A language model assigns a probability to text by predicting each token from the ones before it: . n-gram models counted short histories; neural models (RNNs, then transformers) use the whole context. Decoder-only transformers trained on next-token prediction at scale are today's LLMs.
The standard intrinsic measure is perplexity: the exponential of the average negative log-probability per token on held-out text,
Lower is better; a perplexity of 20 means the model is, on average, as uncertain as choosing uniformly among 20 tokens. Perplexities are only comparable between models with the same tokenizer on the same text, and low perplexity does not guarantee useful or truthful output.
Watch it: Greedy decoding vs beam search
A short narrated animation. Press ▶ Play, or step through it with ⏮ ⏭. Tick 🔊 narrate to hear each step.
Interactive: this part runs in your browser.
Decoding: turning probabilities into text
At each step the model gives a probability for every token in the vocabulary. How you choose the next token changes the output a lot:
- Greedy: always the most likely token. Deterministic, but often repetitive and dull.
- Beam search: keep the k most likely partial sequences. Good for translation and short factual outputs; bland for open-ended text.
- Temperature T: divide logits by T before softmax. T < 1 sharpens (safer, less diverse), T > 1 flattens (more creative, more errors). T → 0 is greedy.
- Top-k: sample only from the k most likely tokens. Top-p (nucleus): sample from the smallest set whose probabilities add up to p (e.g. 0.9), which adapts to how confident the model is.
- Repetition penalties, stop sequences and maximum lengths control the rest.
For clinical summaries or structured extraction, use low temperature (or greedy) and constrained formats; for brainstorming, higher temperature and top-p.
Try it: decoding settings
A model is continuing "The patient should…". Change temperature, top-k and top-p and watch which tokens remain possible.
Interactive: this part runs in your browser.
Evaluating generated text
Generation is hard to evaluate because many different outputs can be correct. Options, from cheap to expensive:
- Overlap metrics compare with reference texts: BLEU (n-gram precision, translation), ROUGE (n-gram and longest-common-subsequence recall, summarisation). Cheap and reproducible, but they reward copying words and miss meaning: "no fever" and "fever" overlap heavily.
- Embedding metrics (BERTScore) compare meaning more than wording.
- Task-specific checks: does the summary contain every medication from the note (extraction recall)? Any facts not supported by the source (hallucinations)? Correct format?
- LLM-as-a-judge: a strong model scores outputs against a rubric. Scales well, but has biases (prefers longer answers, its own style, the first option shown); validate it against human ratings on a sample.
- Human evaluation by domain experts with a clear rubric (accuracy, completeness, safety, usefulness): the gold standard for anything clinical.
Build a fixed evaluation set that covers easy, hard and adversarial cases, version it, and run it on every change of model, prompt or settings.
Try it: when ROUGE disagrees with meaning
Compare candidate summaries of a clinic note with the reference. Which one would a clinician accept, and which one does ROUGE prefer?
Interactive: this part runs in your browser.
Code example: decoding strategies
A worked example: read it step by step (every part is explained), then run it and experiment.
Interactive: this part runs in your browser.
Code example: perplexity and ROUGE
A worked example: read it step by step (every part is explained), then run it and experiment.
Interactive: this part runs in your browser.
Code: generation settings with transformers
The same model, three decoding strategies.
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
name = "Qwen/Qwen2.5-0.5B-Instruct"; tok = AutoTokenizer.from_pretrained(name); model = AutoModelForCausalLM.from_pretrained(name)
msgs = [{"role": "user", "content": "In one sentence: why should patients finish a course of antibiotics?"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt") # the model's own chat template
for kw in [dict(do_sample=False), dict(num_beams=4, do_sample=False), dict(do_sample=True, temperature=0.9, top_p=0.9)]:
out = model.generate(ids, max_new_tokens=60, **kw)
print(kw, "→", tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))Quiz
1. Average negative log-probability per token is 2.0 (natural log). Perplexity is…
2. Which decoding setting suits extracting medications from a note?
3. Top-p = 0.9 samples from…
4. A summary says "fever" where the note said "no fever". ROUGE will…
5. LLM-as-a-judge scores should be…
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- The Curious Case of Neural Text Degeneration (Holtzman et al., 2019) ↗ Introduces nucleus (top-p) sampling
- ROUGE (Lin, 2004) ↗ The summarisation metric
- Speech and Language Processing (Jurafsky & Martin, 3rd ed. draft) ↗ The standard NLP textbook, free online