Natural Language Processing
Topic 4 · NLP · ⏱ about 8 h

Text Classification & Sequence Labelling

Baselines first (TF-IDF + linear models), fine-tuning encoders, imbalanced and multi-label text, named-entity recognition with BIO tags and span-level F1, and when to use an LLM instead.

Show:
In this topic
Beginner

Always start with a strong baseline

TF-IDF + logistic regression is fast, cheap, interpretable and often surprisingly strong. TF-IDF represents a text by its words (and word pairs) weighted by how characteristic they are: frequent in this text, rare across the corpus. A linear model on top shows exactly which words drive each class. Train it in minutes, and only then ask whether a transformer is worth its cost.

Fine-tuning a pretrained encoder (DistilBERT, RoBERTa, DeBERTa, or a domain model such as a clinical BERT) adds a classification head on the [CLS] vector and trains the whole model with a small learning rate (2e-5 to 5e-5) for a few epochs. It handles negation, word order, typos and paraphrases far better, and usually wins when you have a few thousand labelled examples. LLMs with few-shot prompts need no training and are a quick option when labels are scarce, but are slower, costlier, can drift between model versions, and must be evaluated the same way.

Beginner

Metrics, imbalance and labels

  • Use macro-F1 (average F1 over classes) and per-class recall: accuracy hides failure on rare classes. For a safety-critical class ("urgent"), fix a recall target and report precision at it.
  • Imbalance: class weights, oversampling, and threshold tuning on validation data.
  • Multi-label: one sigmoid per label, a threshold per label.
  • Labels: write labelling guidelines with examples; measure agreement (Cohen's κ); review disagreements and confusing pairs. Noisy labels cap any model's accuracy.
  • Leakage: near-duplicate messages (templates, copy-paste) across train and test inflate scores; deduplicate first.
Beginner

Named-entity recognition with BIO tags

NER labels each token: is it part of a person's name, a medicine, a date? Entities can span several tokens, so tags use the BIO scheme: B-MED begins an entity, I-MED continues it, O is outside any entity.

tokens: can lucy patel pick up the metformin 500mg script on monday tags: O B-PER I-PER O O O B-MED I-MED O O B-DATE

A token classifier (an encoder with a linear layer per token) predicts tags; a decoding step turns tags into spans, fixing invalid sequences (an I- after O is treated as a new entity). NER is evaluated at the span level: a predicted entity counts only if its boundaries and type match exactly (seqeval's F1). Token accuracy is misleading because most tokens are O.

NER is central to de-identification of clinical text (find and redact names, addresses, dates), and to extracting structured data (medicines and doses) from notes.

Try it

Try it: what does the baseline rely on?

A TF-IDF + logistic regression message router. Type a message and see the predicted category and the words that pushed it there; try negation and paraphrases.

Interactive: this part runs in your browser.

Try it

Code example: a message router baseline

A worked example: read it step by step (every part is explained), then run it and experiment.

Interactive: this part runs in your browser.

Try it

Code example: BIO tags to spans, and span-level F1

A worked example: read it step by step (every part is explained), then run it and experiment.

Interactive: this part runs in your browser.

Code

Code: fine-tune a classifier with Hugging Face

The core of notebook 11.

Python · transformers
from transformers import AutoTokenizer, AutoModelForSequenceClassification, TrainingArguments, Trainer
import numpy as np, evaluate
name = "distilbert-base-uncased"; tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForSequenceClassification.from_pretrained(name, num_labels=5)
ds = ds.map(lambda b: tok(b["text"], truncation=True, max_length=128), batched=True)   # a datasets.DatasetDict
f1 = evaluate.load("f1")
def metrics(p): return f1.compute(predictions=np.argmax(p.predictions, -1), references=p.label_ids, average="macro")
args = TrainingArguments("out", learning_rate=3e-5, num_train_epochs=3, per_device_train_batch_size=32,
                         warmup_ratio=0.1, weight_decay=0.01, eval_strategy="epoch", save_strategy="epoch",
                         load_best_model_at_end=True, metric_for_best_model="f1")
Trainer(model=model, args=args, train_dataset=ds["train"], eval_dataset=ds["validation"], tokenizer=tok, compute_metrics=metrics).train()
This code needs Hugging Face transformers, which cannot run inside a web page. To run it, press Copy, then paste it into a new notebook in Google Colab ↗ (free; Runtime → Change runtime type → GPU for deep learning).
Check yourself

Quiz

1. Why train TF-IDF + logistic regression first?

2. Tags B-MED I-MED O B-PER describe…

3. A NER model predicts "metformin" as MED but misses "500mg" (true span: "metformin 500mg"). At the span level this counts as…

4. Your router has 94% accuracy but misses 30% of urgent messages. Which metric should have warned you?

5. Fine-tuning learning rate for BERT-style encoders is typically…

Go deeper

📚 Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready