Machine Learning
Topic 3 · ML foundations · ⏱ about 3 h

Data, Features & Splits

Preparing data properly — where most real ML effort goes.

Show:
In this topic
Beginner

Train, validation and test

The most important rule in machine learning is that you must never judge a model using the same data it learned from. A sufficiently flexible model can simply memorise its training examples and score perfectly on them while having learned nothing that transfers. To get an honest picture, we split the data into separate parts with separate jobs.

Random / stratified splittrain 70%: learn parametersval 15%test 15%Time-series split: never train on the futurepastlaterlatesttime
Train on one part, tune on another, and test once on data the model has never seen. For time series, the split must follow time.

The training set, typically about 70% of the data, is what the model learns its parameters from. The validation set, perhaps 15%, is used while you are still developing: to compare different models, to tune hyperparameters, and to decide when to stop training. The test set, the remaining 15% or so, is put aside and looked at only once, at the very end, to report how well the final model is likely to do in the real world. If you keep going back to the test set to make decisions, it gradually stops being an honest test, because your choices have been tuned to it. When data are scarce, cross-validation can replace the fixed validation set.

How you split matters as much as whether you split. For classification problems, use a stratified split, which keeps the proportion of each class the same in every part. Otherwise, with a rare class, the test set might by chance contain almost none of it. For time series, such as daily sales or stock prices, never split randomly. A random split would let the model train on data from next month and be tested on data from last month, which is impossible in reality. Instead, train on the past and test on the future, exactly as the model will be used.

Interactive: this part runs in your browser.

Interactive. Split a dataset with a rare class into train, validation and test, with and without stratification.
Beginner

Watch it: Data leakage: when the future sneaks into training

A short narrated animation. Press ▶ Play, or step through it with ⏮ ⏭. Tick 🔊 narrate to hear each step.

Interactive: this part runs in your browser.

Beginner

Feature engineering & preprocessing

Raw data are rarely ready to be fed into a model, and careful preparation often improves results more than switching to a cleverer algorithm. The clearest way to understand preprocessing is to follow one small table through every step and watch exactly what happens to it. Our table describes customers: their age, income, city, the date they joined and how much they owe.

Each step changes the table in one of four ways, and the pictures use the same colours throughout. A value can be changed (orange, with the old value struck through above it), a gap can be filled in (blue), a new column can be added (green), and a row or column can be removed (red, struck through).

Step 1: remove rows that cannot be used

Before repairing anything, delete rows that are simply wrong or useless. An exact duplicate would count the same customer twice and give that person double influence. A row with almost every value missing carries almost no information. Rows with a single gap are worth keeping, because we can repair them in the next step. This is the only step that changes the number of rows.

Raw: 7 rows
nameageincomecity
Ava3452,000Brisbane
Ben4161,000Sydney
ChenNaN48,000Brisbane
Dia29950,000Melbourne
Eli3857,000Sydney
Ben4161,000Sydney
GusNaNNaNNaN
drop rows→
After: 5 rows
nameageincomecity
Ava3452,000Brisbane
Ben4161,000Sydney
ChenNaN48,000Brisbane
Dia29950,000Melbourne
Eli3857,000Sydney
Row 6 is an exact copy of row 2 (Ben), so it is removed. Gus has 3 of his 4 values missing, so there is almost nothing to learn from that row and it is removed too. Chen is kept: only one value is missing, and we can fill it in.
removed
Removing rows. Exact duplicates and nearly empty rows are deleted, which changes the number of rows from 7 to 5. Rows with just one gap are kept and repaired in the next step.

Step 2: fill in missing values

Most algorithms cannot handle an empty cell, so we either remove the row or fill the gap, which is called imputation. Simple imputation uses the mean, median or most common value of the column. The median is usually the safer choice, because one extreme value can pull the mean a long way. It is often worth adding a yes/no column that records which values were filled in, because the fact that something was missing can itself be informative. For example, people who leave "income" blank may differ from people who fill it in.

Before
nameage
Ava34
Ben41
ChenNaN
Dia29
Eli38
median imputation
+ missing flag
→
After
nameageage_was_missing
Ava340
Ben410
Chen361
Dia290
Eli380
Median of the known ages: sort them → 29, 34, 38, 41. With an even count, take the middle two: (34 + 38) / 2 = 36. Chen gets 36. The new column age_was_missing keeps a record of which value was guessed, so the model can still use that information.
missing value filled innew column or value added
Imputation fills the gap in place, and a new yes/no column is added beside it. The table goes from 1 feature column to 2.

Step 3: deal with outliers and skew

An outlier is a value far from the rest. Before changing it, find out why it exists. It might be a typing error (an extra zero), or it might be real and important, such as a fraudulent transaction. If it is real but too dominant, two common treatments are a log transform, which compresses large values much more than small ones, and clipping, which caps values at a chosen limit.

Before
nameincome
Ava52,000
Ben61,000
Chen48,000
Dia950,000
Eli57,000
log transform
ln(x)
→
After
nameincomelog_income
Ava52,00010.86
Ben61,00011.02
Chen48,00010.78
Dia950,00013.76
Eli57,00010.95
ln(52 000) = 10.86, ln(61 000) = 11.02, … , ln(950 000) = 13.76.
Before: Dia's income is about 17× the median (950 000 / 57 000). After: it is only 2.8 units above the median (13.76 − 10.95). Dia is still the largest, so the order is unchanged, but she no longer dominates the column.
Before
nameincome
Ava52,000
Ben61,000
Chen48,000
Dia950,000
Eli57,000
clip (cap) at
100 000
→
After
nameincome
Ava52,000
Ben61,000
Chen48,000
Dia950,000100,000
Eli57,000
Clipping changes only the values above the cap: every value becomes min(x, 100 000). The four ordinary incomes are untouched. This is useful when an extreme value is believable but you do not want it to dominate.
value changed (old value struck through above)
Two ways to tame an outlier. A log transform changes every value and compresses the long tail, and the column is renamed to say so. Clipping changes only the extreme value.

Step 4: turn categories into numbers

Models work with numbers, so text categories must be converted. For categories with no natural order, such as city, use one-hot encoding: one new 0/1 column per category. Writing Brisbane = 0, Melbourne = 1, Sydney = 2 would be a mistake, because it tells the model that Sydney is "twice" Melbourne and that Melbourne lies "between" the other two. For categories that do have an order, such as clothing sizes, ordinal encoding keeps that order in a single column.

Before
namecity
AvaBrisbane
BenSydney
ChenBrisbane
DiaMelbourne
EliSydney
one-hot encode→
After
namecity_Brisbanecity_Melbournecity_Sydney
Ava100
Ben001
Chen100
Dia010
Eli001
Find the distinct values: Brisbane, Melbourne, Sydney (3 categories). Make one new column for each. In every row, put 1 in the column that matches the city and 0 in the others. Then delete the original text column. 1 column becomes 3.
Before
itemsize
shirt 1S
shirt 2L
shirt 3M
shirt 4S
ordinal encode
S→0, M→1, L→2
→
After
itemsize
shirt 1S0
shirt 2L2
shirt 3M1
shirt 4S0
Sizes have a real order, so a single number column is enough and keeps that order: S < M < L becomes 0 < 1 < 2. The values are replaced in place and no columns are added.
removednew column or value addedvalue changed (old value struck through above)
Encoding categories. One-hot encoding removes the text column and adds one 0/1 column per category. Ordinal encoding replaces each label with its position in the order.

Step 5: create new features

This is often the most valuable step, and it is where knowledge of the problem matters most. You build new columns that express something the model would struggle to work out by itself. A date string becomes the day of the week and a weekend flag. Two money columns become a ratio, which may matter far more than either amount on its own.

Before
namejoined
Ava2024-03-02
Ben2024-03-05
Chen2024-03-09
Dia2024-03-12
Eli2024-03-14
extract date
parts
→
After
nameday_of_weekis_weekend
Ava51
Ben10
Chen51
Dia10
Eli30
A raw date string means nothing to a model. Count the days Monday = 0 … Sunday = 6. 2 March 2024 was a Saturday → day_of_week 5, and Saturday is a weekend day → is_weekend 1. The original date column is then dropped.
Before
nameincomedebt
Ava52,00013,000
Ben61,00030,500
Chen48,0009,600
Dia950,00047,500
Eli57,0005,700
ratio feature
debt ÷ income
→
After
nameincomedebtdebt_to_income
Ava52,00013,0000.25
Ben61,00030,5000.50
Chen48,0009,6000.20
Dia950,00047,5000.05
Eli57,0005,7000.10
Ava: 13 000 ÷ 52 000 = 0.25. Ben: 30 500 ÷ 61 000 = 0.50. Ben owes the most relative to what he earns, which is what a lender cares about. Neither column showed this on its own: Dia has the largest debt but the smallest ratio.
removednew column or value added
Creating features. One date column is replaced by two useful ones. Two existing columns are combined into a ratio that captures what matters, while the originals are kept.

Step 6: put features on the same scale

Our columns now have very different ranges: debt is in the thousands, while the flags are 0 or 1. Algorithms that measure distances or use gradients, such as k-nearest neighbours, SVMs, PCA, neural networks and regularised models, would treat the large-valued columns as more important simply because their numbers are bigger. Scaling fixes this, and it is done last so that it also applies to the features created in step 5. Tree-based models compare values only within one column, so they do not need scaling.

Before
nameage
Ava34
Ben41
Chen36
Dia29
Eli38
standardise
z = (x − μ) / σ
→
After
nameage
Ava34-0.40
Ben411.34
Chen360.10
Dia29-1.64
Eli380.60
Mean μ = (34 + 41 + 36 + 29 + 38) / 5 = 35.6. Standard deviation σ = √[((34−35.6)² + … + (38−35.6)²) / 5] = √16.24 = 4.03.
Ava: (34 − 35.6) / 4.03 = −0.40 (a little below average). Ben: (41 − 35.6) / 4.03 = 1.34 (1.34 standard deviations above).
Before
nameage
Ava34
Ben41
Chen36
Dia29
Eli38
min–max
(x − min) / (max − min)
→
After
nameage
Ava340.42
Ben411.00
Chen360.58
Dia290.00
Eli380.75
min = 29, max = 41, so the range is 12. Ava: (34 − 29) / 12 = 0.42. The youngest (Dia) becomes exactly 0 and the oldest (Ben) exactly 1.
value changed (old value struck through above)
Scaling changes every value in a column, but never the order of the rows or the number of columns. Standardisation centres the column on 0. Min–max squeezes it into [0, 1].

The result

Putting it all together, the raw table of mixed text, dates and numbers has become a table of purely numeric, comparably scaled features. Some columns were changed, some were removed and several new ones were created. One rule applies to every step: any number the step learns from the data (the median used to fill gaps, the categories found for one-hot encoding, the mean and standard deviation used for scaling) must be computed from the training set only, and then reused unchanged on the validation and test sets. The next section explains why.

Raw data7 rows × 5 feature columns
→
Model-ready5 rows × 10 feature columns
Original columnWhat happenedResulting columnsChange
nameidentifier, not a feature (kept only as a label)name0
agemissing value filled with the median, flag added, then standardisedageage_was_missing+1
incomelog transform to tame the outlierlog_income0
cityone-hot encoded, original removedcitycity_Brisbanecity_Melbournecity_Sydney+2
joineddate parts extracted, original removedjoinedday_of_weekis_weekend+1
debtkept, and combined with income into a ratiodebtdebt_to_income+1
value changed (old value struck through above)new column or value addedremoved
The whole journey of one table. Two rows and two columns were removed, one column was transformed and renamed, and seven new columns were created (5 − 2 + 7 = 10), so the model sees 10 numeric features instead of 5 mixed ones.
Try it

Try it: transform a table one step at a time

Press Next step to apply one preprocessing step at a time to the customer table. Each view shows what the step removed, changed, filled or added, with the exact calculation underneath. Change the choices at the top (how to fill gaps, how to treat the outlier, which scaler) and watch the later steps change too.

Interactive: this part runs in your browser.

Beginner

Data leakage — the silent killer

Data leakage is one of the most common and damaging mistakes in machine learning. It happens when information that would not be available at the moment of making a real prediction somehow finds its way into training. The model then appears to perform brilliantly during testing, but fails as soon as it is used in the real world, and because the test results looked so good, the problem often goes unnoticed until it is expensive.

✗ Leaky: preprocessing sees the test dataall datafit scaler on ALLsplitoptimistic score✓ Correct: split first, fit on train onlyall datasplitfit scaler on TRAINtest: apply onlyhonest score
Leakage: if preprocessing is fitted before splitting, information about the test set sneaks into training.

The first classic form of leakage is preprocessing before splitting. Suppose you standardise a feature by computing its mean and standard deviation over the whole dataset, and only then split into training and test sets. The training data have now been scaled using statistics that partly come from the test set, so a little information about the test set has leaked into training. The correct procedure is to split first, compute any statistics (means, standard deviations, imputation values) from the training set only, and then apply those same numbers to the validation and test sets.

The second form is using features that are only known after the outcome. Imagine predicting whether a transaction is fraudulent using a column that says whether a refund was issued. Refunds are usually issued because fraud was detected, so this column almost gives away the answer during training, but at the moment a new transaction arrives, no refund has happened yet. Every feature should be checked against the question: "would I actually know this at prediction time?"

The third form is overlapping examples: duplicate records, or several rows belonging to the same patient or customer, appearing in both the training and test sets. The model can then score well by recognising the individual rather than by learning a general pattern. Group-aware splitting, which keeps all of one person's records on the same side of the split, prevents this.

A simple habit avoids the first problem almost automatically. In scikit-learn, put every preprocessing step inside a Pipeline together with the model. When you then run cross-validation, the library refits the whole pipeline, preprocessing included, separately on each training fold, so information from the validation fold can never leak in.

Intermediate · Maths

The maths of scaling and encoding

📐 Maths you'll need: m-prob · m-eigen
💡 The idea in plain words

If one feature is in millimetres and another in kilometres, the loss surface becomes a long, narrow valley, and gradient descent wastes its time bouncing between the steep walls. Scaling puts features on comparable ranges, which makes the valley round and training fast.

🔤 Symbols in this section
μ, σ\mu,\ \sigmamean and standard deviation of a feature (from training data)
IQR\text{IQR}interquartile range: 75th minus 25th percentile
HHHessian: matrix of second derivatives (curvature)
κ\kappacondition number = largest ÷ smallest eigenvalue
eke_kone-hot vector with a 1 in position k

This section looks at the mathematics behind preprocessing: what the common scaling formulas do, why scaling makes gradient descent so much faster, and a subtle trap with one-hot encoding.

The four most common transformations each serve a slightly different purpose. Standardisation, z=x−μσz = \frac{x - \mu}{\sigma}, gives each feature a mean of 0 and a standard deviation of 1, where μ\mu and σ\sigma must be computed on the training data. Min–max scaling, x′=x−xmin⁡xmax⁡−xmin⁡x' = \frac{x - x_{\min}}{x_{\max} - x_{\min}}, maps the training range onto [0, 1], but a single extreme value can squash all the other values into a tiny interval. Robust scaling, x′=x−median⁡IQR⁡x' = \frac{x - \operatorname{median}}{\operatorname{IQR}}, uses the median and the interquartile range (the spread of the middle half of the data), which are hardly affected by outliers. And a log transform, x′=log⁡(1+x)x' = \log(1 + x), compresses long right-hand tails such as incomes or counts, so that a few huge values no longer dominate.

Why does scaling make gradient descent faster? For the squared-error loss of a linear model, the loss surface is a quadratic bowl whose shape is described by its Hessian matrix, H=2nX⊤XH = \frac{2}{n}X^\top X. The eigenvalues of HH give the curvature of the bowl in different directions, and the ratio of the largest to the smallest is the condition number κ\kappa. If one feature is measured in millimetres and another in kilometres, the curvature in the first direction is enormous compared with the second, κ\kappa is huge, and the bowl becomes a long, narrow valley. Gradient descent must use a small step to avoid overshooting across the steep direction, which means it crawls along the flat direction, zig-zagging from wall to wall. The number of steps it needs grows roughly in proportion to κ\kappa. Standardising the features makes the curvature similar in every direction, so the bowl becomes rounder, κ\kappa drops, and gradient descent heads almost straight for the bottom. (The eigenvectors topic explains condition numbers in more detail.)

Finally, one-hot encoding has a subtle interaction with linear models. A category with KK possible values becomes KK yes/no columns, and in every row exactly one of them equals 1. That means the KK columns always add up to 1, which is exactly the same as the column of 1s that represents the intercept. The columns are therefore perfectly collinear, X⊤XX^\top X cannot be inverted, and the weights are not uniquely determined. This is known as the dummy-variable trap. The standard fix is to drop one of the KK columns, which becomes the reference category against which the others are compared. (Regularised models and tree-based models are not troubled by this, so they often keep all KK columns.)

Key takeaway. Scale features (using training statistics only) so the loss surface is well-conditioned; one-hot encode nominal categories and drop one column for linear models.
Advanced

Imbalanced data, missingness and target encoding

📐 Maths you'll need: m-prob
💡 The idea in plain words

Real data is messy: some classes are rare, some values are missing, some categories have thousands of levels. Each fix (re-weighting, smart imputation, target encoding) is simple, but each can quietly leak information or bias the model if applied at the wrong time.

🔤 Symbols in this section
wcw_cweight given to class c in the loss
ncn_cnumber of examples in class c
KKnumber of classes
u∼U(0,1)u\sim U(0,1)random number between 0 and 1
yˉc, yˉ\bar y_c,\ \bar ymean target in category c, overall mean target
mmsmoothing strength

Real datasets bring three further problems that are easy to handle badly: imbalanced classes, missing values with a pattern, and categorical features with very many levels.

Imbalanced classes. If only 1% of transactions are fraudulent, a model trained in the usual way can achieve 99% accuracy by never predicting fraud at all, because the loss is dominated by the majority class. One remedy is to give each class a weight in the loss so that the classes contribute equally overall:

L=−1n∑iwyi log⁡p(yi∣xi),wc=nK nc. \mathcal L = -\frac1n\sum_i w_{y_i}\,\log p(y_i\mid x_i), \qquad w_c = \frac{n}{K\, n_c}.

Here ncn_c is the number of examples in class cc and KK is the number of classes, so a rare class gets a large weight and a common class a small one. This is what scikit-learn does when you set class_weight="balanced". A second remedy is to change the training data itself, either by discarding some majority examples (under-sampling) or by adding more minority examples (over-sampling). The popular SMOTE method over-samples by creating synthetic minority examples: it picks a minority point xix_i, picks one of its nearest minority neighbours xnnx_{nn}, and creates a new point a random fraction uu of the way between them, xnew=xi+u (xnn−xi)x_{\text{new}} = x_i + u\,(x_{nn} - x_i). Whatever method you use, resample only the training data, inside each cross-validation fold; resampling before splitting would put copies of the same synthetic information on both sides of the split.

Why data are missing. The statistician Donald Rubin distinguished three situations, and the right treatment depends on which one you are in. If values are missing completely at random (MCAR), for example because a sensor occasionally fails for no reason, dropping incomplete rows doesn't bias the results; it only wastes data. If values are missing at random (MAR), meaning the chance of being missing depends on other features you do observe (older patients might be less likely to fill in an online form), then imputing missing values from those other features is appropriate. If values are missing not at random (MNAR), meaning the chance of being missing depends on the missing value itself (people with very high incomes may be the least likely to report their income), then the fact that the value is missing carries real information, and you should add an indicator column recording it.

High-cardinality categories. A feature like "postcode" may have thousands of distinct values, and one-hot encoding would create thousands of sparse columns. Target encoding instead replaces each category with a number summarising the target for that category, typically a smoothed average:

enc(c)=nc yˉc+m yˉnc+m. \text{enc}(c) = \frac{n_c\,\bar y_c + m\,\bar y}{n_c + m}.

Here yˉc\bar y_c is the average target among the ncn_c examples in category cc, yˉ\bar y is the overall average, and mm controls how strongly small categories are pulled towards the overall average. The smoothing matters because the average of a category with only two examples is very unreliable. Because target encoding uses the labels, it is a direct route to leakage: it must be computed using only the training fold (ideally out-of-fold), never using the rows it will be applied to.

Key takeaway. Fix imbalance with class weights or resampling, think about why data is missing, and compute anything that uses labels (target encoding, SMOTE) inside the training folds only.
Code

Code: a leak-free preprocessing pipeline

A ColumnTransformer applies different preprocessing to numeric and categorical columns. Because it sits inside the pipeline, cross-validation re-fits it on each training fold, so nothing leaks.

Python · pandas + scikit-learn
import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score

rng = np.random.default_rng(0)
n = 500
df = pd.DataFrame({
    "age":    rng.normal(40, 12, n),
    "income": rng.lognormal(10, 0.6, n),
    "city":   rng.choice(["Brisbane", "Sydney", "Melbourne"], n),
})
df.loc[rng.random(n) < 0.1, "income"] = np.nan            # 10% missing
y = (df["age"] + rng.normal(0, 10, n) > 45).astype(int)   # toy target

numeric = ["age", "income"]
categorical = ["city"]

preprocess = ColumnTransformer([
    ("num", Pipeline([
        ("impute", SimpleImputer(strategy="median", add_indicator=True)),
        ("scale",  StandardScaler()),
    ]), numeric),
    ("cat", OneHotEncoder(handle_unknown="ignore", drop="first"), categorical),
])

clf = Pipeline([("prep", preprocess),
                ("model", LogisticRegression(class_weight="balanced", max_iter=1000))])

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
scores = cross_val_score(clf, df, y, cv=cv, scoring="roc_auc")
print("ROC-AUC per fold:", scores.round(3), " mean:", scores.mean().round(3))
Open in playground ↗
Check yourself

Quiz

1. When should you fit a StandardScaler?

2. Which model does NOT need feature scaling?

3. Best way to split time-series data?

4. Features on very different scales make gradient descent slow because they increase the…

5. Income is missing mostly for high earners. This is…

Go deeper

📚 Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready