Data, Features & Splits
Preparing data properly — where most real ML effort goes.
Train, validation and test
The most important rule in machine learning is that you must never judge a model using the same data it learned from. A sufficiently flexible model can simply memorise its training examples and score perfectly on them while having learned nothing that transfers. To get an honest picture, we split the data into separate parts with separate jobs.
The training set, typically about 70% of the data, is what the model learns its parameters from. The validation set, perhaps 15%, is used while you are still developing: to compare different models, to tune hyperparameters, and to decide when to stop training. The test set, the remaining 15% or so, is put aside and looked at only once, at the very end, to report how well the final model is likely to do in the real world. If you keep going back to the test set to make decisions, it gradually stops being an honest test, because your choices have been tuned to it. When data are scarce, cross-validation can replace the fixed validation set.
How you split matters as much as whether you split. For classification problems, use a stratified split, which keeps the proportion of each class the same in every part. Otherwise, with a rare class, the test set might by chance contain almost none of it. For time series, such as daily sales or stock prices, never split randomly. A random split would let the model train on data from next month and be tested on data from last month, which is impossible in reality. Instead, train on the past and test on the future, exactly as the model will be used.
Interactive: this part runs in your browser.
Watch it: Data leakage: when the future sneaks into training
A short narrated animation. Press ▶ Play, or step through it with ⏮ ⏭. Tick 🔊 narrate to hear each step.
Interactive: this part runs in your browser.
Feature engineering & preprocessing
Raw data are rarely ready to be fed into a model, and careful preparation often improves results more than switching to a cleverer algorithm. The clearest way to understand preprocessing is to follow one small table through every step and watch exactly what happens to it. Our table describes customers: their age, income, city, the date they joined and how much they owe.
Each step changes the table in one of four ways, and the pictures use the same colours throughout. A value can be changed (orange, with the old value struck through above it), a gap can be filled in (blue), a new column can be added (green), and a row or column can be removed (red, struck through).
Step 1: remove rows that cannot be used
Before repairing anything, delete rows that are simply wrong or useless. An exact duplicate would count the same customer twice and give that person double influence. A row with almost every value missing carries almost no information. Rows with a single gap are worth keeping, because we can repair them in the next step. This is the only step that changes the number of rows.
| name | age | income | city |
|---|---|---|---|
| Ava | 34 | 52,000 | Brisbane |
| Ben | 41 | 61,000 | Sydney |
| Chen | NaN | 48,000 | Brisbane |
| Dia | 29 | 950,000 | Melbourne |
| Eli | 38 | 57,000 | Sydney |
| Ben | 41 | 61,000 | Sydney |
| Gus | NaN | NaN | NaN |
| name | age | income | city |
|---|---|---|---|
| Ava | 34 | 52,000 | Brisbane |
| Ben | 41 | 61,000 | Sydney |
| Chen | NaN | 48,000 | Brisbane |
| Dia | 29 | 950,000 | Melbourne |
| Eli | 38 | 57,000 | Sydney |
Step 2: fill in missing values
Most algorithms cannot handle an empty cell, so we either remove the row or fill the gap, which is called imputation. Simple imputation uses the mean, median or most common value of the column. The median is usually the safer choice, because one extreme value can pull the mean a long way. It is often worth adding a yes/no column that records which values were filled in, because the fact that something was missing can itself be informative. For example, people who leave "income" blank may differ from people who fill it in.
| name | age |
|---|---|
| Ava | 34 |
| Ben | 41 |
| Chen | NaN |
| Dia | 29 |
| Eli | 38 |
+ missing flag→
| name | age | age_was_missing |
|---|---|---|
| Ava | 34 | 0 |
| Ben | 41 | 0 |
| Chen | 36 | 1 |
| Dia | 29 | 0 |
| Eli | 38 | 0 |
Step 3: deal with outliers and skew
An outlier is a value far from the rest. Before changing it, find out why it exists. It might be a typing error (an extra zero), or it might be real and important, such as a fraudulent transaction. If it is real but too dominant, two common treatments are a log transform, which compresses large values much more than small ones, and clipping, which caps values at a chosen limit.
| name | income |
|---|---|
| Ava | 52,000 |
| Ben | 61,000 |
| Chen | 48,000 |
| Dia | 950,000 |
| Eli | 57,000 |
ln(x)→
| name | incomelog_income |
|---|---|
| Ava | 52,00010.86 |
| Ben | 61,00011.02 |
| Chen | 48,00010.78 |
| Dia | 950,00013.76 |
| Eli | 57,00010.95 |
Before: Dia's income is about 17× the median (950 000 / 57 000). After: it is only 2.8 units above the median (13.76 − 10.95). Dia is still the largest, so the order is unchanged, but she no longer dominates the column.
| name | income |
|---|---|
| Ava | 52,000 |
| Ben | 61,000 |
| Chen | 48,000 |
| Dia | 950,000 |
| Eli | 57,000 |
100 000→
| name | income |
|---|---|
| Ava | 52,000 |
| Ben | 61,000 |
| Chen | 48,000 |
| Dia | 950,000100,000 |
| Eli | 57,000 |
Step 4: turn categories into numbers
Models work with numbers, so text categories must be converted. For categories with no natural order, such as city, use one-hot encoding: one new 0/1 column per category. Writing Brisbane = 0, Melbourne = 1, Sydney = 2 would be a mistake, because it tells the model that Sydney is "twice" Melbourne and that Melbourne lies "between" the other two. For categories that do have an order, such as clothing sizes, ordinal encoding keeps that order in a single column.
| name | city |
|---|---|
| Ava | Brisbane |
| Ben | Sydney |
| Chen | Brisbane |
| Dia | Melbourne |
| Eli | Sydney |
| name | city_Brisbane | city_Melbourne | city_Sydney |
|---|---|---|---|
| Ava | 1 | 0 | 0 |
| Ben | 0 | 0 | 1 |
| Chen | 1 | 0 | 0 |
| Dia | 0 | 1 | 0 |
| Eli | 0 | 0 | 1 |
| item | size |
|---|---|
| shirt 1 | S |
| shirt 2 | L |
| shirt 3 | M |
| shirt 4 | S |
S→0, M→1, L→2→
| item | size |
|---|---|
| shirt 1 | S0 |
| shirt 2 | L2 |
| shirt 3 | M1 |
| shirt 4 | S0 |
Step 5: create new features
This is often the most valuable step, and it is where knowledge of the problem matters most. You build new columns that express something the model would struggle to work out by itself. A date string becomes the day of the week and a weekend flag. Two money columns become a ratio, which may matter far more than either amount on its own.
| name | joined |
|---|---|
| Ava | 2024-03-02 |
| Ben | 2024-03-05 |
| Chen | 2024-03-09 |
| Dia | 2024-03-12 |
| Eli | 2024-03-14 |
parts→
| name | day_of_week | is_weekend |
|---|---|---|
| Ava | 5 | 1 |
| Ben | 1 | 0 |
| Chen | 5 | 1 |
| Dia | 1 | 0 |
| Eli | 3 | 0 |
| name | income | debt |
|---|---|---|
| Ava | 52,000 | 13,000 |
| Ben | 61,000 | 30,500 |
| Chen | 48,000 | 9,600 |
| Dia | 950,000 | 47,500 |
| Eli | 57,000 | 5,700 |
debt ÷ income→
| name | income | debt | debt_to_income |
|---|---|---|---|
| Ava | 52,000 | 13,000 | 0.25 |
| Ben | 61,000 | 30,500 | 0.50 |
| Chen | 48,000 | 9,600 | 0.20 |
| Dia | 950,000 | 47,500 | 0.05 |
| Eli | 57,000 | 5,700 | 0.10 |
Step 6: put features on the same scale
Our columns now have very different ranges: debt is in the thousands, while the flags are 0 or 1. Algorithms that measure distances or use gradients, such as k-nearest neighbours, SVMs, PCA, neural networks and regularised models, would treat the large-valued columns as more important simply because their numbers are bigger. Scaling fixes this, and it is done last so that it also applies to the features created in step 5. Tree-based models compare values only within one column, so they do not need scaling.
| name | age |
|---|---|
| Ava | 34 |
| Ben | 41 |
| Chen | 36 |
| Dia | 29 |
| Eli | 38 |
z = (x − μ) / σ→
| name | age |
|---|---|
| Ava | 34-0.40 |
| Ben | 411.34 |
| Chen | 360.10 |
| Dia | 29-1.64 |
| Eli | 380.60 |
Ava: (34 − 35.6) / 4.03 = −0.40 (a little below average). Ben: (41 − 35.6) / 4.03 = 1.34 (1.34 standard deviations above).
| name | age |
|---|---|
| Ava | 34 |
| Ben | 41 |
| Chen | 36 |
| Dia | 29 |
| Eli | 38 |
(x − min) / (max − min)→
| name | age |
|---|---|
| Ava | 340.42 |
| Ben | 411.00 |
| Chen | 360.58 |
| Dia | 290.00 |
| Eli | 380.75 |
The result
Putting it all together, the raw table of mixed text, dates and numbers has become a table of purely numeric, comparably scaled features. Some columns were changed, some were removed and several new ones were created. One rule applies to every step: any number the step learns from the data (the median used to fill gaps, the categories found for one-hot encoding, the mean and standard deviation used for scaling) must be computed from the training set only, and then reused unchanged on the validation and test sets. The next section explains why.
| Original column | What happened | Resulting columns | Change |
|---|---|---|---|
| name | identifier, not a feature (kept only as a label) | name | 0 |
| age | missing value filled with the median, flag added, then standardised | ageage_was_missing | +1 |
| income | log transform to tame the outlier | log_income | 0 |
| city | one-hot encoded, original removed | citycity_Brisbanecity_Melbournecity_Sydney | +2 |
| joined | date parts extracted, original removed | joinedday_of_weekis_weekend | +1 |
| debt | kept, and combined with income into a ratio | debtdebt_to_income | +1 |
Try it: transform a table one step at a time
Press Next step to apply one preprocessing step at a time to the customer table. Each view shows what the step removed, changed, filled or added, with the exact calculation underneath. Change the choices at the top (how to fill gaps, how to treat the outlier, which scaler) and watch the later steps change too.
Interactive: this part runs in your browser.
Data leakage — the silent killer
Data leakage is one of the most common and damaging mistakes in machine learning. It happens when information that would not be available at the moment of making a real prediction somehow finds its way into training. The model then appears to perform brilliantly during testing, but fails as soon as it is used in the real world, and because the test results looked so good, the problem often goes unnoticed until it is expensive.
The first classic form of leakage is preprocessing before splitting. Suppose you standardise a feature by computing its mean and standard deviation over the whole dataset, and only then split into training and test sets. The training data have now been scaled using statistics that partly come from the test set, so a little information about the test set has leaked into training. The correct procedure is to split first, compute any statistics (means, standard deviations, imputation values) from the training set only, and then apply those same numbers to the validation and test sets.
The second form is using features that are only known after the outcome. Imagine predicting whether a transaction is fraudulent using a column that says whether a refund was issued. Refunds are usually issued because fraud was detected, so this column almost gives away the answer during training, but at the moment a new transaction arrives, no refund has happened yet. Every feature should be checked against the question: "would I actually know this at prediction time?"
The third form is overlapping examples: duplicate records, or several rows belonging to the same patient or customer, appearing in both the training and test sets. The model can then score well by recognising the individual rather than by learning a general pattern. Group-aware splitting, which keeps all of one person's records on the same side of the split, prevents this.
A simple habit avoids the first problem almost automatically. In scikit-learn, put every preprocessing step inside a Pipeline together with the model. When you then run cross-validation, the library refits the whole pipeline, preprocessing included, separately on each training fold, so information from the validation fold can never leak in.
The maths of scaling and encoding
If one feature is in millimetres and another in kilometres, the loss surface becomes a long, narrow valley, and gradient descent wastes its time bouncing between the steep walls. Scaling puts features on comparable ranges, which makes the valley round and training fast.
🔤 Symbols in this section
| mean and standard deviation of a feature (from training data) | |
| interquartile range: 75th minus 25th percentile | |
| Hessian: matrix of second derivatives (curvature) | |
| condition number = largest ÷ smallest eigenvalue | |
| one-hot vector with a 1 in position k |
This section looks at the mathematics behind preprocessing: what the common scaling formulas do, why scaling makes gradient descent so much faster, and a subtle trap with one-hot encoding.
The four most common transformations each serve a slightly different purpose. Standardisation, , gives each feature a mean of 0 and a standard deviation of 1, where and must be computed on the training data. Min–max scaling, , maps the training range onto [0, 1], but a single extreme value can squash all the other values into a tiny interval. Robust scaling, , uses the median and the interquartile range (the spread of the middle half of the data), which are hardly affected by outliers. And a log transform, , compresses long right-hand tails such as incomes or counts, so that a few huge values no longer dominate.
Why does scaling make gradient descent faster? For the squared-error loss of a linear model, the loss surface is a quadratic bowl whose shape is described by its Hessian matrix, . The eigenvalues of give the curvature of the bowl in different directions, and the ratio of the largest to the smallest is the condition number . If one feature is measured in millimetres and another in kilometres, the curvature in the first direction is enormous compared with the second, is huge, and the bowl becomes a long, narrow valley. Gradient descent must use a small step to avoid overshooting across the steep direction, which means it crawls along the flat direction, zig-zagging from wall to wall. The number of steps it needs grows roughly in proportion to . Standardising the features makes the curvature similar in every direction, so the bowl becomes rounder, drops, and gradient descent heads almost straight for the bottom. (The eigenvectors topic explains condition numbers in more detail.)
Finally, one-hot encoding has a subtle interaction with linear models. A category with possible values becomes yes/no columns, and in every row exactly one of them equals 1. That means the columns always add up to 1, which is exactly the same as the column of 1s that represents the intercept. The columns are therefore perfectly collinear, cannot be inverted, and the weights are not uniquely determined. This is known as the dummy-variable trap. The standard fix is to drop one of the columns, which becomes the reference category against which the others are compared. (Regularised models and tree-based models are not troubled by this, so they often keep all columns.)
Imbalanced data, missingness and target encoding
Real data is messy: some classes are rare, some values are missing, some categories have thousands of levels. Each fix (re-weighting, smart imputation, target encoding) is simple, but each can quietly leak information or bias the model if applied at the wrong time.
🔤 Symbols in this section
| weight given to class c in the loss | |
| number of examples in class c | |
| number of classes | |
| random number between 0 and 1 | |
| mean target in category c, overall mean target | |
| smoothing strength |
Real datasets bring three further problems that are easy to handle badly: imbalanced classes, missing values with a pattern, and categorical features with very many levels.
Imbalanced classes. If only 1% of transactions are fraudulent, a model trained in the usual way can achieve 99% accuracy by never predicting fraud at all, because the loss is dominated by the majority class. One remedy is to give each class a weight in the loss so that the classes contribute equally overall:
Here is the number of examples in class and is the number of classes, so a rare class gets a large weight and a common class a small one. This is what scikit-learn does when you set class_weight="balanced". A second remedy is to change the training data itself, either by discarding some majority examples (under-sampling) or by adding more minority examples (over-sampling). The popular SMOTE method over-samples by creating synthetic minority examples: it picks a minority point , picks one of its nearest minority neighbours , and creates a new point a random fraction of the way between them, . Whatever method you use, resample only the training data, inside each cross-validation fold; resampling before splitting would put copies of the same synthetic information on both sides of the split.
Why data are missing. The statistician Donald Rubin distinguished three situations, and the right treatment depends on which one you are in. If values are missing completely at random (MCAR), for example because a sensor occasionally fails for no reason, dropping incomplete rows doesn't bias the results; it only wastes data. If values are missing at random (MAR), meaning the chance of being missing depends on other features you do observe (older patients might be less likely to fill in an online form), then imputing missing values from those other features is appropriate. If values are missing not at random (MNAR), meaning the chance of being missing depends on the missing value itself (people with very high incomes may be the least likely to report their income), then the fact that the value is missing carries real information, and you should add an indicator column recording it.
High-cardinality categories. A feature like "postcode" may have thousands of distinct values, and one-hot encoding would create thousands of sparse columns. Target encoding instead replaces each category with a number summarising the target for that category, typically a smoothed average:
Here is the average target among the examples in category , is the overall average, and controls how strongly small categories are pulled towards the overall average. The smoothing matters because the average of a category with only two examples is very unreliable. Because target encoding uses the labels, it is a direct route to leakage: it must be computed using only the training fold (ideally out-of-fold), never using the rows it will be applied to.
Code: a leak-free preprocessing pipeline
A ColumnTransformer applies different preprocessing to numeric and categorical columns. Because it sits inside the pipeline, cross-validation re-fits it on each training fold, so nothing leaks.
import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
rng = np.random.default_rng(0)
n = 500
df = pd.DataFrame({
"age": rng.normal(40, 12, n),
"income": rng.lognormal(10, 0.6, n),
"city": rng.choice(["Brisbane", "Sydney", "Melbourne"], n),
})
df.loc[rng.random(n) < 0.1, "income"] = np.nan # 10% missing
y = (df["age"] + rng.normal(0, 10, n) > 45).astype(int) # toy target
numeric = ["age", "income"]
categorical = ["city"]
preprocess = ColumnTransformer([
("num", Pipeline([
("impute", SimpleImputer(strategy="median", add_indicator=True)),
("scale", StandardScaler()),
]), numeric),
("cat", OneHotEncoder(handle_unknown="ignore", drop="first"), categorical),
])
clf = Pipeline([("prep", preprocess),
("model", LogisticRegression(class_weight="balanced", max_iter=1000))])
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
scores = cross_val_score(clf, df, y, cv=cv, scoring="roc_auc")
print("ROC-AUC per fold:", scores.round(3), " mean:", scores.mean().round(3))Quiz
1. When should you fit a StandardScaler?
2. Which model does NOT need feature scaling?
3. Best way to split time-series data?
4. Features on very different scales make gradient descent slow because they increase the…
5. Income is missing mostly for high earners. This is…
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- scikit-learn: Common pitfalls ↗ Leakage and other mistakes
- Google Machine Learning Crash Course ↗ Free course with exercises