Machine Learning
Topic 47 · Tools & libraries · ⏱ about 4 h

Gradient Boosting Libraries: XGBoost & LightGBM

Why boosted trees usually win on table data, how XGBoost, LightGBM and scikit-learn compare, the parameters that matter, early stopping, categorical features, and explaining the model.

Show:
In this topic
Beginner

Why boosted trees

On table data (rows and columns like a spreadsheet), gradient-boosted decision trees are usually the strongest model with the least tuning: they handle mixed feature types, missing values, non-linear effects and interactions, and need no feature scaling. The idea, adding small trees that each fix the previous ones' errors, is in Ensembles. This topic is about the libraries.

Always start with a simple baseline (for example logistic regression) so you know how much the boosted model actually adds.

Beginner

Three libraries, one idea

scikit-learn HistGradientBoostingXGBoostLightGBM
Number of treesmax_itern_estimatorsn_estimators
Step sizelearning_ratelearning_rate (eta)learning_rate
Tree sizemax_leaf_nodes, max_depthmax_depth (grows level by level)num_leaves (grows leaf by leaf)
Minimum data per leafmin_samples_leafmin_child_weightmin_child_samples
Row / column sampling(not built in)subsample, colsample_bytreesubsample (+ subsample_freq), colsample_bytree
Regularisationl2_regularizationreg_lambda, reg_alpha, gammareg_lambda, reg_alpha
Categorical featurescategorical_featuresenable_categorical=True + pandas categorypandas category columns
Early stoppingearly_stopping=Trueearly_stopping_rounds + eval_setcallbacks=[lgb.early_stopping(50)] + eval_set

All three use fast histogram-based splitting and handle missing values natively. Results are usually close; LightGBM is often fastest on large data, XGBoost is very widely used, and scikit-learn's version needs no extra install (and runs in this browser).

Beginner

The parameters that matter

  1. Learning rate and number of trees go together: a smaller learning rate needs more trees but usually generalises a little better. Fix a learning rate (0.03–0.1) and let early stopping on validation data choose the number of trees.
  2. Tree size (num_leaves 15–127, or max_depth 3–8): bigger trees capture more interactions and overfit sooner.
  3. Minimum data per leaf and sampling (subsample, colsample_bytree around 0.7–0.9) regularise.

Tune a small grid or a random search on validation data (or cross-validation), then refit on train + validation and evaluate on the test set once. Bigger searches rarely add much; better features usually do.

Try it

Try it: learning rate, tree size and early stopping

Watch training and validation loss as trees are added. Find where early stopping would stop, and how the learning rate and tree size change that point.

Interactive: this part runs in your browser.

Try it

Code example: boosting with early stopping, tuned properly

A worked example: read it step by step (every part is explained), then run it and experiment.

Interactive: this part runs in your browser.

Beginner

Explaining a boosted model

  • Built-in importance: "split" importance counts how often a feature is used and is biased toward features with many values; "gain" importance is better but still global and can mislead with correlated features.
  • Permutation importance (sklearn.inspection.permutation_importance) on validation data: how much the score drops when a feature is shuffled.
  • SHAP values (shap.TreeExplainer) explain each prediction as contributions from each feature; summary plots show global patterns. Notebook 15 shows both.
  • Importance shows what the model uses, not what causes the outcome.
Try it

Hands-on: XGBoost and LightGBM in Colab (notebook 15)

The Adult income dataset: a baseline, the three libraries side by side with early stopping, a tuning search, a final test, feature importance and SHAP.

Interactive: this part runs in your browser.

Try it

Cheat sheet: XGBoost & LightGBM

The commands you will use most, on one screen. Come back to it while you work.

Interactive: this part runs in your browser.

Check yourself

Quiz

1. On a table of customer records with mixed feature types, a strong default model is…

2. Best way to choose the number of trees?

3. LightGBM's main tree-size parameter is…

4. Do boosted trees need standardised features?

5. High SHAP importance for a feature means…

Go deeper

📚 Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready