Machine Learning Fundamentals for Engineers · Better Models and Production Concerns
Decision Trees, Random Forests and Gradient Boosting
Model non-linear relationships with trees and ensembles, compare them with linear models, and explain predictions with permutation importance.
What you'll build
A comparison of a linear model, a single decision tree, a random forest and histogram gradient boosting on data with non-linear effects, followed by permutation importance to see which features drive the best model.
The idea
- Decision trees split data by feature thresholds; they capture non-linear effects and interactions but overfit easily.
- Random forests average many trees trained on random subsets, reducing variance.
- Gradient boosting (
HistGradientBoostingClassifier) adds trees that correct previous errors; usually the strongest choice for tabular data, and fast on large datasets. - Permutation importance measures how much the score drops when a feature's values are shuffled, on held-out data; more trustworthy than impurity-based importances.
The code
from sklearn.datasets import make_classification
from sklearn.ensemble import HistGradientBoostingClassifier, RandomForestClassifier
from sklearn.inspection import permutation_importance
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
X, y = make_classification(n_samples=3000, n_features=8, n_informative=4, n_redundant=0,
n_clusters_per_class=3, class_sep=0.8, random_state=11)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=11)
models = {
"logistic regression": LogisticRegression(max_iter=1000),
"decision tree (no limit)": DecisionTreeClassifier(random_state=11),
"random forest": RandomForestClassifier(n_estimators=200, random_state=11),
"gradient boosting": HistGradientBoostingClassifier(random_state=11),
}
for name, m in models.items():
m.fit(X_train, y_train)
print(f"{name:<25} train {m.score(X_train, y_train):.3f} test {m.score(X_test, y_test):.3f}")
best = models["gradient boosting"]
imp = permutation_importance(best, X_test, y_test, n_repeats=5, random_state=11)
ranked = sorted(zip(imp.importances_mean, range(X.shape[1])), reverse=True)
print("most important features:", [f"x{i} ({v:.3f})" for v, i in ranked[:4]])
Output
logistic regression train 0.589 test 0.583 decision tree (no limit) train 1.000 test 0.823 random forest train 1.000 test 0.857 gradient boosting train 0.999 test 0.853 most important features: ['x7 (0.254)', 'x5 (0.192)', 'x6 (0.150)', 'x1 (0.065)']
The unlimited tree scores perfectly on training data and worse on test data: overfitting. The ensembles generalise better.
Common mistakes
- Unlimited tree depth, memorising the training data.
- Trusting impurity-based feature importances, which favour high-cardinality features; use permutation importance on held-out data.
- Reaching for deep learning on small tabular datasets where gradient boosting is stronger and simpler.
Key points
- Trees capture non-linear effects; ensembles reduce overfitting.
- Gradient boosting is a strong default for tabular data.
- Explain models with permutation importance on test data.
Further reading: Ensembles · Permutation feature importance
Practice questions
Why does a random forest usually generalise better than a single deep tree?
Answer
Averaging many trees trained on random subsets of rows and features reduces variance, so it overfits less.
How does permutation importance work?
Answer
It shuffles one feature's values in held-out data and measures how much the model's score drops.
What's a strong default model for tabular data?
Answer
Gradient-boosted trees such as HistGradientBoostingClassifier/Regressor.
Open in the app to mark lessons complete, take notes and track your progress.