Machine Learning Fundamentals for Engineers · Better Models and Production Concerns
Overfitting and Data Leakage
Recognise overfitting from the gap between training and validation scores, and catch leakage from features that wouldn't exist at prediction time.
What you'll build
Two experiments: tree depth versus training and validation scores (the classic overfitting curve), and a model with a "too good to be true" score caused by a leaked feature, found and fixed.
The idea
- Overfitting: training score keeps rising while validation score falls. Fix with simpler models, regularisation, more data or early stopping.
- Leakage: a feature contains information about the target that won't be available when predicting (for example "refund issued" when predicting "will complain", or values computed after the event). Leaked models look excellent offline and fail in production.
- Ask of every feature: would I know this value at the moment I make the prediction?
The code
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, validation_curve
from sklearn.tree import DecisionTreeClassifier
from sklearn.linear_model import LogisticRegression
X, y = make_classification(n_samples=1500, n_features=10, n_informative=4, flip_y=0.1, random_state=5)
train_scores, val_scores = validation_curve(DecisionTreeClassifier(random_state=5), X, y,
param_name="max_depth", param_range=[1, 3, 5, 10, 20], cv=5)
for depth, tr, va in zip([1, 3, 5, 10, 20], train_scores.mean(1), val_scores.mean(1)):
print(f"depth {depth:>2}: train {tr:.3f} validation {va:.3f}")
rng = np.random.default_rng(5)
leaked = y + rng.normal(0, 0.1, len(y)) # e.g. "cancellation reason recorded", known only after churn
X_leak = np.column_stack([X, leaked])
X_tr, X_te, y_tr, y_te = train_test_split(X_leak, y, random_state=5)
with_leak = LogisticRegression(max_iter=1000).fit(X_tr, y_tr).score(X_te, y_te)
without = LogisticRegression(max_iter=1000).fit(X_tr[:, :-1], y_tr).score(X_te[:, :-1], y_te)
print(f"with leaked feature: {with_leak:.3f} (suspicious) | without: {without:.3f} (realistic)")
Output
depth 1: train 0.804 validation 0.798 depth 3: train 0.841 validation 0.822 depth 5: train 0.880 validation 0.819 depth 10: train 0.963 validation 0.807 depth 20: train 0.999 validation 0.781 with leaked feature: 1.000 (suspicious) | without: 0.829 (realistic)
Common mistakes
- Celebrating a near-perfect score instead of hunting for leakage.
- Random splits for data with time order or repeated entities (same customer in train and test); use time-based or group splits (
GroupKFold). - Tuning until validation scores rise by noise-level amounts.
Key points
- A growing train-validation gap means overfitting.
- Every feature must be available at prediction time.
- Split by time or group when rows aren't independent.
Further reading: Underfitting vs overfitting · Common pitfalls
Practice questions
How do you recognise overfitting?
Answer
Training performance is much better than validation performance, and the gap grows as model complexity increases.
What is target leakage?
Answer
Using features that contain information about the target that won't be available at prediction time, which inflates offline scores.
When should you use GroupKFold?
Answer
When several rows belong to the same entity (customer, patient), so all of an entity's rows stay in the same fold.
Open in the app to mark lessons complete, take notes and track your progress.