Cross-Validation and Hyperparameter Tuning

What you'll build

A tuned classifier: cross-validated scores with their spread, a grid search over regularisation strength, and a final evaluation on a test set that played no part in any decision.

The idea

• One train/test split gives a noisy estimate. k-fold cross-validation trains k models, each tested on a different fold, and reports the mean and spread.
• Hyperparameters (like C in logistic regression, or tree depth) are chosen by comparing cross-validated scores, never by test-set scores.
• GridSearchCV (or RandomizedSearchCV for big spaces) does both, refitting the best model on all training data.
• Keep a final test set for one last, honest estimate.

The code

from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV, StratifiedKFold, cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = make_classification(n_samples=600, n_features=20, n_informative=5, flip_y=0.05, random_state=3)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=3)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=3)

pipe = make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000))
scores = cross_val_score(pipe, X_train, y_train, cv=cv, scoring="accuracy")
print("accuracy per fold:", scores.round(3), "mean", round(scores.mean(), 3), "+/-", round(scores.std(), 3))

grid = GridSearchCV(pipe, {"logisticregression__C": [0.001, 0.01, 0.1, 1, 10]}, cv=cv, scoring="accuracy")
grid.fit(X_train, y_train)
for c, mean in zip(grid.cv_results_["param_logisticregression__C"], grid.cv_results_["mean_test_score"]):
    print(f"C={c:<5} mean accuracy {mean:.3f}")
print("best:", grid.best_params_, "| final test accuracy:", round(grid.score(X_test, y_test), 3))

accuracy per fold: [0.688 0.635 0.708 0.75  0.688] mean 0.694 +/- 0.037
C=0.001 mean accuracy 0.669
C=0.01  mean accuracy 0.683
C=0.1   mean accuracy 0.696
C=1.0   mean accuracy 0.694
C=10.0  mean accuracy 0.692
best: {'logisticregression__C': 0.1} | final test accuracy: 0.742

Common mistakes

• Tuning on the test set, then reporting the test score as if it were unbiased.
• Ignoring the spread across folds; a 0.01 difference between settings may be noise.
• Shuffled k-fold on time-ordered data; use TimeSeriesSplit so models never train on the future.
• Huge grids that take hours; start coarse, then refine, or use randomized search.

Key points

• Cross-validation gives a mean and a spread, not one lucky number.
• Choose hyperparameters with cross-validation on training data only.
• Keep the test set for one final estimate.

Further reading: Cross-validation · Tuning hyperparameters