Machine Learning Fundamentals for Engineers · Better Models and Production Concerns
Shipping Models: Validation, Versioning and Drift
Package a trained pipeline with metadata, validate prediction inputs, and detect data drift with statistical tests after deployment.
What you'll build
The production side of a model: save the fitted pipeline with metadata (version, training date, feature list, metrics), load it back, validate incoming requests against the expected schema, and compare live feature distributions with training data to detect drift.
The idea
- Ship the whole pipeline (preprocessing + model), with metadata so you know exactly what's running.
- Save with
joblib/pickle(only load files you trust: unpickling runs code) or theskopsformat; pin library versions, since pickles may not load across scikit-learn versions. - Validate inputs at the API boundary: required features, types and plausible ranges.
- Monitor: prediction volume and distribution, feature drift (for example a Kolmogorov-Smirnov test per feature), and, when labels arrive, real performance. Retrain on a schedule or when drift is detected.
The code
import io
import pickle
import numpy as np
import sklearn
from scipy.stats import ks_2samp
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = make_classification(n_samples=2000, n_features=4, n_informative=3, n_redundant=0, random_state=4)
features = ["tenure", "tickets", "usage", "seats"]
pipe = make_pipeline(StandardScaler(), LogisticRegression()).fit(X, y)
artifact = {"model": pipe, "version": "churn-2026.01.15", "features": features,
"sklearn": sklearn.__version__, "train_means": X.mean(axis=0).round(3).tolist()}
buffer = io.BytesIO()
pickle.dump(artifact, buffer)
loaded = pickle.loads(buffer.getvalue()) # in production: a file you built and trust
print("loaded", loaded["version"], "trained with scikit-learn", loaded["sklearn"])
def predict(payload: dict) -> float:
missing = [f for f in loaded["features"] if f not in payload]
if missing:
raise ValueError(f"missing features: {missing}")
row = np.array([[float(payload[f]) for f in loaded["features"]]])
return round(float(loaded["model"].predict_proba(row)[0, 1]), 3)
print(predict({"tenure": 0.5, "tickets": -1.0, "usage": 0.2, "seats": 1.1}))
try:
predict({"tenure": 0.5, "usage": 0.2})
except ValueError as e:
print("rejected:", e)
rng = np.random.default_rng(4)
live = X[:500].copy()
live[:, 2] += 0.8 # "usage" shifted after a product change
for i, name in enumerate(features):
p = ks_2samp(X[:, i], live[:, i]).pvalue
print(f"{name:<7} drift p-value {p:.4f}{' <- drift' if p < 0.01 else ''}")
Output
loaded churn-2026.01.15 trained with scikit-learn 1.8.0 0.498 rejected: missing features: ['tickets', 'seats'] tenure drift p-value 0.7527 tickets drift p-value 0.4886 usage drift p-value 0.0000 <- drift seats drift p-value 0.5690
Common mistakes
- Shipping only the model and re-implementing preprocessing in the API (training/serving skew).
- Loading pickles from untrusted sources; they can execute arbitrary code.
- Monitoring only uptime and latency, not input drift and prediction quality.
- Retraining automatically without evaluating the new model against the current one.
Key points
- Deploy the full pipeline with version metadata and pinned dependencies.
- Validate inputs before predicting.
- Monitor drift and real performance, and retrain deliberately.
Further reading: Model persistence · Kolmogorov-Smirnov test (SciPy)
Practice questions
Why deploy the whole pipeline rather than just the model?
Answer
Preprocessing must be identical at training and serving time; re-implementing it causes training/serving skew.
What's the security risk of pickle files?
Answer
Unpickling can execute arbitrary code, so only load model files you built and trust (or use a safer format like skops).
What is data drift and how can you detect it?
Answer
A change in the distribution of input features over time; compare live and training distributions per feature, for example with a Kolmogorov-Smirnov test.
Open in the app to mark lessons complete, take notes and track your progress.