Machine Learning Fundamentals for Engineers · Capstone
Capstone: Predict Delivery Times for an E-commerce Site
Design an ML system that predicts delivery dates at checkout: problem framing, data, features, model choice, evaluation, deployment and monitoring.
Scenario
An online shop shows "arrives in 2-4 days" at checkout using a fixed table per country. Customers complain when parcels arrive late, and support costs grow. You're asked to predict delivery time per order from historical shipments: warehouse, carrier, destination postcode, weight, order time, weekday, holidays and the carrier's recent performance.
Requirements
- A prediction shown at checkout in under 100 ms.
- Late deliveries (actual later than promised) are much worse than conservative promises.
- Works for new postcodes and new carriers.
- The team must be able to explain predictions to customer support.
Constraints
- Three years of shipment history (about 5 million rows) in a data warehouse.
- One data scientist and two backend engineers; Python services on Kubernetes.
Deliverables
- Problem framing and metric.
- Data and features.
- Model and evaluation.
- Deployment.
- Monitoring and iteration.
Hints
- What should you predict: the average, or something else?
- Which features would leak information from after checkout?
Draft each deliverable below before revealing the model answer and rubric.
Further reading: scikit-learn user guide · Quantile regression with gradient boosting
Deliverables
How do you frame the problem and choose the metric?
Model answer
- Regression of delivery days, but the promise should be a high quantile (for example the 90th percentile), because late is worse than early: quantile regression or a calibrated prediction interval.
- Metrics: share of deliveries on time against the promise (target 90%+), and average promise length (shorter is better); MAE for the median model. Compare with the current fixed table as the baseline.
Rubric
Look for an asymmetric objective (quantiles or intervals), a business metric tied to promises, and a baseline.
Which data and features do you use, and what do you avoid?
Model answer
- Features known at checkout: warehouse, carrier, destination region/postcode prefix, distance, weight, order hour and weekday, holiday calendar, current warehouse backlog, carrier's on-time rate over the last 14 days.
- Avoid leakage: actual pickup time, tracking events and anything recorded after checkout.
- Handle new postcodes by falling back to coarser regions (prefixes); new carriers to carrier-type averages.
Rubric
Strong answers use only checkout-time features, include recent operational signals, and plan for unseen categories.
Which model and evaluation do you choose?
Model answer
- HistGradientBoostingRegressor with
loss="quantile"(alpha 0.9) and one at 0.5, inside a pipeline with categorical handling. - Time-based split: train on older data, validate on the most recent months (seasonality, carrier changes); never random splits.
- Evaluate coverage (share on time) and promise length by segment (country, carrier, peak season); permutation importance and per-prediction explanations for support.
Rubric
Look for quantile gradient boosting, time-based validation, segment-level evaluation and explainability.
How do you deploy it?
Model answer
- Batch-compute recent carrier statistics hourly into a feature store/table; the checkout service calls a small prediction service that loads the versioned pipeline at startup.
- Strict input validation, a fallback to the old table when the model or features are unavailable, and caching by (warehouse, region, carrier, weekday).
- Shadow mode first (log predictions, keep showing the old promise), then an A/B test on late-delivery rate and conversion.
Rubric
Strong answers meet latency with precomputed features, include fallbacks, and roll out through shadow mode and experiments.
How do you monitor and iterate?
Model answer
- Daily: on-time rate against promises (overall and per segment), prediction distribution, feature drift (KS tests on key features), missing-feature rates.
- Alerts when coverage drops below target, for example during carrier strikes or peaks; temporary manual buffers.
- Monthly retraining with evaluation against the current model before promotion; document versions and changes.
Rubric
Look for business-metric monitoring, drift checks, operational overrides and controlled retraining.
Open in the app to mark lessons complete, take notes and track your progress.