Thresholds, Costs and Calibrated Probabilities

What you'll build

A churn-prevention policy: the model outputs probabilities, a retention offer costs 30 and a lost customer costs 200, and you choose the probability threshold that minimises total cost instead of the default 0.5. Then you check the probabilities' calibration.

The idea

• predict uses a 0.5 threshold; predict_proba gives probabilities you can threshold yourself.
• The best threshold depends on costs: when missing a positive is expensive, lower the threshold. With perfectly calibrated probabilities, acting is worth it when p x 200 > 30, so around p = 0.15.
• Calibration: among customers predicted at 30%, about 30% should churn. Some models (trees, SVMs) are poorly calibrated; CalibratedClassifierCV fixes it.
• scikit-learn's TunedThresholdClassifierCV can tune the threshold with cross-validation.

The code

import numpy as np
from sklearn.calibration import calibration_curve
from sklearn.datasets import make_classification
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=6000, n_features=12, n_informative=4, weights=[0.85], flip_y=0.08,
                           class_sep=0.7, random_state=21)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, stratify=y, random_state=21)
model = HistGradientBoostingClassifier(random_state=21).fit(X_train, y_train)
proba = model.predict_proba(X_test)[:, 1]

OFFER_COST, LOST_CUSTOMER = 30, 200
def total_cost(threshold):
    act = proba >= threshold
    return OFFER_COST * act.sum() + LOST_CUSTOMER * ((~act) & (y_test == 1)).sum()

for t in [0.05, 0.1, 0.15, 0.2, 0.3, 0.5]:
    print(f"threshold {t:.2f}: cost {total_cost(t):>6}")
best = min(np.arange(0.05, 0.95, 0.05), key=total_cost)
print(f"best threshold {best:.2f}, cost {total_cost(best)} vs {total_cost(0.5)} at 0.5")

frac_pos, mean_pred = calibration_curve(y_test, proba, n_bins=5, strategy="quantile")
for p, f in zip(mean_pred, frac_pos):
    print(f"predicted {p:.2f} -> actual {f:.2f}")

threshold 0.05: cost  42860
threshold 0.10: cost  38670
threshold 0.15: cost  37930
threshold 0.20: cost  39020
threshold 0.30: cost  42630
threshold 0.50: cost  46870
best threshold 0.15, cost 37930 vs 46870 at 0.5
predicted 0.01 -> actual 0.05
predicted 0.02 -> actual 0.04
predicted 0.04 -> actual 0.06
predicted 0.09 -> actual 0.10
predicted 0.59 -> actual 0.65

Common mistakes

• Using 0.5 because it's the default, regardless of what errors cost.
• Treating scores from uncalibrated models as probabilities in business calculations.
• Choosing the threshold on the test set and then reporting test performance; tune it on validation data.

Key points

• Decisions need a threshold chosen from business costs.
• Check calibration before using probabilities as probabilities.
• Tune thresholds on validation data, not the test set.

Further reading: Tuning the decision threshold · Probability calibration