Clustering and Anomaly Detection

What you'll build

Customer segments from spending behaviour without any labels, with the number of segments chosen by silhouette score, and an anomaly detector that flags unusual transactions for review.

The idea

• Unsupervised learning finds structure without labels.
• k-means groups points around k centres; scale features first, because it uses distances.
• Silhouette score (from -1 to 1) measures how well separated clusters are; compare several k.
• Isolation Forest isolates unusual points with random splits; anomalies need fewer splits. contamination sets the expected share of anomalies.
• Clusters and anomalies need human interpretation: describe each segment by its averages.

The code

import warnings
import numpy as np
import pandas as pd
from sklearn.cluster import KMeans
from sklearn.ensemble import IsolationForest
from sklearn.metrics import silhouette_score
from sklearn.preprocessing import StandardScaler

warnings.filterwarnings("ignore", module="threadpoolctl")   # a harmless warning from a dependency in the browser
rng = np.random.default_rng(8)
groups = [((20, 2), 150), ((80, 12), 100), ((300, 4), 50)]           # (avg order, orders per year), size
data = np.vstack([np.column_stack([rng.normal(m, m * 0.15, n), rng.normal(f, 1.5, n)]) for (m, f), n in groups])
customers = pd.DataFrame(data, columns=["avg_order", "orders_per_year"])
X = StandardScaler().fit_transform(customers)

for k in [2, 3, 4, 5]:
    labels = KMeans(n_clusters=k, n_init=10, random_state=8).fit_predict(X)
    print(f"k={k}: silhouette {silhouette_score(X, labels):.3f}")
customers["segment"] = KMeans(n_clusters=3, n_init=10, random_state=8).fit_predict(X)
print(customers.groupby("segment").mean().round(1).sort_values("avg_order"))

amounts = np.append(rng.normal(50, 10, 500), [400, 5, 650])           # three unusual transactions
flags = IsolationForest(contamination=0.01, random_state=8).fit_predict(amounts.reshape(-1, 1))
print("flagged amounts:", np.sort(amounts[flags == -1]).round(1))

k=2: silhouette 0.606
k=3: silhouette 0.814
k=4: silhouette 0.666
k=5: silhouette 0.605
         avg_order  orders_per_year
segment                            
2             19.9              2.1
1             79.3             12.2
0            309.9              3.5
flagged amounts: [  5.   78.1  81.6 400.  650. ]

Common mistakes

• Clustering unscaled features, so the feature with the largest numbers dominates.
• Picking k because it "looks nice" without comparing scores, or forcing clusters on data with none.
• Treating every flagged anomaly as fraud; anomalies are candidates for review (above, two ordinary amounts were flagged too).
• Reading meaning into cluster numbers; they're arbitrary labels that can change between runs.

Key points

• Scale features before distance-based methods like k-means.
• Choose k with silhouette scores and interpret segments by their averages.
• Isolation Forest flags unusual points for human review.

Further reading: Clustering · Novelty and outlier detection