Classification Metrics: Beyond Accuracy

What you'll build

A fraud detector on data where only 3% of transactions are fraudulent: a model with 97% accuracy that catches no fraud, and a better model judged by recall, precision and the confusion matrix.

The idea

• Confusion matrix: true negatives, false positives, false negatives, true positives.
• Precision = TP / (TP + FP): of the transactions flagged, how many were fraud?
• Recall = TP / (TP + FN): of all fraud, how much did we catch?
• F1: harmonic mean of precision and recall.
• On imbalanced data, accuracy is dominated by the majority class; choose metrics that match the business cost of each error type. class_weight="balanced" makes models pay more attention to the rare class.

The code

from sklearn.datasets import make_classification
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=5000, n_features=10, n_informative=5, weights=[0.97], flip_y=0, random_state=7)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, stratify=y, random_state=7)

lazy = DummyClassifier(strategy="most_frequent").fit(X_train, y_train)
print("always 'not fraud' accuracy:", round(lazy.score(X_test, y_test), 3))
print(confusion_matrix(y_test, lazy.predict(X_test)))

model = LogisticRegression(class_weight="balanced", max_iter=1000).fit(X_train, y_train)
pred = model.predict(X_test)
print(confusion_matrix(y_test, pred))
print(classification_report(y_test, pred, target_names=["ok", "fraud"], digits=3))

always 'not fraud' accuracy: 0.97
[[1455    0]
 [  45    0]]
[[1253  202]
 [  12   33]]
              precision    recall  f1-score   support

          ok      0.991     0.861     0.921      1455
       fraud      0.140     0.733     0.236        45

    accuracy                          0.857      1500
   macro avg      0.565     0.797     0.579      1500
weighted avg      0.965     0.857     0.901      1500

Common mistakes

• Celebrating high accuracy on imbalanced data.
• Optimising precision when missing a positive (fraud, disease) is the expensive error, or recall when false alarms are.
• Reporting metrics on the training set, or averaging across classes without saying how (macro versus weighted).

Key points

• Read the confusion matrix first.
• Precision and recall trade off; pick by business cost.
• Use class_weight or resampling for rare classes.

Further reading: Classification metrics · Confusion matrix