Evaluating Classification Models

Classification models assign observations to categories. Examples include identifying fraudulent transactions, detecting spam, and predicting whether a customer will leave a service.

Accuracy alone may not be sufficient, especially when the dataset contains imbalanced classes.

Understanding the Confusion Matrix

A confusion matrix compares actual labels with predicted labels. For binary classification, it contains four outcomes:

• True Positive (TP): The model correctly predicts the positive class.
• True Negative (TN): The model correctly predicts the negative class.
• False Positive (FP): The model predicts positive when the actual class is negative.
• False Negative (FN): The model predicts negative when the actual class is positive.

These outcomes are used to calculate several metrics.

• Accuracy: The proportion of all predictions that are correct.
• Precision: The proportion of predicted positives that are actually positive.
• Recall: The proportion of actual positives correctly identified.
• F1-score: The harmonic mean of precision and recall.
• ROC-AUC: Measures how well the model ranks positive examples above negative examples across classification thresholds.

Precision is especially useful when false positives are costly. Recall is important when missing positive cases is a major concern.

Calculating Classification Metrics

from sklearn.metrics import (

accuracy_score,

precision_score,

recall_score,

f1_score,

confusion_matrix,

classification_report,

roc_auc_score

)

from sklearn.linear_model import LogisticRegression

model = LogisticRegression(max_iter=1000)

model.fit(X_train, y_train)

predictions = model.predict(X_test)

probabilities = model.predict_proba(X_test)[:, 1]

print("Accuracy:", accuracy_score(y_test, predictions))

print("Precision:", precision_score(y_test, predictions))

print("Recall:", recall_score(y_test, predictions))

print("F1-score:", f1_score(y_test, predictions))

print("Confusion Matrix:\n", confusion_matrix(y_test, predictions))

print(classification_report(y_test, predictions))

print("ROC-AUC:", roc_auc_score(y_test, probabilities))

This example assumes a binary classification problem where the positive class is encoded as 1. Here, X_train, X_test, y_train, and y_test must contain classification data rather than regression targets.

The model's default classification threshold is typically 0.5. Adjusting the threshold can increase recall or precision, depending on the problem. Choose the threshold using validation data, not by repeatedly optimizing against the test set.

For imbalanced datasets, compare precision, recall, F1-score, and other suitable metrics rather than relying only on accuracy.