Evaluating Supervised Learning Models

Why Is Evaluation Important?

A model's training performance does not necessarily indicate how well it will perform on unseen data.

Evaluation helps determine whether a model has learned useful patterns, compare competing approaches, and identify weaknesses before deployment.

The appropriate metrics depend on whether the problem is regression or classification and on the consequences of prediction errors.

Regression Metrics

Mean Absolute Error (MAE) measures the average absolute difference between actual and predicted values.

[
MAE = \frac{1}{n}\sum_{i=1}^{n}|y_i-\hat{y}_i|
]

Mean Squared Error (MSE) measures the average squared prediction error.

[
MSE = \frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{y}_i)^2
]

Root Mean Squared Error (RMSE) is the square root of MSE:

[
RMSE = \sqrt{MSE}
]

MAE is easier to interpret in the target's original units. MSE and RMSE penalize larger errors more heavily than MAE.

Another metric, (R^2), compares model performance with a baseline that predicts the mean of the observed target values. A higher (R^2) often indicates a better fit, but (R^2) can be negative on evaluation data when the model performs worse than that baseline.

Classification Metrics

For classification, common metrics include:

• Accuracy: The proportion of predictions that are correct.
• Precision: The proportion of predicted positive cases that are actually positive.
• Recall: The proportion of actual positive cases correctly identified.
• F1-score: A harmonic mean of precision and recall.
• Confusion matrix: A table showing counts of correct and incorrect classifications.

Consider a fraud detection model. High recall may be important when missing fraudulent transactions is costly, while high precision may be important when incorrectly blocking legitimate transactions is costly.

Confusion Matrix

A confusion matrix compares predicted classes with actual classes.

For a binary classification problem:

These values help explain the types of mistakes a model makes.

A model can have high accuracy while missing many positive cases if the positive class is rare. Therefore, accuracy should not be used in isolation when classes are imbalanced.

Evaluating a Model in Python

from sklearn.metrics import (
accuracy_score,
classification_report,
confusion_matrix
)
predictions = model.predict(X_test)
print("Accuracy:", accuracy_score(y_test, predictions))
print("Confusion matrix:")
print(confusion_matrix(y_test, predictions))
print("Classification report:")
print(classification_report(y_test, predictions))

This example assumes that model is a trained classifier and X_test and y_test contain the held-out test data.

The confusion matrix and classification report provide more detailed information than accuracy alone.

Cross-Validation

Cross-validation provides a more comprehensive way to evaluate a model using multiple training and validation splits.

In (k)-fold cross-validation, the training dataset is divided into (k) folds. The model is trained on (k-1) folds and evaluated on the remaining fold. This process repeats until each fold has served as the validation fold.

The resulting scores can be averaged to estimate performance across different splits.

Cross-validation is particularly useful when data is limited. However, the final test set should remain separate from model selection and hyperparameter tuning.

For time-dependent data, ordinary random cross-validation may be inappropriate because it can allow future information to influence predictions about the past. Time-aware splitting should be used when the problem requires it.