Evaluating Machine Learning Models

Training a model is only part of a machine learning project. You must also evaluate whether its predictions are useful on data it has not seen during training.

What Is Model Evaluation?

Model evaluation measures how well a trained model performs against a defined objective.

The right evaluation metric depends on the problem. For example, accuracy may be appropriate for some balanced classification tasks, while precision and recall can be more informative when errors have different costs.

Understanding Accuracy

Accuracy is the proportion of predictions that are correct.

[
\text{Accuracy} =
\frac{\text{Correct predictions}}{\text{Total predictions}}
]

Suppose a model correctly classifies 90 out of 100 observations.

[
\text{Accuracy} = \frac{90}{100}=0.90
]

The model has an accuracy of 90%.

However, accuracy can be misleading when one class is much more common than another.

For example, if only 1% of transactions are fraudulent, a model that always predicts "legitimate" could achieve 99% accuracy while detecting no fraud at all.

Precision and Recall

Precision and recall are important metrics for classification problems.

Precision measures the proportion of positive predictions that are actually positive.

[
\text{Precision} =
\frac{TP}{TP+FP}
]

Recall measures the proportion of actual positive cases that the model correctly identifies.

[
\text{Recall} =
\frac{TP}{TP+FN}
]

Where:

• (TP) = true positives.
• (FP) = false positives.
• (FN) = false negatives.

Consider a fraud detection system:

• High precision means fewer legitimate transactions are incorrectly flagged among those marked fraudulent.
• High recall means the system identifies a larger proportion of actual fraudulent transactions.

Improving one metric can sometimes reduce another, depending on the model and decision threshold.

Understanding the F1-Score

The F1-score combines precision and recall using their harmonic mean.

[
F1 = 2 \times
\frac{\text{Precision}\times\text{Recall}}
{\text{Precision}+\text{Recall}}
]

It is useful when both precision and recall matter.

However, the F1-score does not account for true negatives directly and may not capture the business cost of different errors. Metric selection should match the real problem.

Underfitting and Overfitting

Underfitting occurs when a model is too simple to capture important patterns in the data. It often performs poorly on both training and evaluation data.

Overfitting occurs when a model learns patterns specific to the training data, including noise, and fails to generalize well to new examples.

For example, a very deep decision tree might memorize details of its training observations. It may achieve excellent training accuracy but perform substantially worse on unseen data.

Common approaches to reducing overfitting include:

• Collecting more representative training data.
• Limiting model complexity.
• Applying regularization.
• Using cross-validation for model selection.
• Removing data leakage.
• Selecting appropriate features.

Generalization and Real-World Performance

A model generalizes well when it performs effectively on new observations drawn from the intended use environment.

Even a model that performs well during testing can become less reliable when the underlying data changes. For example, customer behavior may change over time, making a recommendation model's historical patterns less useful.

Real-world machine learning systems should therefore be monitored after deployment. Monitoring may include prediction quality, data drift, model errors, and operational performance.