Model Evaluation and Choosing Metrics

Why evaluation matters

• A model must be evaluated on data it wasn't trained on.
• Otherwise reported performance may reflect memorisation, not the ability to generalise.

Classification metrics

Confusion matrix:

Which metric matters?

Regression metrics

Example

Fraud detection with rare fraud:

High accuracy, useless model. This is why accuracy alone is misleading for imbalanced classes.

Common mistakes

• Choosing metrics without considering business risk.
• Ignoring class imbalance.
• Using the default 0.5 threshold without testing alternatives.
• Not reporting a baseline.
• Looking only at one headline score instead of examining the errors themselves.
• Re-using the test set to tune the model, which makes the final score over-optimistic.