Detecting Overfitting and Underfitting

A model's complexity affects its ability to learn patterns from data.

Overfitting occurs when a model learns noise or overly specific patterns in the training data. It performs well on training data but poorly on unseen data.

Underfitting occurs when a model is too simple to capture important patterns. It performs poorly on both training and validation data.

Controlling Model Complexity

For a decision tree, limiting its depth can help prevent overfitting.

from sklearn.tree import DecisionTreeClassifier

from sklearn.metrics import accuracy_score

shallow_tree = DecisionTreeClassifier(

max_depth=3,

random_state=42

)

shallow_tree.fit(X_train, y_train)

train_predictions = shallow_tree.predict(X_train)

valid_predictions = shallow_tree.predict(X_valid)

print(

"Training accuracy:",

accuracy_score(y_train, train_predictions)

)

print(

"Validation accuracy:",

accuracy_score(y_valid, valid_predictions)

)

This example assumes that the training and validation datasets belong to a classification problem. The chosen depth is only an example; the best value depends on the data.

Other approaches include:

• Regularization: Penalizing overly complex model parameters.
• Feature selection: Removing irrelevant or redundant features.
• More representative training data: Helping the model learn broader patterns.
• Learning curves: Comparing training and validation performance as the amount of training data increases.

The objective is not to maximize training performance alone. A useful model performs consistently on data that was not used to train or tune it.