Overfitting, Regularization, and Model Improvement

What Is Overfitting?

Overfitting occurs when a model learns the training data too closely, including noise or accidental patterns, and performs poorly on unseen examples.

For example, a neural network may achieve very high training accuracy but substantially lower validation accuracy.

Underfitting occurs when a model is too simple or insufficiently trained to capture important patterns, resulting in poor performance on both training and validation data.

Techniques to Improve Generalization

1. Train, Validation, and Test Splits

• Training data is used to learn model parameters.
• Validation data helps select architectures and tune hyperparameters.
• Test data provides a final evaluation after model choices are complete.

2. Dropout

Dropout randomly disables a proportion of neurons during training, helping reduce excessive dependence on particular neurons.

keras.layers.Dense(64, activation="relu"),

keras.layers.Dropout(0.3),

keras.layers.Dense(3, activation="softmax")

The dropout rate of 0.3 means that 30% of the eligible neuron outputs are randomly dropped during training. Dropout is disabled during ordinary inference.

3. Early Stopping

Early stopping monitors validation performance and stops training when improvement stalls.

early_stopping = keras.callbacks.EarlyStopping(
monitor="val_loss",
patience=5,
restore_best_weights=True
)
Pass this callback to model.fit() and provide validation data:
model.fit(
X_train,
y_train,
validation_split=0.2,
epochs=100,
callbacks=[early_stopping]
)

When using this approach, the validation split should come only from the training portion of the dataset. Keep the test set separate for final evaluation.

4. Regularization

Techniques such as L1 and L2 regularization penalize certain large or complex parameter values, encouraging simpler models.

5. Data Quality and Appropriate Model Size

Correct labels, representative training examples, suitable feature scaling, and an appropriately sized network can improve generalization. Adding more layers does not automatically improve performance.