Batch Normalization and Dropout

Batch normalization and dropout are two widely used techniques developed specifically to address these challenges, and they are now considered standard tools in modern deep learning rather than optional extras.

While they solve somewhat different problems, both techniques are usually inserted directly into a network's architecture as additional layers, working alongside the regular layers like convolutions or fully connected layers.

What is Overfitting?

Overfitting happens when a model learns the training data too closely, including its noise and random quirks, rather than learning the general patterns that would also apply to new, unseen data. A model that is overfitting will typically show excellent accuracy on its training data, but disappointing accuracy when tested on data it has never seen before.

This usually happens when a model has too much capacity, meaning too many layers or parameters, relative to the amount and variety of training data available. With enough capacity, a network can essentially memorize specific examples rather than learning the broader patterns connecting them. Overfitting is one of the most common practical problems encountered during model training, and recognizing it, typically by comparing training accuracy against validation accuracy.

How Dropout Prevents Overfitting

Step 1: Randomly Disable Neurons

During each training step, dropout randomly disables a certain percentage of neurons.

Step 2: Prevent Over-Reliance

This forces the network to avoid relying too heavily on any single neuron or specific combination of neurons.

Step 3: Learn Generalized Patterns

Because different neurons are dropped during different training steps, the network learns more generalized and redundant patterns instead of memorizing the training data.

Step 4: Disable Dropout During Prediction

Dropout is applied only during training. When the trained model is used for prediction, all neurons become active again, with their outputs adjusted to account for the absence of dropout.

Step 5: Reduce Overfitting

This technique helps reduce overfitting, especially in larger neural networks that have enough capacity to memorize the training data.

When to Use Batch Normalization

Batch normalization works by normalizing the outputs of a layer before passing them to the next layer, adjusting the values so they maintain a consistent scale and distribution throughout training. This helps address a problem where the distribution of values flowing through the network shifts unpredictably as weights are updated during training, which can otherwise slow down learning or cause instability.

Batch normalization is particularly useful in deeper networks, where this internal shifting tends to become more pronounced and problematic as data passes through many layers. It is commonly placed after convolutional or fully connected layers, before the activation function is applied. In practice, networks using batch normalization often train faster and can sometimes tolerate higher learning rates than networks without it, which is why it has become a near-default choice in many modern CNN and deep learning architectures.