Apply Now Apply Now Apply Now
header_logo
Post thumbnail
ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING

What Is Batch Normalization? Explained Simply

By Abhishek Pati

Batch normalization is a layer that adjusts the values flowing through a neural network so they stay on a similar scale. It does this by using the mean and variance of each mini-batch.

Deep networks often train slowly because the inputs to each layer keep shifting as the earlier weights change. Batch normalization keeps those inputs steady, so the network learns faster and with fewer problems.

Table of contents


  1. TL;DR Summary
  2. What Is Batch Normalization?
    • Formula and Intuition
  3. How Batch Normalization Works
    • Step 1: Find the Average of the Mini-Batch
    • Step 2: Measure How Spread Out the Values Are
    • Step 3: Normalize the Values
    • Step 4: Scale and Shift the Result
    • Step 5: Update the Running Averages
  4. Why Use Batch Normalization?
  5. Batch Normalization Code Example in PyTorch
  6. Placement and Inference Mode Mistakes
  7. Batch Norm vs Layer Norm vs Group Norm: Key Differences
  8. Conclusion
  9. FAQs
    • What is batch normalization in simple terms?
    • Why is batch normalization used in deep learning?
    • Where should batch normalization be placed in a neural network?
    • What is the difference between batch normalization and layer normalization?
    • Does batch normalization work with small batch sizes?

TL;DR Summary

  • Batch normalization is a layer that keeps each layer’s inputs on a steady scale, which makes deep networks train faster and more reliably.
  • It standardizes values with the mean and variance of each mini-batch, then restores flexibility through the learnable parameters γ and β.
  • Evaluation mode is essential, since inference relies on stored running statistics instead of the current batch.
  • Very small batches make its statistics noisy, so accuracy can drop when the batch size is limited.
  • Layer norm suits sequence models, and group norm suits image tasks with small batches.

What Is Batch Normalization?

Batch normalization is a layer that keeps the numbers flowing through a neural network within a steady range. For each small group of training examples, called a mini-batch, it centers the values around zero and evens out their spread. This lets the network learn faster and with fewer training problems.

It processes each feature separately, or each channel in convolutional layers.

Formula and Intuition

For a mini-batch of size m, the layer computes the following:

mean:        μ = (1/m) * Σ x_i
variance:    σ² = (1/m) * Σ (x_i - μ)²
normalize:   x̂_i = (x_i - μ) / sqrt(σ² + ε)
output:      y_i = γ * x̂_i + β

Each symbol means the following:

  • x_i: the value of one feature for the i-th sample in the batch
  • μ and σ²: the mean and variance of that feature across the batch
  • ε: a small constant, such as 1e-5, that prevents division by zero
  • γ and β: learnable parameters that set the final scale and offset

Normalization gives each feature a mean of 0 and a variance of 1, so the values entering a layer stay on a steady scale from one batch to the next.

The layer no longer has to adjust to large swings, allowing higher learning rates and more consistent gradient updates. The parameters γ and β then let the network restore any scale or offset that suits the task.

Curious how neural networks really learn? Batch normalization is just one piece of the puzzle. HCL GUVI’s Intel & IITM Pravartak Certified AI/ML Course walks you through deep learning with PyTorch, step by step, with no prior coding experience needed. Explore the course and start building real AI skills!

How Batch Normalization Works

During training, a batch normalization layer processes every mini-batch in the following five steps.

Step 1: Find the Average of the Mini-Batch

The layer adds up the values of one feature across all samples in the mini-batch and divides by the number of samples. This tells the layer where the center of those values is.

μ = (1/m) * Σ x_i
  • m: the number of samples in the mini-batch
  • x_i: the value of the feature for the i-th sample
  • μ: the mean of the feature across the mini-batch

Step 2: Measure How Spread Out the Values Are

The layer then checks how far each value sits from that average. The average of these squared distances shows how widely the values are scattered.

σ² = (1/m) * Σ (x_i - μ)²
  • σ²: the variance of the feature across the mini-batch

Step 3: Normalize the Values

Each value has the mean subtracted, which centers the data at zero. The result is divided by the standard deviation, which brings the spread to a common scale.

x̂_i = (x_i - μ) / sqrt(σ² + ε)
  • x̂_i: the normalized value, with a mean near 0 and a variance near 1
  • ε: a tiny constant (1e-5 by default in PyTorch) that prevents division by zero when the variance is very small

Step 4: Scale and Shift the Result

Forcing every feature to the same scale can limit what the network can express. The layer therefore stretches and moves the normalized values using two parameters that the network learns during training.

y_i = γ * x̂_i + β
  • y_i: the final output passed to the next layer
  • γ: the learnable scale, which stretches or shrinks the values
  • β: the learnable shift, which moves the values up or down

Step 5: Update the Running Averages

The layer keeps a running mean and a running variance that are refreshed after every mini-batch. At inference time, these stored values replace the mini-batch statistics, so a prediction does not depend on which other samples are in the batch.

running_mean = (1 - momentum) * running_mean + momentum * μ
running_var  = (1 - momentum) * running_var  + momentum * σ²
  • running_mean and running_var: the long-term estimates of the mean and variance, used at inference
  • momentum: controls how quickly the running values follow new batches (0.1 by default in PyTorch)

PyTorch updates the running variance with an unbiased estimate, dividing by m – 1 instead of m. The difference is negligible at normal batch sizes.

💡 Did You Know?

Batch normalization was introduced in 2015 by Sergey Ioffe and Christian Szegedy at Google, and it quickly became a standard building block in deep learning models.

Why Use Batch Normalization?

Batch normalization is used because it makes deep networks easier and faster to train.

  • Faster Training: Stable layer inputs allow higher learning rates, so the model reaches good accuracy in fewer epochs.
  • More Stable Gradients: Keeping activations on a consistent scale lowers the risk of vanishing or exploding gradients in deep networks.
  • Less Sensitivity to Initialization: Normalizing the inputs of each layer makes training less dependent on the starting weight values.
  • Mild Regularization: The statistics of each mini-batch add a small amount of noise, which can reduce overfitting.
  • Easier Hyperparameter Tuning: Training stays stable across a wider range of learning rates, so less trial and error is needed.

Also Read: Building a Neural Network Using PyTorch

GUVI Ad

Batch Normalization Code Example in PyTorch

The example below applies BatchNorm1d to a small batch of four samples with three features each.

import torch
import torch.nn as nn

bn = nn.BatchNorm1d(3)

x = torch.tensor([
    [1.0, 10.0, 100.0],
    [2.0, 20.0, 200.0],
    [3.0, 30.0, 300.0],
    [4.0, 40.0, 400.0],
])

bn.train()
y = bn(x)
print(y.detach())

Output:

tensor([[-1.3416, -1.3416, -1.3416],
        [-0.4472, -0.4472, -0.4472],
        [ 0.4472,  0.4472,  0.4472],
        [ 1.3416,  1.3416,  1.3416]])
  • nn.BatchNorm1d(3) creates a layer for inputs with 3 features, and this number must match the input feature dimension.
  • The input x holds 4 samples, and each feature column has a very different scale (1 to 4, 10 to 40, and 100 to 400).
  • Calling bn.train() puts the layer in training mode, so it normalizes with the mean and variance of the current batch.
  • The output y has identical values in all three columns, because each feature was normalized separately to a mean of 0 and a variance of 1.
  • The call y.detach() removes gradient tracking, so only the values are printed.
  • After training, calling bn.eval() makes the layer use its stored running statistics instead of the batch statistics.

Placement and Inference Mode Mistakes

Batch normalization usually goes after a linear or convolutional layer and before the activation function. Most errors come from using the wrong mode, batch size, or layer type.

  • Forgetting model.eval(): In training mode, predictions depend on the current batch. Switch to evaluation mode before inference so the stored running statistics are used.
  • Not calling model.train() again: After evaluation, switch back to training mode or the layer will stop using batch statistics.
  • Very small batches: Statistics from a few samples are noisy and lower accuracy. Use a larger batch, or switch to LayerNorm or GroupNorm.
  • Wrong layer type: Use BatchNorm1d for feature inputs and BatchNorm2d for image inputs. The num_features value must match the channel count, not the batch size.
  • Redundant bias: The shift parameter β already acts as a bias, so set bias=False in the preceding layer.

GUVI Ad

Debugging train and eval mode mistakes gets easier when you have experts beside you. In HCL GUVI’s Intel & IITM Pravartak Certified AI/ML Course, 1:1 doubt sessions and 20+ hands-on projects turn confusing concepts into skills you can trust. Ready to stop guessing and start building with confidence!

Batch Norm vs Layer Norm vs Group Norm: Key Differences

Batch norm, layer norm, and group norm all normalize activations, and they differ in which values they compute the mean and variance over.

FeatureBatch NormLayer NormGroup Norm
Normalizes OverEach channel, across all samples in the batch and all spatial positionsAll features of a single sampleGroups of channels within a single sample, plus spatial positions
Depends on Batch SizeYes, statistics come from the mini-batchNo, statistics come from each sample aloneNo, statistics come from each sample alone
Training vs InferenceUses batch statistics in training and running statistics at inferenceComputes the same way in both modesComputes the same way in both modes
Typical UseConvolutional networks for image tasks trained with moderate to large batchesTransformers and other sequence models in NLPImage tasks with small batches, such as object detection and segmentation

Conclusion

Batch normalization keeps the inputs of each layer on a steady scale, and it delivers its full benefit only when its training and inference behavior is handled correctly. Layer norm and group norm apply the same idea where small batches or sequence models make batch statistics unreliable. Together, they cover most normalization needs of modern architectures.

FAQs

1. What is batch normalization in simple terms?

Batch normalization is a layer that rescales the values inside a neural network so they stay on a steady scale, which helps the network train faster.

2. Why is batch normalization used in deep learning?

It stabilizes layer inputs, which allows higher learning rates, steadier gradients, and faster convergence.

3. Where should batch normalization be placed in a neural network?

It is usually placed after a linear or convolutional layer and before the activation function.

4. What is the difference between batch normalization and layer normalization?

Batch normalization computes statistics across the mini-batch, while layer normalization computes them across the features of a single sample.

5. Does batch normalization work with small batch sizes?

It works poorly because statistics from a few samples are noisy, so layer norm or group norm is the better choice.

Success Stories

Did you enjoy this article?

Schedule 1:1 free counselling

Similar Articles

Loading...
Get in Touch
Chat on Whatsapp
Request Callback
Share logo Copy link
Table of contents Table of contents
Table of contents Articles
Close button

  1. TL;DR Summary
  2. What Is Batch Normalization?
    • Formula and Intuition
  3. How Batch Normalization Works
    • Step 1: Find the Average of the Mini-Batch
    • Step 2: Measure How Spread Out the Values Are
    • Step 3: Normalize the Values
    • Step 4: Scale and Shift the Result
    • Step 5: Update the Running Averages
  4. Why Use Batch Normalization?
  5. Batch Normalization Code Example in PyTorch
  6. Placement and Inference Mode Mistakes
  7. Batch Norm vs Layer Norm vs Group Norm: Key Differences
  8. Conclusion
  9. FAQs
    • What is batch normalization in simple terms?
    • Why is batch normalization used in deep learning?
    • Where should batch normalization be placed in a neural network?
    • What is the difference between batch normalization and layer normalization?
    • Does batch normalization work with small batch sizes?