Menu

Backpropagation in Deep Learning

Neural networks learn by repeatedly generating predictions, measuring errors, and updating their internal parameters.

Two important processes make this possible: forward propagation and backpropagation.

Forward propagation calculates a neural network's output from its inputs. Backpropagation efficiently computes gradients of the loss with respect to trainable parameters using the chain rule.

An optimisation algorithm then uses these gradients to update the network.

What Is Forward Propagation?

Forward propagation passes input data through the network's layers to generate an output.

For layer \(l\):

\[ Z^{(l)}=A^{(l-1)}W^{(l)}+b^{(l)} \]

\[ A^{(l)}=f(Z^{(l)}) \]

Here:

  • \(A^{(l-1)}\) represents the previous layer's activations.
  • \(W^{(l)}\) represents the current layer's weights.
  • \(b^{(l)}\) represents its bias.
  • \(f\) represents the activation function.

Each layer transforms the previous layer's output.

Forward Propagation Using NumPy

import numpy as npX = np.array([[2.0, 3.0]])weights = np.array([    [0.5, 0.2],    [0.8, 0.4]])bias = np.array([0.2, 0.1])z = X @ weights + biasactivation = np.maximum(0, z)print(activation)

Output:

[[3.6 1.7]]

The example uses matrix multiplication, bias addition, and ReLU activation.

What Is a Loss Function?

A loss function measures the disagreement between predictions and actual target values.

Mean Squared Error

For regression:

\[ MSE=\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{y}_i)^2 \]

MSE penalises larger numerical errors more heavily.

Binary Cross-Entropy

For binary classification:

\[ L=-\frac{1}{n}\sum_{i=1}^{n} [y_i\log(p_i)+(1-y_i)\log(1-p_i)] \]

Here, \(p_i\) is the estimated positive-class probability.

Categorical Cross-Entropy

For mutually exclusive multiclass classification:

\[ L=-\frac{1}{n}\sum_{i=1}^{n} \sum_{k=1}^{K}y_{ik}\log(p_{ik}) \]

The loss measures how well predicted class probabilities match the actual categories.

What Is Backpropagation?

Backpropagation calculates how the loss changes with respect to the network's weights and biases.

It applies the chain rule across connected mathematical operations.

Consider a simplified model:

\[ z=wx+b \]

\[ a=f(z) \]

\[ L=L(a) \]

The gradient with respect to \(w\) is:

\[ \frac{\partial L}{\partial w} = \frac{\partial L}{\partial a} \frac{\partial a}{\partial z} \frac{\partial z}{\partial w} \]

This calculation identifies how changing a weight affects the loss.

Deep learning frameworks perform such calculations using automatic differentiation.

How Does Gradient Descent Update Neural Network Weights?

After gradients are calculated, an optimiser updates parameters.

The basic gradient descent rule is:

\[ w_{t+1}=w_t-\eta\frac{\partial L}{\partial w} \]

Here:

  • \(w_t\) represents the current weight.
  • \(\eta\) represents the learning rate.
  • \(\partial L/\partial w\) represents the weight's gradient.

Suppose:

\[ w=2,\quad \eta=0.1,\quad \frac{\partial L}{\partial w}=0.5 \]

Then:

\[ w_{new}=2-0.1(0.5) \]

\[ w_{new}=1.95 \]

The weight decreases according to the gradient direction.

What Are Batch Size and Epochs?

Batch Size: Number of training observations processed together for a parameter update.

Epoch: One complete pass through the training dataset.

Iteration: One optimisation update, typically based on a batch.

For 1,000 observations with a batch size of 100, one epoch generally contains 10 iterations.

For five epochs, training performs 50 iterations when all observations are processed once per epoch.

What Are Common Neural Network Optimizers?

Optimizer Main Characteristic
SGD Updates parameters using stochastic or mini-batch gradients
SGD With Momentum Incorporates previous update directions
RMSprop Adapts updates using squared-gradient estimates
Adam Uses adaptive first- and second-moment estimates
AdamW Uses Adam-style updates with decoupled weight decay

Adam is a common starting point for deep learning, but it is not universally superior.

The appropriate optimiser depends on the model and training conditions.

What Is the Learning Rate?

The learning rate determines the size of optimisation updates.

A very small learning rate may produce slow learning.

An excessively large rate may cause instability or divergence.

Learning-rate schedules change the rate during training.

For example, a model may begin with larger updates and use smaller updates later.

What Are Vanishing and Exploding Gradients?

Vanishing Gradients

Vanishing gradients occur when gradients become very small as they propagate through layers.

Earlier layers may consequently learn slowly.

Exploding Gradients

Exploding gradients occur when gradient magnitudes become excessively large.

This can cause unstable updates and numerical problems.

Techniques that may help include suitable parameter initialisation, residual connections, normalisation, and gradient clipping.

What Is Regularisation in Deep Learning?

Regularisation helps reduce overfitting.

Common techniques include:

L2 Regularisation: Penalises large parameter values.

Dropout: Randomly sets selected activations to zero during training.

Early Stopping: Stops training according to a monitored validation criterion.

Data Augmentation: Creates training variations through suitable transformations.

Dropout Example

from tensorflow import kerasmodel = keras.Sequential([    keras.layers.Input(shape=(10,)),    keras.layers.Dense(32, activation="relu"),    keras.layers.Dropout(0.3),    keras.layers.Dense(1, activation="sigmoid")])

A dropout rate of 0.3 means that approximately 30% of eligible activations are randomly dropped during training.

Dropout is normally disabled during inference.

What Is Early Stopping?

Early stopping monitors a validation metric and stops training after a specified period without sufficient improvement.

early_stopping = keras.callbacks.EarlyStopping(    monitor="val_loss",    patience=5,    restore_best_weights=True)

The callback monitors validation loss.

patience=5 allows five consecutive epochs without improvement before stopping, subject to the callback's configuration.

Common Neural Network Training Mistakes

  1. Confusing backpropagation with the optimisation algorithm.
  2. Training for too many epochs without validation.
  3. Choosing an unsuitable learning rate.
  4. Ignoring exploding or vanishing gradients.
  5. Applying regularisation without checking its impact.
  6. Evaluating performance only on training data.

Data Science Tutorial for Beginners

VA

Vaishali Ardhana



Learn via Video Course

undefined Logo

0 hrs



Get in Touch For Details