Every time a neural network learns to recognise a cat, translate a sentence, or predict tomorrow’s stock price, one algorithm is responsible for making that learning happen: backpropagation.<\/p>\n\n\n\n
Backpropagation, short for backward propagation of errors, is the training algorithm that makes deep learning possible. It calculates how much each weight in a neural network contributed to the prediction error, and uses that information to adjust every weight in a direction that reduces future errors. Without it, training a network with more than one hidden layer would be computationally intractable.<\/p>\n\n\n\n
Despite its centrality to modern AI, backpropagation is often presented as a black box \u2014 something that “just works” inside frameworks like TensorFlow and PyTorch. This article pulls back the curtain: explaining what backpropagation is, how it works mathematically and intuitively, and why it remains the cornerstone of deep learning after more than three decades.<\/p>\n\n\n\n
\n Backpropagation is a supervised learning algorithm used to train neural networks by calculating the gradient of the loss function with respect to every weight in the network. It works by applying the chain rule of calculus during a backward pass from the output layer to the input layer, determining how each weight contributes to prediction error. Combined with gradient descent, backpropagation continuously updates the weights to reduce error and help the network converge toward an optimal solution.\n <\/p>\n\n <\/div>\n\n<\/div>\n\n\n\n
To understand why backpropagation matters, it helps to understand the problem it was designed to solve: the credit assignment problem.m <\/p>\n\n\n\n
A neural network<\/a> with multiple layers contains thousands, sometimes billions of weights. When the network makes a wrong prediction, the error is visible at the output layer. But which weights were responsible? And by how much should each one be adjusted?<\/p>\n\n\n\n In a shallow network with no hidden layers, the answer is straightforward: the output weights are directly connected to the prediction. But in a deep network with many hidden layers, the relationship between any individual weight and the final error is indirect, mediated by every layer between that weight and the output.<\/p>\n\n\n\n This is the credit assignment problem: efficiently determining the contribution of each weight to the overall prediction error, across many layers and many thousands of parameters.<\/p>\n\n\n\n Before backpropagation, training multi-layer networks required either finite difference approximations, estimating gradients by perturbing each weight individually, which requires a full forward pass per weight and scales disastrously or random weight perturbation methods that were too slow and imprecise to be practical.<\/p>\n\n\n\n Backpropagation solved this by applying the chain rule of calculus to compute exact gradients for every weight in a single backward pass. What would take thousands of forward passes to approximate with finite differences, backpropagation computes exactly in two passes, one forward, one backward.<\/p>\n\n\n\n Backpropagation consists of two phases. The first is the forward pass, which produces the network’s prediction. Understanding the forward pass precisely is essential before the backward pass makes sense.<\/p>\n\n\n\n In the forward pass, data flows from the input layer through each hidden layer to the output layer. At each neuron in each layer, two operations occur:<\/p>\n\n\n\n 1. Weighted sum: <\/strong>The neuron computes the weighted sum of its inputs, each input multiplied by its corresponding connection weight, plus a bias term. This is the pre-activation value, often called z.<\/p>\n\n\n\n 2. Activation function: <\/strong>The activation function is applied to z, producing the neuron’s output, its activation value a. This output becomes the input to every neuron in the next layer.<\/p>\n\n\n\n The activation function is critical. It introduces non-linearity into the network, enabling it to learn complex, non-linear patterns. Without activation functions, a deep network would reduce to a single linear transformation regardless of depth. Common activation functions include:<\/p>\n\n\n\n After the forward pass produces a prediction, the loss function measures how far that prediction is from the correct answer. The choice of loss function depends on the task:<\/p>\n\n\n\n The loss value is a single scalar,r, a summary of how wrong the network’s prediction was. The backward pass will use this value to compute gradients.<\/p>\n\n\n\n The backward pass is where backpropagation earns its name. It propagates the error signal from the output layer back through the network, computing the gradient of the loss with respect to every weight in the network.<\/p>\n\n\n\n The chain rule of calculus states that the derivative of a composite function can be computed as the product of the derivatives of its components. Formally, if y = f(g(x)), then:<\/p>\n\n\n\n dy\/dx = (dy\/dg) \u00d7 (dg\/dx)<\/p>\n\n\n\n A neural network is a deeply nested composite function. The output of the final layer is a function of the outputs of the penultimate layer, which are functions of the outputs of the layer before that, all the way back to the input. The chain rule allows the gradient of the loss with respect to any weight, no matter how many layers deep, to be expressed as a product of local gradients computed at each layer.<\/p>\n\n\n\n This is the mathematical insight that makes backpropagation efficient: instead of computing each gradient independently, the algorithm<\/a> reuses intermediate values computed during the forward pass, sharing computation across all gradients through a single backward sweep.<\/p>\n\n\n\n The backward pass begins at the output layer. The gradient of the loss with respect to the output layer’s pre-activation values is computed directly from the loss function and the activation function’s derivative.<\/p>\n\n\n\n This gradient is then propagated backward to the previous layer. For each layer, the backward pass computes:<\/p>\n\n\n\n \u2022 Gradient with respect to weights: <\/strong>How much does the loss change if this weight changes slightly? This is the value used to update the weight.<\/p>\n\n\n\n \u2022 Gradient with respect to biases: <\/strong>How much does the loss change if this bias changes slightly? Used to update the bias term.<\/p>\n\n\n\n \u2022 Gradient with respect to layer inputs: <\/strong>How much does the loss change if this layer’s input changes slightly? This becomes the gradient passed to the previous layer, continuing the backward propagation.<\/p>\n\n\n\n This process repeats for every layer from output back to input, accumulating the gradient signal and distributing it to every weight in the network.<\/p>\n\n\n\n \n Although backpropagation<\/strong> is most strongly associated with the influential 1986 paper<\/strong> by David Rumelhart<\/strong>, Geoffrey Hinton<\/strong>, and Ronald Williams<\/strong>, the mathematical ideas behind it were independently discovered several times across different fields before deep learning became popular. The core principle\u2014efficiently computing gradients through layered computations using the chain rule<\/strong>\u2014appeared in earlier work on control theory, optimization, and automatic differentiation long before neural networks brought it into mainstream AI research.\n <\/p>\n<\/div>\n\n\n\n Computing gradients is the core of backpropagation, but gradients alone do not change the network. The weight update step implemented by gradient descent applies the gradients to modify every weight in the direction that reduces the loss.<\/p>\n\n\n\n The fundamental weight update rule is:<\/p>\n\n\n\n w \u2190 w \u2212 \u03b7 \u00d7 (\u2202L \/ \u2202w)<\/p>\n\n\n\n Where:<\/p>\n\n\n\n \u2022 w: <\/strong>The current weight value.<\/p>\n\n\n\n \u2022 \u03b7 (eta): <\/strong>The learning rate is a hyperparameter that controls how large each update step is.<\/p>\n\n\n\n \u2022 \u2202L \/ \u2202w: <\/strong>The gradient of the loss function with respect to this weight, computed by backpropagation.<\/p>\n\n\n\n The negative sign is critical: subtracting the gradient moves the weight in the direction of steepest descent on the loss surface toward a minimum. Gradients point toward the steepest increase in loss; moving against the gradient reduces it.<\/p>\n\n\n\n The learning rate determines how aggressively the network updates its weights at each step:<\/p>\n\n\n\n Modern practice uses adaptive learning rate methods, such as Adam, RMSProp, and AdaGrad, that automatically adjust the effective learning rate for each weight based on the history of its gradients, significantly reducing the sensitivity to learning rate choice.<\/p>\n\n\n\n In practice, weight updates are not computed using a single training example (stochastic gradient descent) or the entire dataset (batch gradient descent) \u2014 they use mini-batches: small random subsets of the training data, typically 32 to 256 examples. This approach:<\/p>\n\n\n\n \u2022 Provides gradient estimates that are more stable than single-example updates.<\/p>\n\n\n\n \u2022 Fits naturally into GPU parallelism, which processes batches simultaneously.<\/p>\n\n\n\n \u2022 Introduces beneficial noise that helps the network escape sharp local minima.<\/p>\n\n\n\n Backpropagation has two well-known failure modes that become more severe as network depth increases: vanishing gradients and exploding gradients. Both arise from the multiplicative nature of the chain rule.<\/p>\n\n\n\n During the backward pass, the chain rule multiplies local gradients together across every layer. If these local gradients are consistently less than 1, as they are for the sigmoid and tanh activation functions in their saturated regions, the product of many such values becomes exponentially small.<\/p>\n\n\n\n The result is that gradients flowing back to the early layers of the network become vanishingly small. Early layers update their weights by an imperceptible amount, effectively failing to learn. This is why deep networks trained with sigmoid activations often fail to learn good representations in their early layers.<\/p>\n\n\n\n Solutions include:<\/p>\n\n\n\n The opposite problem occurs when local gradients are consistently greater than 1. The chain rule product grows exponentially with depth, producing gradient values so large that weight updates become unstable, and the network diverges rather than converges.<\/p>\n\n\n\n Solutions include:<\/p>\n\n\n\n Standard backpropagation operates on feedforward networks where information flows in one direction from input to output. Recurrent neural networks (RNNs), however, have connections that loop back on themselves, allowing information to persist across sequential inputs. Training RNNs requires a variant of backpropagation called Backpropagation Through Time (BPTT).<\/p>\n\n\n\n BPTT unrolls the recurrent network across time steps, treating the RNN as a very deep feedforward network where each “layer” corresponds to one time step. Standard backpropagation is then applied through this unrolled network, computing gradients with respect to the shared weights at each time step and summing them.<\/p>\n\n\n\n The challenge of BPTT is that RNNs applied to long sequences produce very deep unrolled networks, making them particularly susceptible to the vanishing and exploding gradient problems. This is why Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) architectures were developed. They use gating mechanisms that allow gradients to flow across long sequences without vanishing.<\/p>\n\n\n\n Backpropagation is not just a training algorithm for simple networks; it scales to the largest AI systems ever built. Every major architecture in modern deep learning relies on backpropagation for training.<\/p>\n\n\n\n Modern deep learning frameworks TensorFlow, PyTorch, and JAX implement automatic differentiation (autograd) systems that automatically compute and apply backpropagation gradients for any network architecture defined by the user. This automation is what has made the explosive growth of deep learning research and deployment possible.<\/p>\n\n\n\nWhy Early Approaches Failed<\/strong><\/h3>\n\n\n\n
Forward Pass: From Input to Prediction<\/strong><\/h2>\n\n\n\n
Layer-by-Layer Computation<\/strong><\/h3>\n\n\n\n
\n
Computing the Loss<\/strong><\/h3>\n\n\n\n
\n
Backward Pass: Propagating the Error<\/strong><\/h2>\n\n\n\n
The Chain Rule: The Mathematical Foundation<\/strong><\/h3>\n\n\n\n
Computing Gradients Layer by Layer<\/strong><\/h3>\n\n\n\n
Weight Update: From Gradients to Learning<\/strong> <\/h2>\n\n\n\n
The Gradient Descent Update Rule<\/strong><\/h3>\n\n\n\n
The Learning Rate: A Critical Hyperparameter<\/strong><\/h3>\n\n\n\n
\n
Mini-Batch Gradient Descent<\/strong><\/h3>\n\n\n\n
The Vanishing and Exploding Gradient Problems<\/strong><\/h2>\n\n\n\n
Vanishing Gradients<\/strong><\/h3>\n\n\n\n
\n
Exploding Gradients<\/strong><\/h3>\n\n\n\n
\n
Backpropagation Through Time (BPTT)<\/strong> <\/h2>\n\n\n\n
How BPTT Works<\/strong><\/h3>\n\n\n\n
Backpropagation in Modern Deep Learning<\/strong><\/h2>\n\n\n\n
\n