Gradient Descent in Machine Learning
Machine learning models improve their predictions by adjusting internal parameters during training. The objective is to find parameter values that minimise prediction errors according to a selected loss function.
Gradients and gradient descent provide the mathematical framework for making these adjustments.
A gradient is a vector of partial derivatives that describes how a multivariable function changes with respect to its inputs. Gradient descent is an iterative optimisation algorithm that updates parameters in the direction opposite to the gradient to reduce a loss function.
The chain rule connects these concepts by enabling derivatives to be calculated through composite mathematical operations.
What Is a Gradient in Machine Learning?
A gradient is a collection of partial derivatives of a scalar-valued function.
Consider a loss function:
\[ L(w_1,w_2) \]
The gradient is represented as:
\[ \nabla L= \begin{bmatrix} \frac{\partial L}{\partial w_1}\\ \frac{\partial L}{\partial w_2} \end{bmatrix} \]
Each component measures how the loss changes with respect to one parameter.
For a differentiable function, the gradient points in the direction of the steepest local increase under the standard Euclidean metric.
The opposite direction indicates the steepest local decrease.
How Does Gradient Descent Work?
Gradient descent repeatedly updates model parameters to reduce a differentiable objective function.
The parameter update rule is:
\[ w_{t+1}=w_t-\eta\nabla L(w_t) \]
Here:
- \(w_t\) represents the current parameter vector.
- \(\eta\) represents the learning rate.
- \(\nabla L(w_t)\) represents the gradient at the current parameters.
- \(w_{t+1}\) represents the updated parameter vector.
The learning rate determines the size of each update.
A sufficiently small learning rate may produce gradual improvement, while a learning rate that is too large can cause unstable updates or divergence.
Numerical Example of Gradient Descent
Consider the function:
\[ L(w)=w^2 \]
Its derivative is:
\[ \frac{dL}{dw}=2w \]
Suppose the initial parameter is:
\[ w_0=4 \]
Use a learning rate of:
\[ \eta=0.1 \]
The update rule becomes:
\[ w_{t+1}=w_t-0.1(2w_t) \]
First Iteration
\[ w_1=4-0.1(8) \]
\[ w_1=3.2 \]
The loss changes from:
\[ L(4)=16 \]
to:
\[ L(3.2)=10.24 \]
Second Iteration
\[ w_2=3.2-0.1(6.4) \]
\[ w_2=2.56 \]
The loss becomes:
\[ L(2.56)=6.5536 \]
The parameter gradually approaches zero, which is the global minimum of this particular quadratic function.
How Is Gradient Descent Implemented in Python?
weight = 4.0learning_rate = 0.1for iteration in range(5): gradient = 2 * weight weight = weight - learning_rate * gradient loss = weight ** 2 print( f"Iteration {iteration + 1}: " f"Weight = {weight:.4f}, " f"Loss = {loss:.4f}" )Output:
Iteration 1: Weight = 3.2000, Loss = 10.2400
Iteration 2: Weight = 2.5600, Loss = 6.5536
Iteration 3: Weight = 2.0480, Loss = 4.1943
Iteration 4: Weight = 1.6384, Loss = 2.6844
Iteration 5: Weight = 1.3107, Loss = 1.71799This example demonstrates how repeated parameter updates reduce the loss.
In complex machine learning models, convergence is not always guaranteed, and the optimisation process may approach a local minimum or another stationary point rather than a global minimum.
What Is the Learning Rate?
The learning rate controls the magnitude of gradient descent updates.
It is typically represented by \(\eta\) or \(\alpha\).
| Learning Rate Behaviour | Possible Outcome |
|---|---|
| Very small | Slow convergence |
| Moderate and suitable | Stable progress towards a minimum |
| Excessively large | Oscillation or divergence |
| Decreasing schedule | Smaller updates later in training |
There is no single learning rate that works best for every model.
The suitable value depends on the loss function, feature scaling, model architecture, and optimisation algorithm.
Why Is Feature Scaling Important?
Gradient-based optimisation can be affected by differences in feature magnitudes.
For example, a dataset may contain customer age values between 18 and 70, while annual income values range from ₹2 lakh to ₹50 lakh.
Such differences can produce poorly conditioned optimisation problems.
Standardisation often helps numerical optimisation by placing features on more comparable scales.
However, scaling is not equally important for every machine learning algorithm.
What Is the Chain Rule?
The chain rule calculates derivatives of composite functions.
A composite function contains one function inside another.
For example:
\[ y=(3x+2)^2 \]
Let:
\[ u=3x+2 \]
Then:
\[ y=u^2 \]
The chain rule states:
\[ \frac{dy}{dx}= \frac{dy}{du}\frac{du}{dx} \]
The derivative of \(y\) with respect to \(u\) is:
\[ \frac{dy}{du}=2u \]
The derivative of \(u\) with respect to \(x\) is:
\[ \frac{du}{dx}=3 \]
Therefore:
\[ \frac{dy}{dx}=2(3x+2)(3) \]
\[ \frac{dy}{dx}=18x+12 \]
The chain rule is essential because neural networks are composed of multiple connected functions.
How Is the Chain Rule Used in Neural Networks?
A neural network layer performs a linear transformation followed by an activation function.
For example:
\[ z=wx+b \]
\[ a=f(z) \]
The output \(a\) depends on \(z\), which depends on \(w\).
To determine how a loss changes with respect to \(w\), derivatives must be propagated through the connected operations.
The chain rule gives:
\[ \frac{\partial L}{\partial w} = \frac{\partial L}{\partial a} \frac{\partial a}{\partial z} \frac{\partial z}{\partial w} \]
This expression connects the loss function to an earlier model parameter.
The same principle applies across many layers.
What Is Backpropagation?
Backpropagation is an efficient procedure for computing gradients of a neural network's loss function with respect to its trainable parameters.
It applies the chain rule through the network's computational graph.
A simplified training sequence is:
- Forward Pass: Input features move through the network to generate predictions.
- Loss Calculation: A loss function measures prediction error.
- Backward Pass: Backpropagation calculates gradients with respect to model parameters.
- Parameter Update: An optimisation algorithm updates the weights and biases.
- Repetition: The process continues across training iterations.
Backpropagation computes gradients, while algorithms such as SGD or Adam use those gradients to update parameters.
These are related but distinct operations.
What Are the Common Problems in Gradient Descent?
Slow Convergence
The optimisation process may require many iterations when learning rates are small or the objective is poorly conditioned.
Divergence
Large learning rates may cause parameter updates to overshoot useful regions of the loss function.
Local Minima
Non-convex objective functions can contain multiple local minima.
Gradient descent may converge to a local minimum rather than the global minimum.
Saddle Points
A saddle point is a stationary point that is neither a local minimum nor a local maximum.
Such points can complicate optimisation in high-dimensional models.
Vanishing and Exploding Gradients
Deep neural networks may encounter gradients that become extremely small or large as they propagate through layers.
Appropriate activation functions, initialisation methods, normalisation techniques, and architectural choices can help address these issues.
How Are Gradients Used in Real-World Data Science?
Gradients support optimisation in several machine learning tasks.
| Application | Role of Gradients |
|---|---|
| Linear Regression | Reduce numerical prediction error |
| Logistic Regression | Minimise classification loss |
| Neural Networks | Update weights through backpropagation |
| Deep Learning | Train models with many parameters |
| Regularised Models | Optimise loss functions with penalty terms |
Common Mistakes in Gradient Descent
- Confusing the gradient with the loss value.
- Selecting a learning rate without checking training behaviour.
- Assuming gradient descent always reaches a global minimum.
- Confusing backpropagation with the optimiser itself.
- Ignoring feature scaling and numerical stability.
Data Science Tutorial for Beginners
Vaishali Ardhana








