Gradient Descent Explained

Gradient descent is the optimization algorithm used to train a neural network, by gradually adjusting its weights to reduce prediction errors.

What Problem It Solves

A model starts with random weights, so its predictions are usually inaccurate. Gradient descent finds the direction that reduces error most and updates the weights step by step in that direction.

How It Works

It calculates the gradient, which shows how much the error changes when a weight changes slightly. It then updates the weights in the opposite direction of the gradient, since that direction reduces the error.

weight = weight - learning_rate * gradient     

Role of Learning Rate

The learning rate controls how big each update step is. A learning rate that is too high can overshoot the best solution, while one that is too low makes training extremely slow.

Types of Gradient Descent

• Batch Gradient Descent: Uses the entire dataset to calculate each update, accurate but slow on large data.
• Stochastic Gradient Descent: Updates weights using just one data point at a time, faster but noisier.
• Mini Batch Gradient Descent: Uses small groups of data points, balancing speed and stability.