Gradient Descent Variants

Over time, researchers developed several variants designed to make training faster, more stable, and less sensitive to the specific learning rate chosen, addressing practical limitations that became apparent as networks grew larger and datasets grew bigger.

Understanding these variants matters because the choice of optimizer can meaningfully affect how quickly a model trains and how well it ultimately performs. Rather than treating optimizer selection as an afterthought, experienced practitioners often treat it as one of the key decisions to experiment with when trying to improve a model's training process.

Stochastic Gradient Descent (SGD)

Stochastic gradient descent improves on basic gradient descent by updating the model's weights using only a small, random batch of training examples at a time, rather than the entire dataset all at once. This makes each individual update faster to compute, and since updates happen far more frequently, training overall often progresses more quickly, especially on large datasets where processing the entire dataset for every single update would be impractical.

The trade-off is that because each update is based on only a small batch of data, the path SGD takes toward lower error tends to be noisier and less smooth compared to using the entire dataset for every update. In practice, this noise is not necessarily a bad thing, since it can sometimes help the model avoid getting stuck in certain poor patterns during training. SGD remains a foundational building block that many more advanced optimizers are built upon.

Adam, RMSProp, and More