Loss Functions and Optimizers

A loss function tells a model exactly how far off its prediction was from the correct answer. An optimizer then takes that information and decides how to adjust the model's weights to do better next time.

Gradient descent is the underlying idea most optimizers rely on, gradually moving toward fewer mistakes step by step. Together, these two pieces drive the entire training process of a neural network.

Common Loss Functions

A loss function measures how far a model's prediction is from the correct answer. A lower loss means the model is making better predictions.

• Mean Squared Error (MSE): Commonly used for regression problems. It calculates the average of the squared differences between the predicted values and the actual values. Squaring the differences gives more importance to larger errors.
• Cross-Entropy Loss: Commonly used for classification problems, where the model predicts a category or class. It measures how well the predicted probabilities match the correct class.

What is an Optimizer?

An optimizer is the algorithm responsible for actually updating a model's weights based on the gradients it calculates. Adam/RMSProp adjust learning rates dynamically and are more robust across scenarios.

Adam has become the go-to choice for most deep learning projects today, since it tends to train faster and more reliably than older optimizers. 

Gradient Descent Explained

Gradient descent is the core idea behind how most optimizers actually work. Gradient descent navigates the multidimensional surface formed by the loss function. It uses the slope at each point to move the weights in the direction that reduces the loss, ideally converging to a minimum.

A helpful way to picture it is walking down a foggy mountain, at each step you can only feel the slope right under your feet, so you keep taking small steps downhill until you reach the lowest point you can find.