Optimization and Gradient Descent

What Is Optimization in AI?

Optimization is the process of finding parameter values that improve a defined objective.

In machine learning, a model makes predictions and compares them with the actual values. A loss function measures the error between predictions and actual outcomes.

The training process attempts to find model parameters that minimize this loss.

For example, if a model predicts house prices inaccurately, its training algorithm can adjust the model's weights and bias to reduce prediction error on the training data.

Understanding a Loss Function

A loss function assigns a numerical value to the error made by a model.

One commonly used loss function for regression is mean squared error (MSE):

[
MSE = \frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{y}_i)^2
]

Where:

• (n) is the number of observations.
• (y_i) is the actual value.
• (\hat{y}_i) is the predicted value.

Squaring the errors makes them non-negative and penalizes larger errors more heavily.

For example, if the actual value is 10 and the model predicts 8, the squared error is:

[
(10-8)^2 = 4
]

If the prediction is 5, the squared error is:

[
(10-5)^2 = 25
]

The second prediction has a larger squared error.

What Is Gradient Descent?

Gradient descent is an optimization algorithm that updates model parameters in a direction intended to reduce the loss.

The basic update rule is:

[
\theta_{\text{new}} = \theta_{\text{old}} - \eta \nabla J(\theta)
]

Where:

• (\theta) represents the model parameters.
• (\eta) is the learning rate.
• (J(\theta)) is the loss function.
• (\nabla J(\theta)) is the gradient of the loss.

The gradient indicates how the loss changes as the parameters change. The algorithm moves in the negative-gradient direction to reduce the loss locally.

Understanding the Learning Rate

The learning rate controls the size of each parameter update.

• A learning rate that is too small can make training slow.
• A learning rate that is too large can cause the algorithm to overshoot useful parameter values or fail to converge.
• A suitable learning rate helps the algorithm make useful progress toward lower loss.

The best learning rate depends on the model, the dataset, and the optimization problem.

6.5 A Simple Gradient Descent Example

Consider the function:
[
J(w) = (w-3)^2
]
The minimum occurs at (w=3), where the loss is zero.
The derivative is:
[
\frac{dJ}{dw} = 2(w-3)
]
We can use gradient descent to move a starting value of (w) toward 3.
w = 0.0
learning_rate = 0.1
for step in range(20):
gradient = 2 * (w - 3)
w = w - learning_rate * gradient
print("Optimized w:", round(w, 4))
print("Final loss:", round((w - 3) ** 2, 6))

Output:

Optimized w: 2.9654

Final loss: 0.0012

The parameter moves closer to 3 with each update. After 20 steps, the value is close to the minimum, although it has not reached it exactly.

How Optimization Supports Machine Learning

Optimization algorithms are used to train many types of machine learning models, including neural networks.

During training, the model typically:

1. Produces predictions from input data.
2. Calculates a loss value.
3. Computes gradients for its parameters.
4. Updates parameters using an optimization algorithm.
5. Repeats the process across training batches and iterations.

Gradient descent and its variants, such as stochastic gradient descent and Adam, are widely used in deep learning.

Practice Exercise

Modify the previous Python program to use a learning rate of 0.05 and run it for 20 steps.

Compare the final value of w and the final loss with the original program. Explain why changing the learning rate affects the speed of convergence.