Menu

Optimization Techniques in Machine Learning

Machine learning models learn from data by adjusting their parameters to improve predictions. This process requires an optimisation algorithm that determines how parameters should change according to a specified objective function.

Different optimisation algorithms use different strategies for computing gradients and updating parameters.

Optimisation in machine learning is the process of finding parameter values that minimise or maximise an objective function, such as prediction error or a regularised training loss.

Common methods include batch gradient descent, stochastic gradient descent, mini-batch gradient descent, momentum-based optimisation, and adaptive algorithms such as Adam.

What Is an Objective Function in Machine Learning?

An objective function mathematically represents what a model is trying to optimise.

For supervised learning, the objective commonly includes a loss function measuring prediction errors.

A general optimisation problem can be expressed as:

\[ \theta^*=\arg\min_{\theta}L(\theta) \]

Here:

  • \(\theta\) represents the model parameters.
  • \(L(\theta)\) represents the loss function.
  • \(\theta^*\) represents a parameter value that minimises the objective.

The selected loss function depends on the learning task.

What Are Common Loss Functions?

Mean Squared Error

Mean Squared Error (MSE) is commonly used in regression.

\[ MSE=\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{y}_i)^2 \]

It calculates the average squared difference between observed and predicted values.

Because errors are squared, larger prediction errors receive greater weight.

Mean Absolute Error

Mean Absolute Error (MAE) calculates the average absolute difference between actual and predicted values.

\[ MAE=\frac{1}{n}\sum_{i=1}^{n}|y_i-\hat{y}_i| \]

MAE is less sensitive to extreme errors than MSE and uses the same units as the target variable.

Binary Cross-Entropy

Binary cross-entropy is commonly used for binary classification.

\[ L=-\frac{1}{n} \sum_{i=1}^{n} \left[ y_i\log(p_i)+(1-y_i)\log(1-p_i) \right] \]

Here:

  • \(y_i\) represents the true binary label.
  • \(p_i\) represents the predicted probability of the positive class.
  • \(n\) represents the number of observations.

The loss penalises predicted probabilities that assign low probability to the correct outcome.

In numerical implementations, stable formulations are important when probabilities approach zero or one.

What Is Batch Gradient Descent?

Batch gradient descent calculates each gradient update using the complete training dataset.

The process involves:

  1. Generating predictions for the complete training dataset.
  2. Calculating the training loss.
  3. Computing the gradient using all training observations.
  4. Updating model parameters.
  5. Repeating until the stopping condition is satisfied.

The update rule is:

\[ \theta_{t+1}=\theta_t-\eta\nabla L(\theta_t) \]

Advantages

  • Uses the full training dataset to calculate each update.
  • Produces deterministic gradients for a fixed dataset and model state.
  • Can provide stable convergence for suitable objectives.

Limitations

  • Each update may be computationally expensive for large datasets.
  • Full-batch processing may require substantial memory or computation.
  • Updates occur less frequently relative to the amount of data processed.

Batch gradient descent is often practical for smaller datasets or optimisation problems where full gradient calculations are inexpensive.

What Is Stochastic Gradient Descent?

Stochastic Gradient Descent (SGD) updates parameters using one training observation at a time.

Instead of calculating the gradient using the entire dataset, it estimates the gradient from an individual example.

The update can be represented as:

\[ \theta_{t+1} = \theta_t-\eta\nabla L_i(\theta_t) \]

Here, \(L_i\) represents the loss associated with a single training observation.

Advantages

  • Supports frequent parameter updates.
  • Can be used with large datasets and streaming observations.
  • Introduces gradient noise that may help optimisation move through some difficult regions.

Limitations

  • Updates can exhibit substantial variation.
  • The loss may fluctuate during training.
  • Learning-rate scheduling may be necessary for reliable convergence.

SGD is commonly used in machine learning and neural network optimisation.

What Is Mini-Batch Gradient Descent?

Mini-batch gradient descent combines characteristics of batch and stochastic gradient descent.

Instead of using the complete dataset or a single observation, it calculates gradients using small groups of training examples.

Common mini-batch sizes include 16, 32, 64, 128, or larger values depending on the model and available hardware.

The update rule can be expressed as:

\[ \theta_{t+1} = \theta_t-\eta\frac{1}{m} \sum_{i\in B_t}\nabla L_i(\theta_t) \]

Here:

  • \(B_t\) represents the current mini-batch.
  • \(m\) represents the number of observations in that batch.
  • \(\eta\) represents the learning rate.

Mini-batching can use vectorised hardware operations efficiently while providing multiple parameter updates per epoch.

An epoch represents one complete pass through the training dataset.

Batch vs Stochastic vs Mini-Batch Gradient Descent

Feature Batch Gradient Descent Stochastic Gradient Descent Mini-Batch Gradient Descent
Data per Update Complete dataset One observation Small group
Update Frequency Low High Moderate
Gradient Variation Low High Moderate
Memory Requirement Can be high Usually low Depends on batch size
Common Application Smaller optimisation problems Online or incremental learning Neural network training

Mini-batch gradient descent is widely used in deep learning because it supports efficient numerical computation on modern hardware.

What Is Momentum in Optimisation?

Momentum is an optimisation technique that incorporates information from previous parameter updates.

Rather than following only the current gradient, momentum maintains a running update direction.

A common formulation is:

\[ v_t=\mu v_{t-1}+\nabla L(\theta_t) \]

\[ \theta_{t+1}=\theta_t-\eta v_t \]

Here:

  • \(v_t\) represents the accumulated update direction.
  • \(\mu\) represents the momentum coefficient.
  • \(\eta\) represents the learning rate.

Momentum can help reduce oscillations and accelerate movement along directions where gradients remain consistent.

It is particularly useful when the loss function has steep curvature in one direction and shallow curvature in another.

What Is the Adam Optimizer?

Adam stands for Adaptive Moment Estimation.

It combines estimates of the first and second moments of gradients to adapt parameter updates.

The first moment tracks an exponentially weighted estimate of recent gradients.

The second moment tracks an exponentially weighted estimate of recent squared gradients.

A simplified representation is:

\[ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t \]

\[ v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 \]

Here:

  • \(g_t\) represents the current gradient.
  • \(m_t\) represents the first-moment estimate.
  • \(v_t\) represents the second-moment estimate.
  • \(\beta_1\) and \(\beta_2\) control exponential decay.

Adam also applies bias correction to the moment estimates before updating parameters.

A common update form is:

\[ \theta_{t+1} = \theta_t- \eta\frac{\hat{m}_t}{\sqrt{\hat{v}_t}+\epsilon} \]

The parameter \(\epsilon\) is a small positive constant used for numerical stability.

Why Is Adam Popular?

Adam is widely used because it adapts parameter-specific update magnitudes and often performs well with relatively little initial tuning.

However, Adam is not universally superior to SGD or other optimisers.

The best choice depends on the objective function, model architecture, dataset, and generalisation requirements.

What Is Regularisation in Machine Learning Optimisation?

Regularisation introduces additional constraints or penalties to discourage undesirable model complexity.

A regularised objective can be expressed as:

\[ L_{total}=L_{prediction}+\lambda R(\theta) \]

Here:

  • \(L_{prediction}\) represents prediction error.
  • \(R(\theta)\) represents the regularisation penalty.
  • \(\lambda\) controls the strength of regularisation.

L1 Regularisation

L1 regularisation penalises the sum of absolute parameter values.

\[ R(\theta)=\sum_j|\theta_j| \]

It can encourage sparse coefficient vectors by shrinking some coefficients to zero.

L2 Regularisation

L2 regularisation penalises the sum of squared parameter values.

\[ R(\theta)=\sum_j\theta_j^2 \]

It discourages excessively large coefficients and can improve numerical stability in certain models.

Regularisation can reduce overfitting, but excessive regularisation may cause underfitting.

How Are Optimization Algorithms Implemented in Python?

Scikit-learn provides implementations of several machine learning algorithms with different optimisation approaches.

For example, SGDRegressor performs regression using stochastic gradient descent.

import numpy as npfrom sklearn.linear_model import SGDRegressorfrom sklearn.preprocessing import StandardScalerfrom sklearn.pipeline import make_pipelineX = np.array([    [1],    [2],    [3],    [4],    [5]])y = np.array([2, 4, 6, 8, 10])model = make_pipeline(    StandardScaler(),    SGDRegressor(        max_iter=1000,        tol=1e-4,        random_state=42    ))model.fit(X, y)prediction = model.predict([[6]])print("Predicted Value:", prediction[0])

The model learns an approximate linear relationship between the input and target.

The pipeline standardises the input features before fitting the regression model.

The prediction for an input of 6 should be approximately 12, although the exact value depends on optimisation settings and convergence.

What Is Hyperparameter Tuning?

Hyperparameters are settings selected before or outside the main parameter-learning process.

Examples include:

  • Learning rate
  • Mini-batch size
  • Momentum coefficient
  • Regularisation strength
  • Number of training epochs
  • Neural network architecture

Hyperparameter tuning evaluates different configurations to identify settings that perform well on validation data.

Common approaches include grid search, random search, and Bayesian optimisation.

Grid Search

Grid search evaluates a predefined collection of hyperparameter combinations.

For example, a model might be evaluated using learning rates of 0.001, 0.01, and 0.1 alongside different regularisation strengths.

Random Search

Random search samples configurations from specified parameter distributions.

It can be more computationally efficient than exhaustive grid search when only some hyperparameters strongly influence performance.

Bayesian Optimisation

Bayesian optimisation uses a model of the objective to guide the selection of promising configurations.

It is useful when evaluating each configuration is computationally expensive.

What Is Early Stopping?

Early stopping terminates iterative training when validation performance stops improving according to a predefined rule.

For example, a neural network may continue reducing training loss while validation loss begins increasing.

This pattern may indicate overfitting.

Early stopping can reduce unnecessary computation and help prevent training beyond the point of best observed validation performance.

However, suitable stopping criteria and patience settings are needed because validation metrics may fluctuate between epochs.

How Do You Choose the Right Optimisation Algorithm?

Optimisation Method Main Characteristic Suitable Scenario
Batch Gradient Descent Uses full-data gradients Small or moderate optimisation problems
SGD Uses one-example gradient estimates Incremental learning
Mini-Batch Gradient Descent Uses groups of observations Neural network training
Momentum SGD Incorporates previous update directions Objectives with oscillatory gradients
Adam Adapts parameter-specific updates Many deep learning tasks
AdamW Uses decoupled weight decay Regularised deep learning training

The choice should reflect computational constraints and validation performance rather than convenience alone.

What Is the Difference Between Optimisation and Generalisation?

Optimisation measures how effectively a model reduces its training objective.

Generalisation describes how well the trained model performs on new, unseen observations.

A model can achieve very low training loss while performing poorly on test data.

This often occurs when the model learns noise or dataset-specific patterns instead of relationships that generalise.

Therefore, optimisation should be evaluated alongside validation performance, model complexity, regularisation, and data quality.

Common Mistakes in Machine Learning Optimisation

  1. Assuming lower training loss always means better model performance.
  2. Selecting an excessively high learning rate.
  3. Ignoring feature scaling in gradient-sensitive models.
  4. Using test data repeatedly during hyperparameter tuning.
  5. Assuming Adam is always better than SGD.
  6. Training for too many epochs without monitoring validation performance.

Data Science Tutorial for Beginners

VA

Vaishali Ardhana



Learn via Video Course

undefined Logo

0 hrs



Get in Touch For Details