Menu

Regression Algorithms in Machine Learning

Regression algorithms predict continuous numerical outcomes using relationships between input features and target variables.

For example, a logistics company may estimate delivery duration using route distance, traffic conditions, and vehicle type.

Regression is a supervised machine learning technique that predicts numerical target values from one or more input features.

Common algorithms include linear regression, polynomial regression, decision tree regression, random forest regression, and gradient boosting regression.

What Is Linear Regression?

Linear regression models a numerical target as a linear combination of input variables.

For one feature:

\[ \hat{y}=\beta_0+\beta_1x \]

Here:

  • \(\hat{y}\) is the prediction.
  • \(x\) is the input feature.
  • \(\beta_0\) is the intercept.
  • \(\beta_1\) is the coefficient.

Consider a hypothetical sales model:

\[ \hat{y}=5000+3x \]

For advertising expenditure of ₹10,000:

\[ \hat{y}=5000+3(10000)=35000 \]

The predicted sales revenue is ₹35,000.

This represents a fitted relationship, not necessarily a causal effect of advertising.

What Is Multiple Linear Regression?

Multiple linear regression uses several input variables.

\[ \hat{y}=\beta_0+\beta_1x_1+\beta_2x_2+\cdots+\beta_px_p \]

For example, a house price model may use:

  • Property area
  • Number of bedrooms
  • Property age
  • Distance from the city centre

Each coefficient represents the model's estimated change in the target associated with one feature, holding the other included features constant.

Multiple Linear Regression in Python

import pandas as pdfrom sklearn.linear_model import LinearRegressiondf = pd.DataFrame({    "area": [1000, 1500, 2000, 2500, 3000],    "bedrooms": [2, 3, 3, 4, 4],    "price": [40, 55, 70, 85, 100]})X = df[["area", "bedrooms"]]y = df["price"]model = LinearRegression()model.fit(X, y)prediction = model.predict(    pd.DataFrame({        "area": [2200],        "bedrooms": [3]    }))print(prediction)

The model predicts a numerical price using both input variables.

The example is instructional. Its small dataset is insufficient for reliable real-world price estimation.

How Does Linear Regression Learn Coefficients?

A common method is Ordinary Least Squares (OLS).

OLS selects coefficients that minimise the sum of squared residuals.

A residual is:

\[ e_i=y_i-\hat{y}_i \]

The optimisation objective is:

\[ \min_{\beta}\sum_{i=1}^{n}(y_i-\hat{y}_i)^2 \]

The resulting coefficients minimise squared training errors among the candidate linear models.

Linear regression can be solved using numerical linear algebra methods.

Gradient-based optimisation is also used for certain regression models and large-scale training tasks.

What Is Polynomial Regression?

Polynomial regression introduces powers of input variables to capture curved relationships.

For example:

\[ \hat{y}=\beta_0+\beta_1x+\beta_2x^2 \]

The model is nonlinear in \(x\) but remains linear in its coefficients.

Polynomial features are useful when the relationship cannot be adequately described by a straight line.

Python Example

import numpy as npfrom sklearn.pipeline import make_pipelinefrom sklearn.preprocessing import PolynomialFeaturesfrom sklearn.linear_model import LinearRegressionX = np.array([[1], [2], [3], [4], [5]])y = np.array([3, 7, 13, 21, 31])model = make_pipeline(    PolynomialFeatures(degree=2),    LinearRegression())model.fit(X, y)print(model.predict([[6]]))

Output:

[43.]

Increasing polynomial degree may improve training fit but can also increase overfitting risk.

What Is Decision Tree Regression?

Decision tree regression divides the feature space into regions using conditional splits.

Each terminal region produces a numerical prediction, commonly based on the average target value of its training observations.

For example, a delivery-time tree may split observations using route distance and traffic conditions.

Decision trees can capture nonlinear relationships and feature interactions.

However, unrestricted trees may overfit training data.

What Is Random Forest Regression?

Random forest regression combines predictions from multiple decision trees.

Each tree is trained using a bootstrap sample of the training observations, and feature randomness is introduced during splitting.

The final regression prediction is typically the average of the individual tree predictions.

\[ \hat{y}=\frac{1}{T}\sum_{t=1}^{T}\hat{y}_t \]

Here, \(T\) represents the number of trees.

Python Example

from sklearn.ensemble import RandomForestRegressormodel = RandomForestRegressor(    n_estimators=100,    random_state=42)model.fit(X, y)predictions = model.predict(X)

Random forests can capture nonlinear patterns and often require less feature scaling than distance-based algorithms.

However, they may be computationally expensive and usually do not extrapolate beyond the range of observed target values as linear models can.

What Is Gradient Boosting Regression?

Gradient boosting builds predictive models sequentially.

Each new model attempts to improve the current ensemble by reducing the selected loss.

Tree-based gradient boosting is widely used for structured tabular datasets.

Common implementations include Scikit-learn's gradient boosting models, XGBoost, LightGBM, and CatBoost.

Unlike random forests, which build trees with substantial independence, boosting develops an additive model through sequential updates.

How Are Regression Models Evaluated?

Mean Absolute Error

\[ MAE=\frac{1}{n}\sum_{i=1}^{n}|y_i-\hat{y}_i| \]

MAE measures average absolute prediction error in the target's units.

Mean Squared Error

\[ MSE=\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{y}_i)^2 \]

MSE penalises larger errors more heavily because errors are squared.

Root Mean Squared Error

\[ RMSE=\sqrt{MSE} \]

RMSE returns error magnitude in the same units as the target.

R-Squared

\[ R^2=1- \frac{\sum(y_i-\hat{y}_i)^2} {\sum(y_i-\bar{y})^2} \]

R² measures performance relative to predicting the mean of the evaluation targets.

It can be negative on test data when a model performs worse than that baseline.

Regression Evaluation in Python

from sklearn.metrics import (    mean_absolute_error,    mean_squared_error,    r2_score)import numpy as npactual = np.array([100, 150, 200, 250])predicted = np.array([110, 140, 190, 260])mae = mean_absolute_error(actual, predicted)rmse = np.sqrt(mean_squared_error(actual, predicted))r2 = r2_score(actual, predicted)print("MAE:", mae)print("RMSE:", rmse)print("R2:", r2)

Output:

MAE: 10.0
RMSE: 10.0
R2: 0.968

The model has an average absolute error of 10 units in this example.

Regression Algorithms Compared

Algorithm Main Strength Limitation
Linear Regression Simple and interpretable Limited to linear combinations of supplied features
Polynomial Regression Models curved relationships May overfit at high degrees
Decision Tree Captures nonlinear splits Can be unstable and overfit
Random Forest Combines multiple trees Less interpretable
Gradient Boosting Strong tabular predictive performance Requires careful tuning

Where Is Regression Used?

Regression supports:

  • House price estimation
  • Demand forecasting
  • Delivery time prediction
  • Revenue prediction
  • Energy consumption forecasting
  • Equipment remaining-life estimation

Common mistakes: Evaluating on training data, assuming relationships are causal, ignoring large residuals, selecting models solely through R², and using random splits for time-dependent forecasting problems.


Data Science Tutorial for Beginners

VA

Vaishali Ardhana



Learn via Video Course

undefined Logo

0 hrs



Get in Touch For Details