Menu

Classification Algorithms in Machine Learning

Classification algorithms predict categories or class probabilities from input features.

For example, a financial institution may classify transactions as potentially fraudulent or legitimate, while a telecom company may predict whether a customer is likely to churn.

Classification is a supervised learning technique used to predict discrete target categories or estimate probabilities associated with those categories.

Common algorithms include logistic regression, decision trees, random forests, K-Nearest Neighbours, and Support Vector Machines.

What Are Binary and Multiclass Classification?

Binary Classification

Binary classification predicts one of two classes.

Examples include:

  • Fraud or legitimate
  • Spam or not spam
  • Churn or retention

The target is commonly represented as:

\[ y\in\{0,1\} \]

Multiclass Classification

Multiclass classification predicts one of three or more mutually exclusive categories.

For example, a support ticket may be classified as billing, technical, account, or general support.

Multilabel classification is different because one observation may belong to several categories simultaneously.

What Is Logistic Regression?

Despite its name, logistic regression is commonly used for classification.

It models the probability of a binary outcome.

First, a linear score is calculated:

\[ z=\beta_0+\beta_1x_1+\cdots+\beta_px_p \]

The sigmoid function converts the score into a probability:

\[ P(y=1|X)=\frac{1}{1+e^{-z}} \]

The output lies between 0 and 1.

A decision threshold converts the probability into a class prediction.

For example, with a threshold of 0.50:

\[ \hat{y}= \begin{cases} 1 & P(y=1|X)\geq0.50\\ 0 & P(y=1|X)<0.50 \end{cases} \]

The appropriate threshold depends on the consequences of false positives and false negatives.

Logistic Regression in Python

import numpy as npfrom sklearn.linear_model import LogisticRegressionX = np.array([    [1, 1000],    [2, 1500],    [3, 2500],    [5, 5000],    [8, 8000],    [10, 10000]])y = np.array([1, 1, 1, 0, 0, 0])model = LogisticRegression(    max_iter=1000)model.fit(X, y)probability = model.predict_proba(    [[4, 3500]])print(probability)

The model estimates probabilities for both classes.

The example uses a small illustrative dataset. Practical classification models require more representative training data and proper evaluation.

What Is a Decision Tree Classifier?

A decision tree classifies observations using a hierarchy of feature-based rules.

For example:

  • Is account age below six months?
  • Has the customer experienced payment failures?
  • Has monthly usage declined?

Each split divides observations into smaller groups.

Decision trees commonly use impurity measures such as Gini impurity or entropy.

Gini Impurity

\[ Gini=1-\sum_{k=1}^{K}p_k^2 \]

Here, \(p_k\) is the proportion of observations belonging to class \(k\).

A Gini impurity of zero indicates that all observations in a node belong to one class.

Decision trees are interpretable but can overfit when allowed to grow without sufficient constraints.

What Is Random Forest Classification?

Random forest classification combines predictions from many decision trees.

Each tree learns from a bootstrap sample, with additional feature randomness during splitting.

The final class prediction may be selected by majority vote or by aggregating class probabilities.

Random forests often provide good performance for structured datasets containing nonlinear relationships.

They also support feature-importance estimation, although importance scores do not establish causal influence.

What Is K-Nearest Neighbours?

K-Nearest Neighbours (KNN) predicts a class using nearby observations in feature space.

For example, if most of a customer's five nearest neighbours belong to the churn class, KNN may predict churn.

A common distance metric is Euclidean distance:

\[ d(x,y)=\sqrt{\sum_{i=1}^{p}(x_i-y_i)^2} \]

The hyperparameter \(k\) determines how many neighbours are considered.

KNN is sensitive to feature scale because variables with larger numerical magnitudes may dominate distance calculations.

Standardisation is therefore often useful.

What Is a Support Vector Machine?

A Support Vector Machine (SVM) identifies a decision boundary between classes.

For linearly separable data, a linear SVM seeks a separating hyperplane with a large margin.

A hyperplane can be represented as:

\[ w^Tx+b=0 \]

Kernel methods allow SVMs to construct nonlinear decision boundaries.

SVMs can work well on certain high-dimensional datasets but may require careful feature scaling and parameter tuning.

Classification Algorithms Compared

Algorithm Main Strength Limitation
Logistic Regression Interpretable probability model Linear decision boundary in supplied feature space
Decision Tree Understandable decision rules May overfit
Random Forest Captures nonlinear relationships More difficult to interpret
KNN Simple similarity-based approach Sensitive to scaling and dataset size
SVM Effective decision boundaries Can require substantial tuning

What Is a Confusion Matrix?

A confusion matrix compares actual classes with predicted classes.

For binary classification:

Predicted Positive Predicted Negative
Actual Positive True Positive (TP) False Negative (FN)
Actual Negative False Positive (FP) True Negative (TN)

The matrix reveals different types of classification errors.

This matters because incorrectly rejecting a legitimate transaction and failing to identify fraud may have different costs.

What Are Precision, Recall, and F1-Score?

Precision

\[ Precision=\frac{TP}{TP+FP} \]

Precision measures the proportion of positive predictions that are correct.

Recall

\[ Recall=\frac{TP}{TP+FN} \]

Recall measures the proportion of actual positive observations correctly identified.

F1-Score

\[ F1=2\frac{Precision\times Recall}{Precision+Recall} \]

F1-score balances precision and recall using their harmonic mean.

Accuracy

\[ Accuracy=\frac{TP+TN}{TP+TN+FP+FN} \]

Accuracy measures the overall proportion of correct predictions.

However, it can be misleading for highly imbalanced datasets.

Classification Evaluation in Python

from sklearn.metrics import (    confusion_matrix,    classification_report)actual = [1, 0, 1, 1, 0, 0]predicted = [1, 0, 1, 0, 0, 1]print(confusion_matrix(actual, predicted))print(classification_report(actual, predicted))

Confusion Matrix Output:

[[2 1]
 [1 2]]

The model correctly identifies two positives and two negatives, while making one false-positive and one false-negative prediction.

What Are ROC-AUC and PR-AUC?

ROC-AUC summarises the relationship between true-positive and false-positive rates across classification thresholds.

PR-AUC summarises the precision-recall relationship across thresholds.

Precision-recall analysis is often particularly informative for rare positive classes such as fraud.

The exact interpretation depends on how the area is computed, including distinctions between average precision and trapezoidal area under the precision-recall curve.

What Is Class Imbalance?

Class imbalance occurs when some classes have substantially more observations than others.

For example:

  • Legitimate transactions: 99,000
  • Fraudulent transactions: 1,000

A model predicting every transaction as legitimate achieves 99% accuracy but detects no fraud.

Approaches include:

  • Class weighting
  • Resampling training data
  • Threshold adjustment
  • Appropriate evaluation metrics
  • Probability calibration

Resampling must occur only within training data, not across the complete dataset before evaluation.

Where Is Classification Used?

Classification supports fraud detection, medical risk screening, customer churn prediction, spam filtering, sentiment analysis, and quality inspection.

Common mistakes: Relying only on accuracy, ignoring class imbalance, treating model probabilities as perfectly calibrated, using inappropriate thresholds, and introducing leakage through preprocessing.


Data Science Tutorial for Beginners

VA

Vaishali Ardhana



Learn via Video Course

undefined Logo

0 hrs



Get in Touch For Details