Menu

Eigenvalues and Eigenvectors in Data Science

Large datasets frequently contain numerous numerical features, some of which provide overlapping information. Processing every feature independently may increase computational requirements and make patterns difficult to interpret.

Eigenvalues, eigenvectors, and matrix decomposition provide mathematical techniques for analysing transformations and identifying important directions within numerical data.

Eigenvalues and eigenvectors describe how certain vectors behave under linear transformations, while matrix decomposition expresses matrices through simpler mathematical components.

These concepts are important in dimensionality reduction, Principal Component Analysis (PCA), data compression, and recommendation systems.

What Are Eigenvalues and Eigenvectors?

Consider a square matrix \(A\) representing a linear transformation.

An eigenvector is a nonzero vector whose direction remains unchanged or is reversed when the transformation is applied.

The corresponding eigenvalue determines the scaling factor.

Mathematically:

\[ Av=\lambda v \]

Here:

  • \(A\) represents a square matrix.
  • \(v\) represents a nonzero eigenvector.
  • \(\lambda\) represents the corresponding eigenvalue.

If the eigenvalue is positive, the transformed vector retains its direction.

If the eigenvalue is negative, its direction reverses.

If the eigenvalue is zero, the corresponding eigenvector is mapped to the zero vector.

How Do Eigenvalues and Eigenvectors Work?

Consider the matrix:

\[ A= \begin{bmatrix} 2&0\\ 0&3 \end{bmatrix} \]

Take the vector:

\[ v_1= \begin{bmatrix} 1\\ 0 \end{bmatrix} \]

Multiplying:

\[ Av_1= \begin{bmatrix} 2\\ 0 \end{bmatrix} \]

Therefore:

\[ Av_1=2v_1 \]

The vector is an eigenvector with eigenvalue 2.

Now consider:

\[ v_2= \begin{bmatrix} 0\\ 1 \end{bmatrix} \]

Then:

\[ Av_2= \begin{bmatrix} 0\\ 3 \end{bmatrix} \]

Therefore:

\[ Av_2=3v_2 \]

The second vector has eigenvalue 3.

This example demonstrates how a matrix scales different eigenvector directions by different amounts.

How Are Eigenvalues Calculated?

Eigenvalues are obtained by solving the characteristic equation.

\[ \det(A-\lambda I)=0 \]

Here:

  • \(A\) represents the square matrix.
  • \(I\) represents the identity matrix.
  • \(\lambda\) represents an eigenvalue.
  • \(\det\) represents the determinant.

For the matrix:

\[ A= \begin{bmatrix} 2&0\\ 0&3 \end{bmatrix} \]

The characteristic equation becomes:

\[ (2-\lambda)(3-\lambda)=0 \]

Therefore:

\[ \lambda_1=2 \]

\[ \lambda_2=3 \]

Each eigenvalue can then be used to solve for its corresponding eigenvectors.

How Can Eigenvalues and Eigenvectors Be Calculated in Python?

NumPy provides numerical linear algebra functions.

import numpy as npA = np.array([    [2, 0],    [0, 3]])eigenvalues, eigenvectors = np.linalg.eig(A)print("Eigenvalues:", eigenvalues)print("Eigenvectors:")print(eigenvectors)

Output:

Eigenvalues: [2. 3.]
Eigenvectors:
[[1. 0.]
 [0. 1.]]

The function returns eigenvalues and corresponding eigenvectors.

Eigenvector signs and ordering may vary because an eigenvector can be multiplied by a nonzero scalar and remain a valid eigenvector.

For real symmetric matrices, np.linalg.eigh() is often more appropriate because it uses algorithms designed for Hermitian or symmetric matrices.

What Is Matrix Decomposition?

Matrix decomposition involves expressing a matrix as a product of simpler matrices.

Different decomposition techniques serve different purposes.

Decomposition Mathematical Form Common Application
Eigendecomposition \(A=V\Lambda V^{-1}\) Analysing diagonalizable square matrices
Singular Value Decomposition \(A=U\Sigma V^T\) Dimensionality reduction
QR Decomposition \(A=QR\) Solving least-squares problems
LU Decomposition \(PA=LU\) Solving systems of linear equations
Cholesky Decomposition \(A=LL^T\) Positive-definite matrix computations

Each technique requires suitable mathematical conditions.

For example, eigendecomposition in the displayed form requires a diagonalizable matrix, whereas SVD is defined for general real matrices.

What Is Singular Value Decomposition?

Singular Value Decomposition (SVD) expresses a matrix as the product of three matrices.

\[ A=U\Sigma V^T \]

Here:

  • \(U\) contains left singular vectors.
  • \(\Sigma\) contains singular values.
  • \(V^T\) contains transposed right singular vectors.

Singular values are non-negative and describe the scaling associated with the corresponding singular directions.

SVD is used in data compression, dimensionality reduction, and low-rank approximation.

Why Is SVD Useful?

Consider a dataset containing thousands of numerical features.

Some features may be highly correlated, meaning the dataset contains redundant information.

SVD can approximate the original matrix using a smaller number of dominant components.

A rank-\(k\) approximation can be written as:

\[ A_k=U_k\Sigma_kV_k^T \]

The approximation retains the largest \(k\) singular values and their corresponding singular vectors.

This can reduce storage requirements and computational complexity.

However, retaining fewer components also removes information from the original matrix.

What Is Principal Component Analysis?

Principal Component Analysis (PCA) is a dimensionality reduction technique that transforms numerical features into new orthogonal directions called principal components.

The first principal component captures the greatest variance in the centred dataset.

The second captures the greatest remaining variance subject to being orthogonal to the first.

Subsequent components follow the same principle.

How Does PCA Work?

A simplified PCA workflow contains five stages.

  1. Prepare the Data: Select suitable numerical features and handle missing values.
  2. Centre the Features: Subtract feature means so the variables are centred around zero.
  3. Calculate Principal Directions: Use eigendecomposition of the covariance matrix or SVD of the centred data.
  4. Select Components: Choose the number of components based on explained variance and the analytical objective.
  5. Transform the Dataset: Project observations onto the selected component directions.

Feature standardisation may also be necessary when variables use substantially different units or scales.

What Is Explained Variance in PCA?

Explained variance indicates how much of the dataset's total variance is captured by each principal component.

For covariance-matrix PCA, the explained variance ratio of component \(i\) can be expressed as:

\[ EVR_i=\frac{\lambda_i}{\sum_{j=1}^{p}\lambda_j} \]

Here, \(\lambda_i\) represents the eigenvalue associated with the component.

Consider a hypothetical PCA result.

Principal Component Explained Variance
PC1 55%
PC2 25%
PC3 12%
PC4 8%

The first two components together explain 80% of the total variance.

A data scientist may retain these two components when that level of variance preservation is suitable for the analysis.

However, explained variance does not automatically indicate predictive usefulness.

How Is PCA Implemented in Python?

Scikit-learn provides a PCA implementation.

import numpy as npfrom sklearn.preprocessing import StandardScalerfrom sklearn.decomposition import PCAX = np.array([    [10, 100, 5],    [20, 200, 8],    [30, 300, 12],    [40, 400, 15]])scaler = StandardScaler()X_scaled = scaler.fit_transform(X)pca = PCA(n_components=2)X_reduced = pca.fit_transform(X_scaled)print("Original Shape:", X.shape)print("Reduced Shape:", X_reduced.shape)print("Explained Variance:", pca.explained_variance_ratio_)

The example transforms three numerical features into two principal components.

The original dataset has shape (4, 3).

The transformed dataset has shape (4, 2).

In a predictive modelling workflow, scaling and PCA must be fitted on training data only to prevent leakage into validation or test results.

Where Are Eigenvalues and Matrix Decomposition Used?

Recommendation Systems

A user-item interaction matrix can represent customer ratings or purchasing preferences.

Matrix factorisation approximates this information using lower-dimensional representations of users and items.

The resulting representations can support recommendation ranking.

Image Compression

Images can be represented numerically as matrices.

SVD can approximate image matrices using fewer singular components, reducing the information needed for an approximate reconstruction.

Data Visualisation

PCA can transform high-dimensional numerical datasets into two or three principal components.

These components can then be visualised to examine clusters, trends, or unusual observations.

Noise Reduction

Low-rank approximations can remove some variation associated with smaller singular values.

However, important information can also exist in lower-variance directions, so dimensionality reduction must be evaluated carefully.

What Are the Limitations of PCA?

PCA is useful but has several limitations.

  • PCA captures directions of maximum variance, which may not correspond to predictive importance.
  • Standard PCA represents linear combinations of features and may not capture complex nonlinear structures.
  • Components can be more difficult to interpret than original variables.
  • Results may be sensitive to feature scaling and extreme observations.
  • Dimensionality reduction may remove information relevant to the target variable.

Therefore, PCA should be selected based on the analytical objective rather than applied automatically.

Common Mistakes in PCA and Matrix Decomposition

  1. Confusing eigenvalues with eigenvectors.
  2. Assuming all square matrices can be diagonalised.
  3. Applying PCA to features with incompatible scales without examining standardisation.
  4. Treating high explained variance as proof of strong predictive performance.
  5. Fitting PCA before separating training and test data.

Data Science Tutorial for Beginners

VA

Vaishali Ardhana



Learn via Video Course

undefined Logo

0 hrs



Get in Touch For Details