Dimensionality Reduction and Principal Component Analysis

What Is Dimensionality Reduction?

A dataset may contain hundreds or thousands of features. Many features can be redundant, correlated, or unnecessary for a particular analysis.

Dimensionality reduction transforms data into a smaller number of features while attempting to preserve useful information.

It can help with:

• Visualizing complex datasets.
• Reducing computational requirements.
• Removing redundant information.
• Preparing data for downstream machine learning tasks.

Understanding Principal Component Analysis (PCA)

Principal Component Analysis, or PCA, is a widely used dimensionality reduction technique.

PCA creates new features called principal components. These components are combinations of the original features and are ordered according to how much variance they explain.

The first principal component captures the greatest possible variance under PCA's linear transformation. The second captures the greatest remaining variance while being orthogonal to the first, and so on.

PCA does not guarantee that all information relevant to a prediction task is preserved. It is primarily a method for representing variation in the input data.

Implementing PCA in Python

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import load_iris
# Load a sample dataset
data = load_iris()
X = data.data
# Standardize features
X_scaled = StandardScaler().fit_transform(X)
# Reduce four features to two components
pca = PCA(n_components=2)
X_reduced = pca.fit_transform(X_scaled)
print("Original shape:", X.shape)
print("Reduced shape:", X_reduced.shape)
print(
"Explained variance ratio:",
pca.explained_variance_ratio_
)
print(
"Total variance explained:",
pca.explained_variance_ratio_.sum()
)

Interpreting the Results

The original dataset has four features, while the transformed dataset has two principal components.

The explained variance ratio indicates how much of the dataset's variance is represented by each component. Adding the ratios shows the total variance represented by the selected components.

Standardizing the features is important when their scales differ significantly because PCA is sensitive to feature variance.

PCA can also support two-dimensional visualization, but the resulting components may be less intuitive than the original features.