Implementing Agglomerative Clustering using Sklearn

Agglomerative clustering is a hierarchical clustering method in machine learning that identifies hidden patterns in unlabeled datasets. Unlike K-Means clustering, it follows a bottom-up approach in which each data point starts as its own cluster before merging into larger groups.

Using sklearn’s Agglomerative Clustering and scipy dendrogram makes hierarchical clustering easier to implement.

In this article, you will learn how agglomerative clustering works, different linkage methods like ward linkage and complete linkage, how cluster merging happens, and how to implement Agglomerative Clustering using Sklearn with Python examples.

TL;DR

1. Agglomerative clustering is a hierarchical algorithm following a bottom-up approach.
2. Every data point starts as its own cluster before cluster merging begins.
3. sklearn AgglomerativeClustering helps implement hierarchical clustering in Python.
4. Different linkage methods, like ward linkage and complete linkage, affect cluster formation.
5. A scipy dendrogram visualizes hierarchical relationships between clusters.
6. Agglomerative clustering works well for unsupervised clustering tasks without labels.

What is Agglomerative Clustering?

Agglomerative clustering is a hierarchical clustering method used for unsupervised tasks. It starts with each data point as an individual cluster. The algorithm continuously merges clusters until all points belong to one cluster or a certain condition is met.

Because it builds from smaller to larger clusters, it is often called bottom-up clustering.

Unlike k-means clustering, agglomerative clustering does not need to set cluster centers. Instead, it relies on distance measurements and linkage methods to determine how clusters merge.

Understanding Hierarchical Clustering

Hierarchical clustering builds a hierarchy of clusters. It mainly comes in two forms:

Agglomerative Hierarchical Clustering

This approach starts with individual points and merges them step by step.

Divisive Hierarchical Clustering

This approach starts with one large cluster and breaks it down into smaller clusters.

Among these two methods, agglomerative clustering is more popular because it is easier to use and more efficient.

How Bottom-Up Clustering Works

Bottom-up clustering involves these steps:

1. Treat each data point as a separate cluster.
2. Calculate the distance matrix for all clusters.
3. Identify the two closest clusters.
4. Merge those clusters.
5. Update the distance matrix.
6. Repeat until the desired number of clusters is formed.

The way the algorithm measures distances between clusters depends on the chosen linkage methods.

Curious about how these concepts work? Download HCL GUVI’s free AI ebook to learn more about machine learning concepts, Agglomerative Clustering, and real-world AI applications. 

How Cluster Merging Happens in Agglomerative Clustering

Cluster merging is the core process behind agglomerative clustering. Initially, every data point acts as an individual cluster. The algorithm then calculates the distance between clusters and merges the two closest clusters.

After every merge, the distance matrix gets updated to reflect the newly formed cluster. This process continues repeatedly until the required number of clusters is achieved.

The merging behavior depends heavily on the selected linkage methods:

1. Ward linkage focuses on minimizing cluster variance.
2. Complete linkage uses the maximum distance between clusters.
3. Single linkage uses the minimum distance between clusters.
4. Average linkage considers the average distance between all points.

Because of this step-by-step merging structure, agglomerative clustering is considered a bottom-up clustering technique.

Types of Linkage Methods

Linkage methods determine how the distance between clusters is calculated during merging.

Ward Linkage

Ward linkage minimizes the variance between clusters, merging them in a way that maintains compact and balanced clusters. It is one of the most commonly used methods in sklearn’s AgglomerativeClustering.

Complete Linkage

Complete linkage measures the maximum distance between points in two clusters. This method creates tighter clusters and is less affected by noise.

Single Linkage

Single linkage uses the minimum distance between clusters. While it can identify irregular shapes, it is very sensitive to outliers.

Average Linkage

Average linkage calculates the average distance between all pairs of points in two clusters. It balances complete and single linkage.

While hierarchical clustering follows a bottom-up cluster merging approach, algorithms like K-Means Clustering use centroid-based grouping methods to create clusters differently in machine learning tasks. 

What is a Distance Matrix?

A distance matrix records the pairwise distances between all data points or clusters.

During agglomerative clustering, the algorithm updates the distance matrix after each merging step. The distance matrix plays a crucial role in deciding which clusters to merge next.

Common distance metrics include:

1. Euclidean distance
2. Manhattan distance
3. Cosine distance

What is a Dendrogram?

A dendrogram is a tree-like diagram that visualizes hierarchical clustering. It helps us understand:

1. The order of cluster merging
2. The similarity between clusters
3. The optimal number of clusters

Using a scipy dendrogram makes it easier to analyze how hierarchical clustering forms groups.

Implementing Agglomerative Clustering using Sklearn

Now, let us implement agglomerative clustering with sklearn’s AgglomerativeClustering.

Step 1: Import the Required Libraries

import numpy as np

import matplotlib.pyplot as plt

from sklearn.datasets import make_blobs

from sklearn. cluster import AgglomerativeClustering

Step 2: Create a Sample Dataset

X, y = make_blobs(

n_samples=300,

centers=4,

cluster_std=1.2,

random_state=42

)

This dataset includes four distinct groups.

Step 3: Apply Agglomerative Clustering

model = AgglomerativeClustering(

n_clusters=4,

linkage=’ward’

)

labels = model.fit_predict(X)

Here:

1. n_clusters sets the number of clusters.
2. linkage=’ward’ uses ward linkage.
3. fit_predict assigns cluster labels.

Step 4: Visualize the Clusters

plt.scatter(X[:, 0], X[:, 1], c=labels)

plt.title(‘Agglomerative Clustering’)

plt.xlabel(‘Feature 1’)

plt.ylabel(‘Feature 2’)

plt.show()

The visualization shows how sklearn’s AgglomerativeClustering grouped similar data points.

Visualizing Clusters using a Scipy Dendrogram

A scipy dendrogram helps visualize the hierarchical clustering process.

Step 1: Import Scipy Libraries

from scipy.cluster.hierarchy import dendrogram, linkage

Step 2: Create the Linkage Matrix

linked = linkage(X, method=’ward’)

Step 3: Plot the Dendrogram

plt.figure(figsize=(10, 5))

dendrogram(linked)

plt.title(‘Dendrogram’)

plt.xlabel(‘Data Points’)

plt.ylabel(‘Distance’)

plt.show()

The dendrogram visually represents the operations of cluster merging at various distance levels.

Advantages of Agglomerative Clustering

Agglomerative clustering has several advantages:

1. Does not require cluster centroids
2. Works well with smaller datasets
3. Produces clear dendrograms
4. Useful when the number of clusters is unknown
5. Supports different linkage methods

Limitations of Agglomerative Clustering

Despite its benefits, agglomerative clustering has some limitations.

High Computational Cost

Hierarchical clustering can be expensive for large datasets since the distance matrix is updated continuously.

Sensitive to Noise

Some linkage methods are very sensitive to outliers.

Difficult to Undo Merges

Once clusters merge, the algorithm cannot go back.

Real World Applications of Agglomerative Clustering

Agglomerative clustering is used widely across various fields.

Customer Segmentation

Businesses cluster customers based on their buying behavior.

Document Clustering

Search engines and recommendation systems group similar documents together.

Image Segmentation

Computer vision applications use bottom-up clustering for grouping regions in images.

Bioinformatics

Researchers apply dendrogram structures for gene analysis and DNA clustering.

Conclusion

Agglomerative clustering is a strong hierarchical clustering algorithm that identifies hidden structures in unlabeled data. By using a bottom-up approach, it effectively analyzes relationships between clusters.

With sklearn’s AgglomerativeClustering, implementing hierarchical clustering is straightforward, even for newcomers. Features like linkage methods, distance matrix calculations, and scipy dendrogram visualizations make this method clear and interpretable.

If you want to strengthen your machine learning foundation, learning about agglomerative and hierarchical clustering is essential, as they lay the groundwork for many advanced unsupervised methods.

Want to explore more machine learning concepts with practical examples? Check out HCL GUVI’s AI and Machine Learning programs designed for beginners and professionals.

FAQs

1. What is agglomerative clustering in machine learning?

Agglomerative clustering is a hierarchical clustering method where each data point starts as its own cluster before gradually merging with nearby clusters.

2. What is the difference between hierarchical clustering and K-Means clustering?

Hierarchical clustering builds clusters through continuous merging, while K-Means clustering divides data into fixed groups using cluster centroids.

3. What are linkage methods in agglomerative clustering?

Linkage methods determine how distances between clusters are calculated during cluster merging, including ward linkage, complete linkage, single linkage, and average linkage.

4. What is the purpose of a dendrogram in hierarchical clustering?

A dendrogram visualizes hierarchical clustering by showing cluster relationships, cluster merging order, and the optimal number of clusters.

5. Why is sklearn AgglomerativeClustering widely used?

sklearn AgglomerativeClustering provides a simple and efficient way to implement hierarchical clustering using multiple linkage methods and clustering configurations.

6. When should you use agglomerative clustering?

Agglomerative clustering is useful when the number of clusters is unknown and when understanding relationships between clusters is important in unsupervised clustering tasks.