KNN Algorithm in Machine Learning: Complete Guide with Python Code (2026)

Ask any data scientist to name the first algorithm they ever coded from scratch, and there’s a good chance they’ll say KNN.

The K-Nearest Neighbors (KNN) algorithm is one of the simplest, most intuitive algorithms in machine learning: to classify a new point, it just looks at its closest neighbors and goes with the majority.

In this guide, you’ll learn how the KNN Algorithm works step by step, how to implement it from scratch in Python and with scikit-learn, how to pick the right value of K, and how KNN stacks up against Decision Trees and SVMs.

KNN Algorithm in two lines: K-Nearest Neighbors is a supervised learning algorithm that classifies a new data point based on the majority label among its K closest points in the training set, using a distance metric to define “closest.”

Common distance metrics used by the KNN Algorithm:

• Euclidean distance (straight-line distance, the default choice)
• Manhattan distance (distance measured like navigating city blocks)
• Minkowski distance (a generalized form covering both Euclidean and Manhattan)
• Hamming distance (used for categorical or binary data)

TL;DR Summary

• The KNN Algorithm is a “lazy learner”: it stores the training data and does all its work at prediction time, computing distances to find the K closest neighbors.
• For classification, KNN takes a majority vote among the K neighbors; for regression, it averages their values.
• A small K (like 1) tends to overfit, capturing noise; a large K (close to the dataset size) tends to underfit, smoothing out real patterns.
• Feature scaling matters a lot for the KNN Algorithm, since features with larger numeric ranges can dominate distance calculations if you don’t normalize first.
• KNN is easy to implement but slow to query on large datasets and weak in high-dimensional spaces (the curse of dimensionality); Decision Trees and SVMs handle those situations differently.

What are K-Nearest Neighbors (KNN)?

K-Nearest Neighbors, or the KNN Algorithm, is a supervised learning algorithm that can be used for both classification and regression tasks. The core idea behind the KNN Algorithm is simple: to make a prediction for a new data point, it looks at the “K” closest data points in the training set and bases the prediction on those neighbors.

The KNN Algorithm is often described as a “lazy” learning algorithm, and for good reason. Unlike many other algorithms, KNN does not build an explicit model or perform intensive training computations upfront.

There is essentially no training phase; the algorithm simply stores the training data. All the heavy lifting (calculating distances, finding neighbors, etc.) happens at prediction time, when you query the algorithm with a new data point.

This lazy approach means the KNN Algorithm is very easy to implement and understand, but it also implies that prediction can be slow if the dataset is large, since it might need to scan through all training points to make each prediction. We’ll discuss these trade-offs more later.

Did you know?

The basic ideas behind KNN were introduced way back in 1951 by researchers Evelyn Fix and Joseph Hodges, and later expanded by Thomas Cover in 1967. This makes KNN one of the earliest machine learning algorithms. It’s still taught today as a foundational technique due to its simplicity and effectiveness on small problems.

If you want to learn more about how the KNN Algorithm works in machine learning and how it can boost your learning, consider enrolling in HCL GUVI’s Intel and IITM Pravartak Certified Artificial Intelligence and Machine Learning Course that teaches NLP, Cloud technologies, Deep learning, and much more that you can learn directly from industry experts.

How does the KNN algorithm work?

At its heart, the KNN Algorithm can be summarized in a few straightforward steps. Imagine you have a dataset of labeled points (for example, students labeled by whether they passed or failed a course, based on their study hours and sleep hours). Here’s how KNN would approach this:

1. Choose the Value of K

You start by selecting K, the number of neighbors the KNN Algorithm should consider. This is a user-defined number, typically an odd value like 3, 5, or 7 for classification tasks to avoid ties.

2. Measure Distance from the New Data Point

To find which neighbors are “closest,” the algorithm calculates the distance between the new data point and all points in the training set, using one of the distance metrics listed above.

Note: It’s important to normalize your data before calculating distances, or else features with larger scales might dominate.

3. Identify the K Nearest Neighbors

Once all distances are computed, the KNN Algorithm sorts the training points based on proximity to the new input.

• The K closest data points (based on the chosen distance metric) are selected.
• These are the points that will influence the prediction.

Think of this as forming a tight little neighborhood around your query point.

4. Make the Prediction

Now comes the decision-making.

• For classification, the algorithm performs a majority vote among the K neighbors. Whichever class appears most frequently becomes the predicted class.
• For regression, it takes the average of the target values of the K neighbors and uses that as the prediction.

For example, if K=5 and 3 out of 5 neighbors belong to class A, the KNN Algorithm predicts class A for the new point.

Bonus Insight: Weighted Neighbors

In some cases, the KNN Algorithm can be made smarter by weighting neighbors based on their distance, giving closer neighbors more influence than farther ones. This can help reduce noise and improve accuracy.

What Happens Behind the Scenes?

Even though the KNN Algorithm feels simple, here’s what it does at prediction time:

1. Scans through the entire training dataset
2. Calculates the distance between the query point and all training points
3. Sorts the results
4. Picks the top K
5. Aggregates their outputs (majority vote or average)
6. Returns the final prediction

Notice that KNN doesn’t “learn” during training. It just stores the data. All the work happens when you ask it a question, which is why it’s called a lazy learning algorithm.

That’s essentially the whole algorithm. As you can see, no mathematical model fitting or training coefficients are involved: the “model” is just the stored data itself, and the prediction is made by these simple calculations at query time. This simplicity is what makes the KNN Algorithm appealing.

KNN Algorithm in Python: From Scratch and with Scikit-learn

Reading about the steps is one thing; watching the KNN Algorithm run on real numbers makes it click. Here’s the same logic built two ways: a from-scratch implementation using only NumPy, and the production-ready scikit-learn version.

1. Building the KNN Algorithm from Scratch

This version implements every step of the KNN Algorithm manually: distance calculation, sorting neighbors, and majority voting.

import numpy as np
from collections import Counter

class KNNClassifier:
    def __init__(self, k=5):
        self.k = k

    def fit(self, X, y):
        self.X_train = np.array(X)
        self.y_train = np.array(y)

    def _euclidean_distance(self, a, b):
        return np.sqrt(np.sum((a - b) ** 2))

    def _predict_one(self, x):
        distances = [self._euclidean_distance(x, x_train) for x_train in self.X_train]
        k_indices = np.argsort(distances)[:self.k]
        k_nearest_labels = [self.y_train[i] for i in k_indices]
        most_common = Counter(k_nearest_labels).most_common(1)
        return most_common[0][0]

    def predict(self, X):
        X = np.array(X)
        return np.array([self._predict_one(x) for x in X])

Trying it on a tiny toy dataset:

X_train = [[1, 2], [2, 3], [3, 3], [6, 6], [7, 7], [8, 6]]
y_train = [0, 0, 0, 1, 1, 1]

model = KNNClassifier(k=3)
model.fit(X_train, y_train)

print(model.predict([[2, 2], [7, 6]]))

Output:

[0 1]

2. Using Scikit-learn’s KNeighborsClassifier

In practice, you’ll almost always reach for scikit-learn’s KNN Algorithm implementation, which is optimized, well-tested, and includes built-in support for scaling, cross-validation, and different distance metrics.

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import accuracy_score

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

knn = KNeighborsClassifier(n_neighbors=5, metric="euclidean")
knn.fit(X_train_scaled, y_train)

predictions = knn.predict(X_test_scaled)
print("Accuracy:", accuracy_score(y_test, predictions))

Output:

Accuracy: 1.0

Scaling with StandardScaler before fitting is what keeps the KNN Algorithm’s distance calculations fair across features that live on very different numeric scales.

Choosing the Value of K

One important decision when using the KNN Algorithm is selecting an appropriate value for K, the number of neighbors. The choice of K can significantly impact your model’s performance.

If you choose a very small K (like K = 1), the model becomes very sensitive to individual data points. It may capture noise or outliers in the training data, leading to overfitting (high variance, low bias).

On the other hand, choosing a very large K (say K = N, the size of the whole dataset) would make the model overly generalized, essentially averaging everything and ignoring useful local patterns, which can lead to underfitting (high bias).

There is a trade-off in the KNN Algorithm: lower K tends to yield more complex models (flexible but potentially noisy), while higher K yields smoother, more generalized models.

K Value Comparison Table

The table below sums up how different K choices affect the KNN Algorithm’s behavior, and when each range makes sense.

How to Choose the Right K Value in KNN?

Picking K for the KNN Algorithm isn’t guesswork; a few practical techniques make it systematic.

Start with a heuristic. A common rule of thumb is K equal to the square root of the number of training samples, rounded to an odd number for binary classification to avoid tie votes.

Use cross-validation. Split your data into folds, train the KNN Algorithm with different K values (say, 1 through 30), and track validation accuracy or error for each. This is the most reliable way to pick K objectively.

Plot the error curve. Plotting K against validation error typically shows a U-shape: error is high for very small K (overfitting), drops to a minimum, then rises again for very large K (underfitting). The “elbow” near the minimum is usually your best K.

Consider your class count. For multi-class problems, avoid values of K that are exact multiples of the number of classes, since that increases the odds of a tie in the majority vote.

Factor in noise and dataset size. Noisier datasets generally benefit from a slightly larger K to smooth out irregularities, while very clean, well-separated data can tolerate a smaller K.

In scikit-learn, this search is usually automated with GridSearchCV sweeping over a range of n_neighbors values and selecting whichever K gives the best cross-validated score.

Applications of KNN

KNN might be simple, but it has a wide range of applications, especially for straightforward tasks or as a baseline to compare against more complex models. Here are some areas where the KNN Algorithm can be, and has been, applied:

1. Healthcare and Medical Diagnosis (a real-world KNN Algorithm use case)

The KNN Algorithm is frequently applied in predictive diagnostics. Given a patient’s health metrics, it compares them with past cases to predict risks like heart disease or diabetes.

• If most similar patients had a certain condition, the new patient is flagged for the same.
• Works well in scenarios where labeled medical data is available.

For example, KNN has been used in breast cancer detection by comparing tumor characteristics to historical cases.

2. Recommendation Systems (another KNN Algorithm application)

Some basic recommendation engines use the KNN Algorithm to suggest content based on user similarity.

• If you watch a certain set of movies, KNN finds users with similar watch patterns.
• Recommendations come from what your “neighbors” liked.

It’s simple but effective, especially when combined with other models in hybrid systems.

3. Data Imputation (a preprocessing use of the KNN Algorithm)

What if your dataset has missing values? The KNN Algorithm can be used to fill in missing data by looking at similar rows.

• For a missing value, KNN finds K closest rows (based on other features) and averages their values for the missing feature.
• This is called KNN imputation and is popular in data preprocessing pipelines.

4. Finance: Credit Scoring and Stock Prediction (a KNN Algorithm use case)

In the financial domain, the KNN Algorithm can help assess credit risk or forecast market trends.

• For credit scoring, KNN compares a loan applicant to past applicants and predicts if they’re likely to default.
• In stock price analysis, it compares current market conditions to similar historical patterns.

Of course, for large financial datasets, more scalable models are often used, but KNN is great for quick prototyping.

5. Pattern Recognition: Image and Text Classification (a classic KNN Algorithm application)

The KNN Algorithm is commonly used in image recognition and handwriting classification tasks.

• A classic example is MNIST digit classification, where KNN classifies handwritten digits by comparing pixel values.
• In text classification, documents are turned into vectors, and KNN finds the closest topic match based on word usage.

It’s often used as a baseline for benchmarking against more advanced models.

Advantages and Disadvantages of KNN

Like any algorithm, the KNN Algorithm has its pros and cons. It’s important to understand where the KNN Algorithm shines and where it struggles, especially if you’re considering it for a project.

Advantages of KNN

• Simple and Easy to Implement: KNN is about as straightforward as it gets in machine learning. There’s no complex math or optimization under the hood, just distance calculations and counting neighbors.
• No Explicit Training Phase: Since KNN is a lazy learner, you don’t need to spend time training a model (no model parameters are learned). All you do is store the data.
• Versatile, Works for Classification and Regression: KNN naturally handles both classification and regression tasks. The same algorithm can be applied to predict a discrete class or a continuous value by just changing the voting or averaging scheme.
• Reasonably Effective for Low-Dimensional Problems: For small datasets with a few features (dimensions), KNN can perform quite well and often competitively with more complex models, especially if the relationship between features and the target is not too complicated.

Disadvantages of KNN

• Slow for Large Datasets: The flip side of having no training phase is that prediction (query) time in the KNN Algorithm can be slow. In the worst case, to classify one new point, KNN might have to compute distances to every single point in the training dataset. That’s fine for small data, but if you have millions of points, that’s millions of distance calculations per query, which is very computationally expensive.
• Curse of Dimensionality: KNN tends to struggle as the number of features (dimensions) in your data grows large. In high-dimensional space, points tend to all be far apart from each other, a phenomenon often called the curse of dimensionality.
• Sensitive to Noisy or Irrelevant Features: Because KNN uses all features in computing distance, if some features are noisy or not relevant to the outcome, they can negatively impact distance calculations and lead to incorrect neighbor choices.
• Potential for Overfitting or Underfitting Depending on K: The choice of K is critical. As discussed, a small K (like 1) can lead to overfitting; your model memorizes individual points, including noise, and may misclassify new examples that are just noisy variations of one training point.

In summary, the KNN Algorithm is easy to use and can be quite powerful for small, well-structured problems, but it faces challenges with big, high-dimensional, or noisy data.

It’s often used as a baseline or a teaching tool rather than the go-to algorithm for production systems, especially as data grows.

KNN vs Decision Tree vs SVM: When to Use Which?

These three are among the most common classification algorithms you’ll encounter, and each shines in different situations for the KNN Algorithm and its alternatives. Here’s how the KNN Algorithm stacks up against Decision Trees and Support Vector Machines.

If you need a quick, interpretable baseline, the KNN Algorithm is hard to beat for small datasets. If you need explainable rules for stakeholders, a Decision Tree usually wins. If you’re working with high-dimensional or sparse data (like text) and need strong accuracy, SVM is often the better fit.

Quick Quiz – Test Your Understanding

Let’s make the learning interactive! Try answering the following questions to check your understanding of the KNN algorithm.

1. KNN is an example of which type of machine learning?
   A. Supervised Learning
   B. Unsupervised Learning
   C. Reinforcement Learning
   D. Deep Learning
   
2. For regression tasks, how does KNN derive a predicted value for a new data point?
   A. It takes a majority vote among the nearest neighbors’ labels.
   B. It averages the values of the nearest neighbors.
   C. It chooses the value of the single closest neighbor.
   D. It uses a linear regression on the nearest neighbors.
   
3. What is a likely outcome of choosing a very large value of K (say, K = 100) for a KNN classifier on a moderate-sized dataset?
   A. The model may underfit, because it smooths out differences by considering so many neighbors.
   B. The model may overfit to noise in the training data.
   C. The computation time for making predictions will be independent of K.
   D. The decision boundaries become more complex and wiggly.
   
4. Why might KNN be a poor choice for extremely large datasets or very high-dimensional data?
   A. It requires storing and scanning through all training data for each prediction (slow and memory-intensive).
   B. Distances in high dimensions can be misleading (many points end up far apart or equidistant).
   C. It doesn’t perform any feature selection, so irrelevant features can confuse it.
   D. All of the above.
   

Answers: 1: A, 2: B, 3: A, 4: D.

If you want to learn more about how the KNN Algorithm works in machine learning and how it can boost your learning, consider enrolling in HCL GUVI’s Intel and IITM Pravartak Certified Artificial Intelligence and Machine Learning Course that teaches NLP, Cloud technologies, Deep learning, and much more that you can learn directly from industry experts.

Conclusion

In this article, we covered the k-Nearest Neighbors algorithm in machine learning in depth – from its definition and how it works, to tips on choosing the right K and the importance of feature scaling. We discussed where the KNN algorithm in machine learning can be applied, as well as its advantages and limitations. 

KNN’s core philosophy is easy to grasp: “birds of a feather flock together”, meaning points with similar features likely share the same label. This intuitive approach makes KNN a great learning tool and a baseline for comparisons.

Feel free to experiment with KNN on your datasets. Try implementing it, tweak the number of neighbors, and see how it affects the results. And always remember to look at your data – sometimes the simplest method, like KNN, can surprise you with how well it works when its assumptions align with your problem. Happy learning!

FAQs