Support Vector Machines and K-Nearest Neighbors

What Is a Support Vector Machine?

A support vector machine (SVM) is a supervised learning algorithm that can be used for classification and regression.

For classification, an SVM attempts to find a decision boundary that separates classes. In its maximum-margin formulation, it seeks a boundary with a large margin between the classes, based on the relevant support vectors.

A support vector is a training observation that helps determine the position of the decision boundary.

Understanding Kernels

Some datasets cannot be separated effectively using a straight-line decision boundary in their original feature space.

SVMs can use kernel functions to model more complex boundaries. Common options include linear and radial basis function (RBF) kernels.

An RBF kernel can model nonlinear relationships, but its performance depends on appropriate choices of hyperparameters.

SVMs often benefit from feature scaling, especially when numerical features use very different ranges.

Building an SVM Classifier

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score
iris = load_iris()
X = iris.data
y = iris.target
X_train, X_test, y_train, y_test = train_test_split(
X,
y,
test_size=0.2,
random_state=42,
stratify=y
)
model = make_pipeline(
StandardScaler(),
SVC(kernel="rbf")
)
model.fit(X_train, y_train)
predictions = model.predict(X_test)
print("Accuracy:", accuracy_score(y_test, predictions))

The pipeline ensures that the scaler is fitted using the training data and then applied consistently during prediction.

What Is K-Nearest Neighbors?

K-nearest neighbors (KNN) predicts an output based on nearby training examples.

For classification, it typically selects the most common class among the nearest neighbors. For regression, it commonly averages the target values of nearby observations.

The parameter (k) determines how many neighbors are considered.

For example, with (k=5), a classifier examines the five nearest training observations and uses their class labels to make a prediction.

Building a KNN Classifier

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import accuracy_score
model = make_pipeline(
StandardScaler(),
KNeighborsClassifier(n_neighbors=5)
)
model.fit(X_train, y_train)
predictions = model.predict(X_test)
print("Accuracy:", accuracy_score(y_test, predictions))

KNN is sensitive to feature scaling because it uses distances between observations. Its prediction cost can also become significant for large datasets when many training observations must be examined.

Comparing SVM and KNN

Neither algorithm is universally better. Performance depends on the dataset, the preprocessing steps, and the chosen hyperparameters.