f(x) = \u03a3 \u03b1i * yi * \u27e8xi, x\u27e9 + b<\/strong><\/p>\n\n\n\nWhere xi are support vectors, \u03b1i are their weights, yi are their class labels, and \u27e8xi, x\u27e9 is the inner product between support vector xi and the new point x.<\/p>\n\n\n\n
When the feature mapping \u03c6 is applied, every inner product \u27e8xi, x\u27e9 is replaced by the inner product in the transformed space \u27e8\u03c6(xi), \u03c6(x)\u27e9. The kernel trick observes that for certain transformations, this inner product in the transformed space can be computed directly in the original space by a kernel function:<\/p>\n\n\n\n
K(xi, x) = \u27e8\u03c6(xi), \u03c6(x)\u27e9<\/strong><\/p>\n\n\n\nThe Computational Advantage<\/strong><\/h3>\n\n\n\nInstead of computing the expensive transformation \u03c6(x) for every data point and then computing inner products in the high-dimensional space, the kernel function K(xi, x) computes the same value directly from the original inputs. The transformation is never explicitly performed.<\/p>\n\n\n\n
For the RBF kernel, the implicitly defined feature space is infinite-dimensional, yet the kernel function computes the corresponding inner product with a single simple formula applied to the original data points. This is the power of the kernel trick: infinite-dimensional feature mapping at constant computational cost.<\/p>\n\n\n\n
\n
\ud83d\udca1 Did You Know?<\/strong>\n \n The mathematical foundation behind the kernel trick<\/strong> was established decades before modern machine learning existed. Mercer\u2019s theorem<\/strong>, introduced in 1909<\/strong>, defined the conditions under which a function K(x, z)<\/strong> can behave like an inner product in a higher-dimensional feature space. Later mathematical work by researchers such as Nachman Aronszajn<\/strong> helped formalize reproducing kernel Hilbert spaces, providing the theoretical basis that eventually enabled algorithms like Support Vector Machines (SVMs)<\/strong> to perform powerful nonlinear classification using kernels without explicitly computing high-dimensional transformations.\n <\/p>\n<\/div>\n\n\n\nKernel Functions: RBF, Polynomial, and Beyond<\/strong><\/h2>\n\n\n\nThe choice of kernel function determines the type of decision boundary the non-linear SVM can learn. Each kernel implicitly defines a different feature space and a different set of possible boundary shapes.<\/p>\n\n\n\n
RBF Kernel (Radial Basis Function \/ Gaussian Kernel)<\/strong><\/h3>\n\n\n\nThe RBF kernel, also called the Gaussian kernel, is by far the most widely used kernel for non-linear SVM classification. It is defined as:<\/p>\n\n\n\n
K(x, z) = exp(-\u03b3 ||x – z||^2)<\/strong><\/p>\n\n\n\nThe RBF kernel measures how similar two data points are based on their Euclidean distance. Points that are close together in the original space have a kernel value near 1; points far apart have a kernel value near 0. The parameter \u03b3 (gamma) controls the width of the Gaussian \u2014 how quickly the kernel value decays with distance.<\/p>\n\n\n\n
\u2022 High \u03b3: <\/strong>Narrow Gaussian. Each training point influences only a small local region. The decision boundary closely follows the training data risk of overfitting.<\/p>\n\n\n\n\u2022 Low \u03b3: <\/strong>Wide Gaussian. Each training point influences a large region. The decision boundary is smoother and more global risk of underfitting.<\/p>\n\n\n\nThe RBF kernel produces decision boundaries of arbitrary shape and is the recommended default for non-linear SVM when no domain knowledge suggests a specific kernel.<\/p>\n\n\n\n
Polynomial Kernel<\/strong><\/h3>\n\n\n\nThe polynomial kernel maps the input features to a feature space that includes all polynomial combinations of features up to degree d:<\/p>\n\n\n\n
K(x, z) = (x \u00b7 z + c)^d<\/strong><\/p>\n\n\n\nWhere d is the polynomial degree and c is a free parameter (coef0 in scikit-learn) that controls the influence of higher-degree vs lower-degree terms. A degree-2 polynomial kernel considers all pairwise feature interactions (x1x2, x1^2, x2^2, etc.) implicitly.<\/p>\n\n\n\n
\u2022 Degree 1: <\/strong>Equivalent to a linear kernel produces a linear decision boundary.<\/p>\n\n\n\n\u2022 Degree 2: <\/strong>Captures pairwise feature interactions and quadratic relationships.<\/p>\n\n\n\n\u2022 Degree 3+: <\/strong>More complex boundaries but increasing risk of overfitting with limited data.<\/p>\n\n\n\nPolynomial kernels work well for problems where feature interactions are important \u2014 such as natural language processing tasks where word co-occurrence patterns are meaningful.<\/p>\n\n\n\n