Hyperparameter Tuning and Model Selection

What Are Hyperparameters?

Hyperparameters are settings chosen before or during the model selection process rather than learned directly as ordinary model parameters during fitting.

Examples include:

• Maximum tree depth.
• Number of trees in a random forest.
• Number of neighbors in KNN.
• Regularization strength in logistic regression.
• Learning rate in some optimization algorithms.

Appropriate hyperparameter choices can improve model performance and help control complexity.

Parameters vs Hyperparameters

Model parameters are learned from training data.

For example, linear regression learns weights and a bias.

Hyperparameters control how the model is structured or trained. For example, the maximum depth of a decision tree is a hyperparameter.

Understanding this distinction helps explain why training a model and tuning its configuration are separate activities.

Grid Search

Grid search evaluates a defined collection of hyperparameter combinations using a specified validation strategy.

For example, you might compare several values of max_depth for a decision tree.

from sklearn.datasets import load_iris
from sklearn.model_selection import (
train_test_split,
GridSearchCV
)
from sklearn.tree import DecisionTreeClassifier
iris = load_iris()
X = iris.data
y = iris.target
X_train, X_test, y_train, y_test = train_test_split(
X,
y,
test_size=0.2,
random_state=42,
stratify=y
)
model = DecisionTreeClassifier(random_state=42)
param_grid = {
"max_depth": [2, 3, 4, 5, None],
"min_samples_leaf": [1, 2, 4]
}
search = GridSearchCV(
estimator=model,
param_grid=param_grid,
cv=5,
scoring="accuracy"
)
search.fit(X_train, y_train)
print("Best parameters:", search.best_params_)
print("Best CV accuracy:", round(search.best_score_, 3))
print("Test accuracy:", round(search.score(X_test, y_test), 3))

Grid search evaluates the parameter combinations using five-fold cross-validation on the training set. It then reports the best configuration according to the selected metric.

The test set is used only after tuning to estimate the final model's performance.

Randomized Search

Grid search can become expensive when there are many hyperparameters or candidate values.

Randomized search samples a specified number of combinations from defined parameter distributions or candidate lists. It can explore a large search space with a fixed computational budget.

The choice between grid search and randomized search depends on the size of the search space, available computing resources, and how thoroughly you need to explore different configurations.

Avoiding Overfitting During Model Selection

Repeatedly trying different configurations and choosing the one with the best test score indirectly trains your decisions on the test set.

This can make the final performance estimate overly optimistic.

A better approach is to:

1. Split the data into training and test sets.
2. Use cross-validation or a validation set to select model configurations.
3. Fit the chosen model using the appropriate training data.
4. Evaluate it once on the reserved test set.
5. Use additional monitoring after deployment.

For small datasets, nested cross-validation may be useful when an unbiased estimate of the entire model-selection process is required.

Choosing a Final Model

The model with the highest score is not always the best operational choice.

You may also need to consider:

• Prediction latency.
• Training time.
• Memory usage.
• Interpretability.
• Fairness and reliability.
• Cost of false positives and false negatives.
• Ease of deployment and maintenance.

A simpler model may be preferable when it performs sufficiently well and is easier to explain, monitor, or maintain.