Hyperparameter Tuning and Improving Models

Machine learning algorithms have settings that control how they learn. These are called hyperparameters. Examples include the maximum depth of a decision tree, the regularization strength of logistic regression, and the number of neighbors in a K-nearest neighbors model.

Hyperparameter tuning searches for settings that produce better validation performance.

Using GridSearchCV

GridSearchCV evaluates specified combinations of hyperparameters using cross-validation.

from sklearn.model_selection import GridSearchCV

from sklearn.tree import DecisionTreeClassifier

model = DecisionTreeClassifier(random_state=42)

param_grid = {

"max_depth": [2, 4, 6, 8],

"min_samples_split": [2, 5, 10]

}

search = GridSearchCV(

estimator=model,

param_grid=param_grid,

cv=5,

scoring="f1",

n_jobs=-1

)

search.fit(X_train, y_train)

print("Best parameters:", search.best_params_)

print("Best cross-validation F1:", search.best_score_)

best_model = search.best_estimator_

The example assumes a classification dataset and uses F1-score to select the best parameter combination. Choose a scoring metric that matches the project's objective.

For larger search spaces, RandomizedSearchCV can test a limited number of randomly selected combinations instead of evaluating every possible combination.

Preventing Data Leakage

Data leakage happens when information that would not be available at prediction time influences model training or evaluation. It can lead to unrealistically strong results.

Common causes include:

• Using test data to choose hyperparameters.
• Calculating scaling or imputation statistics from the full dataset before splitting.
• Including features that reveal the target indirectly.
• Using future information when predicting past events.

Scikit-learn pipelines help prevent leakage by ensuring preprocessing is fitted separately within each training fold.

from sklearn.pipeline import Pipeline

from sklearn.preprocessing import StandardScaler

from sklearn.linear_model import LogisticRegression

from sklearn.model_selection import GridSearchCV

pipeline = Pipeline([

("scaler", StandardScaler()),

("model", LogisticRegression(max_iter=1000))

])

param_grid = {

"model__C": [0.01, 0.1, 1, 10]

}

search = GridSearchCV(

pipeline,

param_grid,

cv=5,

scoring="f1"

)

search.fit(X_train, y_train)

print("Best parameters:", search.best_params_)

print("Best cross-validation F1:", search.best_score_)

This pipeline standardizes numerical features within the cross-validation process. For categorical or mixed-type data, use appropriate preprocessing steps, such as encoding categorical columns.

A Practical Model Improvement Workflow

A repeatable evaluation process usually follows these steps:

1. Define the prediction objective and select a suitable metric.
2. Split the data appropriately before fitting preprocessing steps.
3. Establish a baseline model.
4. Compare candidate models using cross-validation.
5. Tune hyperparameters using training data.
6. Investigate errors, data quality, and feature relevance.
7. Evaluate the selected model on the untouched test set.
8. Record the dataset version, parameters, metrics, and random seeds.

Model improvement is an iterative process. A more complex model is not automatically better, and the best choice depends on predictive performance, reliability, interpretability, and the cost of incorrect predictions.