Using Cross-Validation for Reliable Evaluation

A single train-test split can produce results that depend heavily on which records end up in each subset. Cross-validation reduces this dependence by evaluating a model across multiple splits of the training data.

In k-fold cross-validation, the training data is divided into k folds. The model is trained on all but one fold and evaluated on the remaining fold. This process repeats until each fold has been used for validation.

Performing Cross-Validation

from sklearn.model_selection import cross_val_score, KFold

from sklearn.linear_model import LinearRegression

model = LinearRegression()

cv = KFold(

n_splits=5,

shuffle=True,

random_state=42

)

scores = cross_val_score(

model,

X_train,

y_train,

cv=cv,

scoring="neg_mean_absolute_error"

)

mae_scores = -scores

print("MAE for each fold:", mae_scores)

print("Average MAE:", mae_scores.mean())

Scikit-learn returns negative values for loss metrics such as MAE when used through cross_val_score, because its scoring convention treats higher scores as better. Negating the results gives positive MAE values.

For classification, StratifiedKFold is often a good choice because it attempts to preserve class proportions across folds.

from sklearn.model_selection import (

StratifiedKFold,

cross_val_score

)

from sklearn.linear_model import LogisticRegression

model = LogisticRegression(max_iter=1000)

cv = StratifiedKFold(

n_splits=5,

shuffle=True,

random_state=42

)

scores = cross_val_score(

model,

X_train,

y_train,

cv=cv,

scoring="f1"

)

print("F1 scores:", scores)

print("Average F1:", scores.mean())

Cross-validation helps compare models more reliably. However, the final test set should remain separate until the model and its settings have been selected.

For time-series data, use a time-aware validation strategy instead of randomly shuffling observations, since future information must not leak into training.