Evaluating and Saving a Machine Learning Model

Training a model is only one part of machine learning. Evaluation helps determine whether the model makes useful predictions on unseen observations.

Evaluating a Regression Model

For regression, common metrics include mean absolute error (MAE), mean squared error (MSE), root mean squared error (RMSE), and the coefficient of determination ((R^2)).

from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score

import numpy as np

actual = [50, 60, 70, 80]

predicted = [52, 58, 68, 85]

mae = mean_absolute_error(actual, predicted)

mse = mean_squared_error(actual, predicted)

rmse = np.sqrt(mse)

r2 = r2_score(actual, predicted)

print("MAE:", mae)

print("MSE:", mse)

print("RMSE:", rmse)

print("R-squared:", r2)

MAE measures the average absolute prediction error, while RMSE gives greater weight to larger errors. (R^2) compares the model's predictions with a baseline based on the target mean; it can be negative when the model performs worse than that baseline.

The appropriate metric depends on the cost and meaning of prediction errors.

Evaluating a Classification Model

Classification models can be assessed using accuracy, precision, recall, F1-score, and a confusion matrix.

from sklearn.metrics import accuracy_score, classification_report, confusion_matrix

actual = [0, 1, 1, 0, 1, 0]

predicted = [0, 1, 0, 0, 1, 1]

print("Accuracy:", accuracy_score(actual, predicted))

print("Confusion matrix:")

print(confusion_matrix(actual, predicted))

print("Classification report:")

print(classification_report(actual, predicted, zero_division=0))

Accuracy measures the proportion of correct predictions. Precision measures how many predicted positive cases were actually positive, while recall measures how many actual positive cases were detected. The F1-score balances precision and recall using their harmonic mean.

When classes are imbalanced, accuracy alone may hide poor performance on the minority class.

Saving and Loading a Trained Model

Once a model has been trained and validated, it may be saved for later use.

import joblib

from sklearn.linear_model import LinearRegression

model = LinearRegression()

model.fit(

[[1], [2], [3], [4], [5]],

[30, 40, 50, 60, 70]

)

joblib.dump(model, "student_score_model.joblib")

loaded_model = joblib.load("student_score_model.joblib")

print(loaded_model.predict([[6]]))

The joblib library serializes the trained model to a file and restores it when needed. Only load serialized files from trusted sources, because loading untrusted pickle-based files can execute malicious code.

For production systems, save the preprocessing steps along with the model, track model versions, and validate predictions against representative data. A model's performance should also be monitored after deployment because real-world data can change over time.