Evaluating AI Model Performance

Why Is Model Evaluation Important?

Model evaluation measures how well an AI system performs on data it has not seen during training. It helps determine whether the model is suitable for its intended purpose.

A model that performs well on training data may fail on new examples because of overfitting, data quality issues, or differences between training and real-world data.

Training, Validation, and Test Data

Datasets are commonly divided into three parts:

• Training set: Used to learn model parameters.
• Validation set: Used to compare models and tune hyperparameters.
• Test set: Used for final evaluation after model selection.

Keeping these datasets separate helps produce a more realistic estimate of performance.

Common Evaluation Metrics

The appropriate metric depends on the task.

For classification:

• Accuracy: The proportion of predictions that are correct.
• Precision: The proportion of predicted positive cases that are actually positive.
• Recall: The proportion of actual positive cases that the model identifies.
• F1-score: The harmonic mean of precision and recall.

For regression:

• Mean Absolute Error (MAE): The average absolute difference between predictions and actual values.
• Mean Squared Error (MSE): The average squared prediction error.
• Root Mean Squared Error (RMSE): The square root of MSE, expressed in the same units as the target.

Accuracy can be misleading when classes are imbalanced. For example, a model that predicts every transaction as legitimate may appear accurate when fraudulent transactions are extremely rare, while failing to identify fraud.

Example: Evaluating a Classification Model

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import (
accuracy_score,
classification_report
)
data = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
data.data,
data.target,
test_size=0.2,
random_state=42,
stratify=data.target
)
model = RandomForestClassifier(random_state=42)
model.fit(X_train, y_train)
predictions = model.predict(X_test)
print("Accuracy:", accuracy_score(y_test, predictions))
print(classification_report(y_test, predictions))

The classification report displays precision, recall, F1-score, and support for each class. These metrics provide more information than accuracy alone.