Monitoring, Data Drift, and Model Maintenance

Deploying a model is not the end of the machine learning lifecycle. Models must be monitored because real-world conditions can change.

What Is Data Drift?

Data drift occurs when the statistical properties of incoming data change compared with the data used during training.

For example, a demand forecasting model trained on historical shopping patterns may receive very different inputs after a major change in customer behavior.

Concept drift refers to a change in the relationship between the input features and the target. Even if the input distribution remains similar, the relationship the model needs to learn may change.

What Should Be Monitored?

• Prediction quality: Accuracy or other metrics when actual outcomes become available.
• Data quality: Missing values, invalid inputs, and unexpected feature ranges.
• Latency: The time taken to return predictions.
• Availability: Whether the prediction service is operational.
• Resource usage: CPU, memory, and GPU consumption.
• Prediction patterns: Unexpected changes in the distribution of outputs.

Monitoring prediction patterns alone cannot prove that a model's accuracy has declined. Actual performance evaluation generally requires trustworthy outcome labels or another suitable validation process.

Retraining Models

A model may need retraining when its performance deteriorates or the underlying data changes.

A typical maintenance workflow is:

1. Detect a potential issue.
2. Investigate the data and model behavior.
3. Collect or validate updated training data.
4. Train a candidate model.
5. Compare it against the current model.
6. Deploy the new version only after appropriate checks.
7. Monitor the updated system.

Retraining should not be automatic without safeguards. New data can contain errors or introduce unwanted biases, so updated models must be evaluated before release.