Handling Missing Values and Duplicate Data

What Are Missing Values?

A missing value indicates that a field does not contain an available value.

Missing data may occur because:

• A user skipped a form field.
• A sensor failed to record a measurement.
• A data import was incomplete.
• A value was not applicable to a particular observation.
• Information was unavailable when the dataset was collected.

The appropriate treatment depends on why the value is missing and how important the feature is.

Removing Missing Values

One option is to remove rows or columns with missing values.

import pandas as pd
df = pd.DataFrame({
"Age": [25, 30, None, 40],
"Income": [30000, None, 50000, 60000]
})
cleaned_df = df.dropna()
print(cleaned_df)

dropna() removes rows containing missing values by default.

This method can be appropriate when only a small number of records are affected and removing them does not introduce significant bias. However, deleting too much data can reduce the dataset size or remove important groups of observations.

Filling Missing Values

Another option is to replace missing values with an appropriate estimate.

Common strategies include:

• Mean imputation: Replace missing numerical values with the mean.
• Median imputation: Replace missing numerical values with the median.
• Mode imputation: Replace missing values with the most frequent category or value.
• Constant imputation: Replace missing values with a chosen constant, where appropriate.

Example using pandas:

df["Age"] = df["Age"].fillna(df["Age"].median())
df["Income"] = df["Income"].fillna(df["Income"].mean())
print(df)

The median is often preferable for skewed numerical data because it is less affected by extreme values. The mean can be suitable for numerical data with a relatively balanced distribution and limited extreme observations.

The correct choice depends on the meaning and distribution of the data.

Avoiding Data Leakage During Imputation

When preparing data for machine learning, calculate imputation values using the training data only.

For example, if you calculate the median using the entire dataset before splitting it, information from the test set influences the preprocessing step.

A safer approach is to use a scikit-learn pipeline. It learns the median from the training data and applies that same value to validation, test, and future observations.

from sklearn.impute import SimpleImputer
imputer = SimpleImputer(strategy="median")
X_train_clean = imputer.fit_transform(X_train)
X_test_clean = imputer.transform(X_test)

Here, fit_transform() learns the imputation values from the training data and transforms it. transform() applies the learned values to the test data without fitting again.

Removing Duplicate Records

Duplicate rows may distort analysis by causing some observations to be counted more than once.

df = df.drop_duplicates()

If a dataset contains a customer ID, you may need to check duplication using that identifier instead of checking whether every column matches.

df = df.drop_duplicates(subset=["CustomerID"])

Use this only if the dataset's rules establish that each customer should have one record. In transaction datasets, multiple records for the same customer may be legitimate.

When Should You Remove or Impute Data?

These are guidelines, not universal rules. Always consider why the data is missing and whether the treatment could introduce bias.