Handling Missing Data

Missing data occurs when a value has not been recorded, is unavailable, or cannot be determined. Missing values may appear as NaN, None, or blank fields, depending on how the data was collected and imported.

Detecting Missing Values

import pandas as pd
import numpy as np
df = pd.DataFrame({
"Name": ["Aman", "Priya", "Ravi", "Neha"],
"Age": [25, 30, np.nan, 28],
"Salary": [45000, np.nan, 52000, 48000]
})
print(df.isnull())
print(df.isnull().sum())

The isnull() method identifies missing entries, while sum() counts the missing values in each column.

Removing Missing Values

You can remove rows or columns containing missing data when doing so is appropriate.

complete_rows = df.dropna()
print(complete_rows)
To remove columns that contain missing values:
cleaned_columns = df.dropna(axis=1)
print(cleaned_columns)

Removing missing records can reduce the amount of usable data. If missing values are concentrated in a particular group, deleting them may also introduce bias.

Filling Missing Values

Another approach is to replace missing values with suitable estimates or predefined values.

df["Age"] = df["Age"].fillna(df["Age"].median())
df["Salary"] = df["Salary"].fillna(df["Salary"].median())
print(df)

The median is often useful for numerical data when extreme values could distort the mean. For categorical columns, the most frequent category or an explicit "Unknown" label may be appropriate.

The correct strategy depends on why the data is missing and how the variable will be used. For machine learning, calculate replacement values from the training data and apply the same fitted transformation to validation and test data to prevent data leakage.