Investigating Missing Values and Outliers During EDA

Missing values and unusual observations can influence statistical results. EDA helps identify these issues before decisions are made about how to handle them.

Analyzing Missing Values

import pandas as pd

import numpy as np

df = pd.DataFrame({

"Age": [25, 30, np.nan, 40, 35],

"Income": [40000, np.nan, 55000, 70000, 62000],

"Spending": [5000, 7000, 6500, np.nan, 8000]

})

missing_count = df.isnull().sum()

missing_percentage = df.isnull().mean() * 100

missing_summary = pd.DataFrame({

"Missing_Count": missing_count,

"Missing_Percentage": missing_percentage

})

print(missing_summary)

This summary reveals which columns have missing data and the proportion of unavailable values.

The next step is to investigate whether missingness follows a pattern. For example, income may be missing more frequently for a particular customer group. Such patterns can affect the reliability of later analysis.

Detecting Potential Outliers with IQR

sales = pd.Series([

10000, 12000, 11500, 13000, 12500,

14000, 13500, 15000, 90000

])

q1 = sales.quantile(0.25)

q3 = sales.quantile(0.75)

iqr = q3 - q1

lower_bound = q1 - 1.5 * iqr

upper_bound = q3 + 1.5 * iqr

outliers = sales[

(sales < lower_bound) |

(sales > upper_bound)

]

print("Potential outliers:", outliers.tolist())

The IQR method flags observations outside a commonly used statistical range. These observations should be examined in context before they are removed, capped, or retained.

Comparing Distributions Before and After Cleaning

After cleaning, compare summary statistics and visualizations to understand how the changes affected the dataset.

print("Original median:", sales.median())

filtered_sales = sales[

(sales >= lower_bound) &

(sales <= upper_bound)

]

print("Filtered median:", filtered_sales.median())

Filtering is shown here only as an analytical comparison. In a real project, removing an observation should be justified by the objective and data quality requirements rather than by the presence of an outlier alone.