Missing Values and Their Meaning

Why can data be missing?

The reason matters because missingness can be related to the outcome you're studying.

Types of missingness:

How to inspect missingness

• Count missing values by column.
• Calculate the percentage missing.
• Check missingness by subgroup: time period, region, device, customer segment.
• Visualise with a heatmap of missing values (e.g., the missingno library).

df.isna().mean().sort_values(ascending=False) # % missing per column
df.groupby('device')['income'].apply(lambda s: s.isna().mean()) # by subgroup

Ways to handle missing values

Example

• If income is missing more often among respondents who decline sensitive questions (MNAR), filling every gap with the overall average creates a misleading picture.
• Better: report the missing share, compare responders with non-responders on other variables, and state whether the remaining sample may be biased.

Common mistakes

• Assuming missing means zero.
• Not documenting how many values were affected.
• Not comparing results before and after a major imputation decision.
• Imputing the target variable in a model.