Handling Missing Values and Duplicates

Real datasets are rarely perfect. This topic shows you how to find missing values and duplicate rows, and how to fix them without harming your results.

a. Detecting Missing Values

Real data is rarely perfect. Some cells are empty because information was not collected or was lost. Pandas shows these as NaN, which stands for Not a Number.

print(df.isnull().sum())

The line above counts the missing values in each column. This gives you a quick overview of how serious the problem is. You can also use df.info() to compare the non-null count against the total rows.

Before fixing anything, try to understand why the data is missing. A blank might mean zero, unknown, or not applicable. The reason decides the best way to handle it.

b. Dropping and Filling Missing Values

One option is to remove rows or columns with missing data using dropna(). This is simple, but you lose information. It works best when only a few rows are affected.

df_clean = df.dropna()

The other option is to fill the gaps using fillna(). You can fill with a fixed value, the average, the median, or the most common value.

df["age"] = df["age"].fillna(df["age"].median())

The median is often safer than the mean when the data has extreme values. For text columns, you can fill with a label like "Unknown".

c. Finding and Removing Duplicates

Duplicate rows can come from data entry errors or from joining files. They can make totals and counts wrong. Always check for them.

print(df.duplicated().sum())
df = df.drop_duplicates()

The duplicated() method marks repeated rows. The drop_duplicates() method removes them and keeps the first copy. You can also check duplicates in selected columns using subset.

df = df.drop_duplicates(subset=["email"])

Be careful when deciding what counts as a duplicate. Two customers can share a name but still be different people. Pick columns that truly identify a unique record.

d. Best Practices for Missing Data

Never delete or fill data without thinking. Look at how much data is missing in each column. If a column is mostly empty, it may be better to drop the whole column.

Keep a copy of your original data before making changes. Work on a copy with df.copy() so you can always go back. Write down the steps you took so others can follow your logic.

Also check the result after cleaning. Compare the row count, summary statistics, and a few sample rows before and after. This helps you catch accidental mistakes early.