Outlier Detection and Treatment

Outliers are values that sit far from the rest. They can be errors or important discoveries, so handling them well takes care.

a. What Are Outliers?

An outlier is a data point that is very different from the others. For example, one customer spending 5 million in a store where most spend under 5000. Such points can pull averages and distort charts.

Outliers can come from typing mistakes, faulty sensors, or unusual but real events. A fraud transaction is an outlier too, and it may be exactly what you want to find.

Because of this, never remove outliers automatically. First, understand why they exist. The cause decides what you should do with them.

b. Detecting Outliers with IQR and Z-Score

The IQR method uses quartiles. First, find the first quartile (Q1) and the third quartile (Q3). The IQR is Q3 minus Q1. Values below Q1 minus 1.5 times IQR, or above Q3 plus 1.5 times IQR, are marked as outliers.

q1 = df["salary"].quantile(0.25)
q3 = df["salary"].quantile(0.75)
iqr = q3 - q1
lower = q1 - 1.5 * iqr
upper = q3 + 1.5 * iqr
outliers = df[(df["salary"] < lower) | (df["salary"] > upper)]

The Z-score shows how many standard deviations a value is from the mean. A common rule marks values with a Z-score above 3 or below -3 as outliers.

z = (df["salary"] - df["salary"].mean()) / df["salary"].std()
outliers = df[z.abs() > 3]

The Z-score works best for data that is roughly bell shaped. The IQR method is safer for skewed data.

c. Visualizing Outliers

Charts make outliers easy to see. A box plot shows them as dots outside the whiskers. A scatter plot shows them as points far from the main group.

sns.boxplot(data=df, x="salary")
plt.show()

Histograms also help. A long tail on one side often points to extreme values. For two columns, a scatter plot can reveal points that look normal alone but odd together.

Always plot before deciding what to do. A picture often tells you if a value is a mistake or a real but rare case.

d. Deciding Whether to Remove or Keep Outliers

If an outlier is clearly an error, such as an age of 500, fix it or remove it. If it is a real value, think carefully. Removing real extreme values can hide important facts.

A middle option is capping, also called winsorizing. You replace extreme values with the nearest acceptable limit.

df["salary"] = df["salary"].clip(lower=lower, upper=upper)

Another option is a transformation, like a log, which reduces the effect of very large values. You can also use the median instead of the mean, since it is less affected by extremes.

Whatever you choose, document it. Write down which values were changed and why, so your analysis stays clear and honest