Detecting and Handling Outliers

An outlier is an observation that differs substantially from the overall pattern of a dataset. Outliers may result from data entry errors, unusual but valid events, or natural variation.

Identifying outliers is important because extreme values can strongly affect means, standard deviations, and some machine learning algorithms.

Using the Interquartile Range (IQR)

The interquartile range measures the spread of the middle 50% of the data.

[
IQR = Q_3 - Q_1
]

Where:

• (Q_1) is the first quartile.
• (Q_3) is the third quartile.

A common rule flags observations below (Q_1 - 1.5 \times IQR) or above (Q_3 + 1.5 \times IQR).

import pandas as pd

sales = pd.Series([

1200, 1500, 1350, 1600, 1450,

1550, 1400, 1700, 1800, 15000

])

q1 = sales.quantile(0.25)

q3 = sales.quantile(0.75)

iqr = q3 - q1

lower_bound = q1 - 1.5 * iqr

upper_bound = q3 + 1.5 * iqr

outliers = sales[

(sales < lower_bound) |

(sales > upper_bound)

]

print("Lower bound:", lower_bound)

print("Upper bound:", upper_bound)

print("Potential outliers:", outliers.tolist())

The value 15000 is likely to be flagged because it is much larger than the other observations.

Deciding What to Do with Outliers

There are several ways to handle outliers, depending on the analytical objective.

• Investigate: Check whether the observation is a recording error or a valid event.
• Remove: Exclude the record only when there is a defensible reason.
• Cap values: Limit extreme values to selected thresholds when the methodology supports it.
• Use robust methods: Consider medians or algorithms that are less sensitive to extreme values.

For example, capping values at an upper threshold can be performed as follows:

capped_sales = sales.clip(

lower=lower_bound,

upper=upper_bound

)

print(capped_sales)

Capping changes the original values, so retain the raw data and document the transformation. Outlier detection rules should also be chosen carefully for skewed distributions and datasets containing naturally extreme observations.