Outliers and Validation Rules

What is an outlier?

An outlier is a value that differs markedly from most observations. It may be:

Statistical rules flag candidates; context decides what to do.

Validation checks

Example

A product price of 0 could be:

• A data error
• A free sample
• A promotional item
• A placeholder for an unpriced product

Compare it with product records and transaction context before changing it.

How it works

Separate detection from treatment:

1. Detect: flag suspicious values with rules.
2. Investigate: check the source and context.
3. Treat: choose one of:
   • Correct it from a trusted source
   • Keep it, with an explanation
   • Exclude it from a specific calculation
   • Escalate it to the data owner
4. Document: record the rule, the number of records affected, and the impact.

Common mistakes

• Deleting every extreme value, which removes important incidents or high-value customers.
• Not recording the rule used.
• Not measuring the effect of the decision on the final result.
• Fixing data silently without telling the data owner, so the source problem continues.

Try it: Create five validation rules for a sales dataset and explain what you would do when each rule fails.