Duplicates, Inconsistent Labels, and Data Types

Why consistency matters

The same category recorded differently is counted as separate groups:

Without cleaning, a "revenue by city" report would show five rows for one city.

Duplicates

Always define what a duplicate means for the table's unit of analysis (grain).

df.duplicated().sum() # exact duplicates
df.duplicated(subset=['order_id']).sum() # duplicate keys
df[df.duplicated(subset=['order_id'], keep=False)] # view all copies

Standardisation

• Specify date formats explicitly. 03/04/2026 means 3 April in India but 4 March in the US.
• Preserve original values in a separate column when the transformation could change interpretation or an audit trail is needed.

Example

• Before removing duplicate order IDs, check the table's grain:
  • One row per order → repeated IDs may be errors.
  • One row per item → repeated IDs are expected.
• Removing repeated order IDs from a line-item table would delete valid items and undercount revenue.

Key takeaways

• A cleaning rule must fit the data's structure and business meaning.
• Count affected rows and validate totals after each important transformation.

Try it: Write a cleaning plan for a table containing inconsistent region names, duplicate-looking order IDs, and dates stored in mixed formats.