Removing Duplicates and Standardizing Data

Duplicate records and inconsistent text values can distort counts, summaries, and comparisons. Standardizing the dataset makes it easier to group and analyze records consistently.

Identifying and Removing Duplicates

import pandas as pd

df = pd.DataFrame({

"Customer_ID": [101, 102, 103, 101],

"Name": ["Aman", "Priya", "Ravi", "Aman"],

"Purchase": [1200, 2500, 1800, 1200]

})

print(df.duplicated())

To remove completely duplicated rows:

df = df.drop_duplicates()

print(df)

Sometimes, two rows share the same customer ID but contain different transaction details. In such cases, they should not automatically be treated as duplicates.

You can identify duplicates using selected columns:

duplicate_customers = df[df.duplicated(

subset=["Customer_ID"],

keep=False

)]

print(duplicate_customers)

This identifies repeated customer IDs without assuming that all their transactions are identical.

Standardizing Text Values

Text fields may contain inconsistent capitalization, leading or trailing spaces, or slightly different representations of the same category.

df = pd.DataFrame({

"City": [" Delhi ", "delhi", "CHENNAI", "Chennai "]

})

df["City"] = df["City"].str.strip().str.title()

print(df)

The strip() method removes surrounding whitespace, and title() standardizes capitalization. After cleaning, " Delhi " and "delhi" become "Delhi".

For more complex categorical data, you may need explicit mapping rules to handle abbreviations, spelling mistakes, and alternate names.

city_mapping = {

"Madras": "Chennai",

"Bombay": "Mumbai"

}

df["City"] = df["City"].replace(city_mapping)

Standardization is particularly important before grouping records or creating charts based on categories.