Cleaning and Transforming Data

Datasets collected from different sources may contain missing values, duplicate records, inconsistent text, or incorrect data types. Cleaning these issues is essential before drawing conclusions.

Handling Missing Values

Pandas provides methods to identify, remove, or replace missing values.

import pandas as pd

import numpy as np

df = pd.DataFrame({

"Product": ["Laptop", "Tablet", "Phone", "Monitor"],

"Price": [55000, np.nan, 30000, 18000],

"Rating": [4.5, 4.2, np.nan, 4.0]

})

print(df.isnull().sum())

To remove rows containing missing values:

complete_rows = df.dropna()

print(complete_rows)

To fill missing numeric values with a column median:

df["Price"] = df["Price"].fillna(df["Price"].median())

df["Rating"] = df["Rating"].fillna(df["Rating"].median())

print(df)

The appropriate method depends on why the values are missing. Removing records may discard useful information, while replacing values can introduce bias if applied without understanding the data.

Removing Duplicate Records

Duplicate rows can distort totals and averages.

df = pd.DataFrame({

"Product": ["Laptop", "Tablet", "Laptop", "Phone"],

"Price": [55000, 25000, 55000, 30000]

})

df = df.drop_duplicates()

print(df)

By default, drop_duplicates() identifies rows with identical values across all columns. You can use the subset parameter when duplication should be checked using specific columns.

Transforming Column Values

You can standardize text values and create new columns for analysis.

df = pd.DataFrame({

"Product": [" laptop ", "TABLET", " Phone "],

"Price": [55000, 25000, 30000]

})

df["Product"] = df["Product"].str.strip().str.title()

df["Price_With_Tax"] = df["Price"] * 1.18

print(df)

Here, strip() removes surrounding whitespace, title() standardizes capitalization, and a new column calculates the price including 18% tax.