Understanding Data Quality Issues

Data cleaning is the process of identifying and correcting problems in a dataset before analysis. Real-world data often contains missing values, duplicate records, inconsistent formats, incorrect data types, and unusual measurements.

Poor data quality can lead to misleading visualizations, incorrect statistics, and unreliable machine learning models. Cleaning data helps ensure that analytical results are based on consistent and meaningful information.

Identifying Common Data Problems

Consider a dataset containing customer information.

import pandas as pd

import numpy as np

df = pd.DataFrame({

"Customer": ["Aman", "Priya", "Ravi", "Neha", "Aman"],

"Age": [25, 30, np.nan, 150, 25],

"City": ["Delhi", "Chennai", "Mumbai", "delhi ", "Delhi"],

"Purchase": [1200, 2500, 1800, -500, 1200]

})

print(df)

This dataset contains several potential issues:

• A missing age value.
• An age of 150 that may be invalid for the intended analysis.
• Inconsistent city names, such as "Delhi" and "delhi ".
• A negative purchase amount that requires investigation.
• A repeated customer record.

Not every unusual value is necessarily an error. For example, a negative transaction amount may represent a refund. Before modifying data, understand what each column represents and what values are valid in the business context.

Inspecting Data Quality

Pandas provides methods for examining missing values, duplicate rows, and column types.

print(df.info())

print(df.isnull().sum())

print("Duplicate rows:", df.duplicated().sum())

print(df.describe(include="all"))

These checks provide an initial view of the dataset. They can reveal incomplete columns, repeated records, unexpected types, and suspicious numerical values.