Data Cleaning and Type Conversion

Raw data is rarely ready to use. This topic shows you how to fix wrong data types, clean column names, and make values consistent so your analysis can be trusted.

a. Why Data Cleaning Matters

Data cleaning is the step where you fix errors and make the data usable. Analysts often spend more than half of their project time on it. Clean data leads to correct results, and messy data leads to wrong ones.

Common problems include wrong data types, spelling differences, extra spaces, and impossible values like a negative age. Even one bad column can change an average or a total. That is why cleaning comes before any analysis.

A good habit is to inspect the data first using info(), describe(), and value_counts(). Note every problem you find. Then fix them one by one and check the result after each change.

b. Fixing Data Types

Pandas sometimes reads numbers as text, especially when a column has symbols like commas or currency signs. A column stored as text cannot be used for math. You need to convert it first.

df["price"] = df["price"].str.replace(",", "")
df["price"] = pd.to_numeric(df["price"], errors="coerce")

The errors="coerce" option turns values that cannot be converted into NaN instead of stopping with an error. This is useful when a column has a few bad entries. You can then decide how to handle those NaN values.

You can also use astype() for simple conversions, like df["age"].astype(int). Use pd.to_datetime() for dates. Always check df.dtypes after converting to confirm the change worked.

c. Renaming and Standardizing Columns

Column names from real files are often messy. They may have capital letters, spaces, or special characters. This makes them hard to type and easy to get wrong.

df.columns = (
df.columns.str.strip()
.str.lower()
.str.replace(" ", "_")
)

The code above removes extra spaces, converts names to lowercase, and replaces spaces with underscores. After this, Customer Name becomes customer_name. It is a small step that saves a lot of trouble later.

To rename only a few columns, use the rename() method with a dictionary. For example, df.rename(columns={"cust_id": "customer_id"}). Keep names short, clear, and consistent across all your files.

d. Cleaning Inconsistent Values

Text columns often contain the same value written in different ways. For example, "Delhi", "delhi", and "Delhi " are all the same city, but Pandas sees three values. This breaks grouping and counting.

df["city"] = df["city"].str.strip().str.title()
df["city"] = df["city"].replace({"Bombay": "Mumbai"})

Use value_counts() to find these differences. It lists every unique value with its count. Spelling mistakes and odd entries are easy to spot in that list.

You should also check for impossible values, such as ages above 120 or dates in the future. Use filters to find them. Then correct, remove, or mark them, and note what you did.