Understanding Data Preparation

What Is Data Preparation?

Data preparation is the process of cleaning, transforming, and organizing data before using it for analysis or machine learning.

Consider a dataset used to predict customer churn. It might contain records like these:

The dataset contains missing values. Other datasets may include duplicate customers, inconsistent contract names, or values stored in the wrong data type.

If these problems are not handled appropriately, they can affect the quality of analysis and model performance.

Common Data Quality Problems

Missing values: Some observations do not contain values for every feature.

Duplicate records: The same observation appears more than once.

Inconsistent formats: For example, a date may appear as 2026-01-10 in one row and 10/01/2026 in another.

Incorrect values: An age field may contain a negative number or an unrealistic value.

Inconsistent categories: Values such as Annual, annual, and Yearly may represent the same category.

Data leakage: A feature contains information that would not legitimately be available when the model makes its prediction.

Irrelevant features: Some columns may not provide useful information for the prediction task.

Exploring Data with Pandas

Pandas is a Python library commonly used to inspect, clean, and transform tabular data.

Install it using:

pip install pandas

Create a small dataset:

import pandas as pd

data = {

"Age": [28, 35, None, 42, 35],

"MonthlySpend": [500, 800, 650, None, 800],

"Contract": ["Monthly", "Annual", "Monthly", "Annual", "Annual"]

}

df = pd.DataFrame(data)

print(df.head())

print(df.info())

print(df.describe(include="all"))

The head() method displays the first few rows, info() provides information about columns and missing values, and describe() summarizes the data.

Checking Missing Values and Duplicates

You can identify missing values and duplicate rows using built-in pandas methods.

print(df.isnull().sum())

print("Duplicate rows:", df.duplicated().sum())

The first statement counts missing values in each column. The second counts rows that duplicate earlier rows across all columns.

Duplicate records should not always be removed automatically. First determine whether they represent accidental repetition or legitimate repeated observations.

Why Data Preparation Matters

Data preparation can improve the reliability of model training, reduce avoidable errors, and make datasets easier to understand.

However, cleaning does not mean changing every unusual value or forcing all data into a particular shape. Transformations should be based on the problem, the meaning of the data, and the assumptions of the chosen algorithm.