Building a Reusable Data Cleaning Workflow

A repeatable cleaning workflow helps ensure that the same rules are applied consistently whenever new data arrives. Instead of manually repeating operations, you can organize common cleaning tasks into a function.

Creating a Cleaning Function

Consider a customer dataset with inconsistent text, missing ages, and repeated records.

import pandas as pd

def clean_customer_data(data):

data = data.copy()

Standardize customer names

data["Name"] = data["Name"].str.strip().str.title()

Convert age values to numeric

data["Age"] = pd.to_numeric(

data["Age"],

errors="coerce"

)

Remove completely duplicated rows

data = data.drop_duplicates()

Remove records with missing names

data = data.dropna(subset=["Name"])

return data

The function makes a copy of the input DataFrame so that the original data is not directly modified. It then applies a sequence of cleaning operations and returns the processed dataset.

Applying the Workflow

customers = pd.DataFrame({

"Name": [" aman ", "PRIYA", "Ravi", " aman "],

"Age": ["25", "30", "unknown", "25"],

"City": ["Delhi", "Chennai", "Mumbai", "Delhi"]

})

cleaned_customers = clean_customer_data(customers)

print(cleaned_customers)

print(cleaned_customers.isnull().sum())

The function standardizes names, converts invalid age entries to missing values, and removes completely duplicated rows. It does not automatically fill missing ages because that decision should depend on the intended analysis.

Validating the Final Dataset

After cleaning, perform a final quality check.

print(cleaned_customers.info())

print(cleaned_customers.describe(include="all"))

print("Duplicate rows:", cleaned_customers.duplicated().sum())

A reliable workflow should also document the cleaning rules, preserve the original source data, and record important changes. For machine learning pipelines, transformations should be fitted only on the training dataset and then applied consistently to other datasets.