A Reproducible Cleaning Pipeline

What is a cleaning pipeline?

A cleaning pipeline is an ordered set of steps that turns raw input into an analysis-ready dataset.

A practical sequence

Example

import pandas as pd
def load(path):
return pd.read_csv(path)
def standardise(df):
df = df.copy()
df['region'] = df['region'].str.strip().str.title()
df['order_date'] = pd.to_datetime(df['order_date'], errors='coerce')
return df
def remove_exact_duplicates(df):
before = len(df)
df = df.drop_duplicates()
print(f"Removed {before - len(df)} exact duplicate rows")
return df
def validate(df):
required = {'order_id', 'order_date', 'region', 'revenue'}
assert required.issubset(df.columns), "Missing required columns"
assert (df['revenue'] >= 0).all(), "Negative revenue found"
return df
raw = load('sales_raw.csv')
clean = validate(remove_exact_duplicates(standardise(raw)))
clean.to_csv('sales_clean.csv', index=False)

• Each step is a clearly named function.
• assert statements stop the pipeline if expected conditions fail.
• Row counts are printed so changes are visible.

Documentation

Record:

• Source and extraction date
• Field definitions (a data dictionary)
• Transformations applied
• Exclusions and the reasons for them
• Known limitations

Data dictionary example:

A short data dictionary is more useful than a long note saying only "the data was cleaned".

Key takeaways

• Reproducibility protects against silent changes.
• A good pipeline explains what changed, why, and how the output was validated.

Try it: Create a checklist for a repeatable cleaning process and include at least three checks to run after cleaning.