Building a Data Preprocessing Pipeline

What Is a Preprocessing Pipeline?

A preprocessing pipeline combines data transformation steps into a consistent workflow.

Instead of separately cleaning, encoding, and scaling data every time, you can define the required steps and reuse them during training and prediction.

Scikit-learn provides Pipeline and ColumnTransformer to organize these operations.

A pipeline can help prevent data leakage by ensuring that transformations are fitted on the training data and then applied consistently to other data.

Prepare a Sample Dataset

In this example, you will build a model to classify Iris flowers. The dataset contains numerical measurements and a target class.

First, load the data and split it into training and testing sets.

from sklearn.datasets import load_iris

from sklearn.model_selection import train_test_split

iris = load_iris(as_frame=True)

X = iris.data.copy()

y = iris.target

X_train, X_test, y_train, y_test = train_test_split(

X,

y,

test_size=0.2,

random_state=42,

stratify=y

)

All features in this dataset are numerical, so the example requires imputation and scaling rather than categorical encoding.

Create the Preprocessing Pipeline

Use a median imputer to handle missing numerical values and a standard scaler to scale the features.

from sklearn.pipeline import Pipeline

from sklearn.impute import SimpleImputer

from sklearn.preprocessing import StandardScaler

from sklearn.linear_model import LogisticRegression

pipeline = Pipeline([

("imputer", SimpleImputer(strategy="median")),

("scaler", StandardScaler()),

("model", LogisticRegression(max_iter=500))

])

The pipeline executes each step in sequence.

1. SimpleImputer replaces missing values using medians learned from the training data.
2. StandardScaler standardizes the numerical features.
3. LogisticRegression learns the classification model.

Train the Pipeline

pipeline.fit(X_train, y_train)

Calling fit() trains the transformations and model using the training data.

The pipeline keeps the preprocessing steps associated with the trained model, helping ensure that later predictions use the same transformations.

Evaluate the Pipeline

from sklearn.metrics import accuracy_score, classification_report

predictions = pipeline.predict(X_test)

print("Accuracy:", accuracy_score(y_test, predictions))

print(classification_report(y_test, predictions))

The accuracy indicates the proportion of correct predictions. The classification report provides precision, recall, F1-score, and support for each class.

Because the Iris dataset is small and relatively clean, this example demonstrates the workflow rather than representing the challenges of a large production dataset.

Adding Categorical Features

For datasets containing both numerical and categorical features, ColumnTransformer can apply different preprocessing steps to each group.

For example:

from sklearn.compose import ColumnTransformer

from sklearn.preprocessing import OneHotEncoder

from sklearn.impute import SimpleImputer

from sklearn.pipeline import Pipeline

from sklearn.preprocessing import StandardScaler

numeric_pipeline = Pipeline([

("imputer", SimpleImputer(strategy="median")),

("scaler", StandardScaler())

])

categorical_pipeline = Pipeline([

("imputer", SimpleImputer(strategy="most_frequent")),

("encoder", OneHotEncoder(handle_unknown="ignore"))

])

preprocessor = ColumnTransformer([

("numeric", numeric_pipeline, ["Age", "MonthlySpend"]),

("categorical", categorical_pipeline, ["Contract"])

])

This example defines preprocessing for two numerical columns and one categorical column. To use it for training, the dataset must contain those columns with compatible data types.

You can combine preprocessor with a classifier in an outer Pipeline, ensuring that all preprocessing steps are fitted appropriately during model training.

Why Pipelines Matter in Real Projects

Pipelines provide several benefits:

• They make preprocessing steps reproducible.
• They reduce the risk of fitting transformations on test data.
• They simplify model training and prediction.
• They make it easier to compare different preprocessing strategies.
• They help keep production preprocessing consistent with training.

A well-designed pipeline is an important part of creating reliable machine learning application