Building a Reusable Data Preparation Pipeline

A preprocessing pipeline organizes transformations into a repeatable workflow. It helps ensure that the same operations are applied consistently during model training and prediction.

Scikit-learn provides tools such as Pipeline and ColumnTransformer to combine preprocessing steps for different feature types.

Preparing Numerical and Categorical Features

Consider a dataset used to predict whether a customer will subscribe to a service.

import pandas as pd

df = pd.DataFrame({

"Age": [25, 40, 35, 50, 28, 45],

"Income": [30000, 70000, 50000, 90000, 35000, 80000],

"City": ["Delhi", "Mumbai", "Chennai", "Delhi", "Chennai", "Mumbai"],

"Subscribed": [0, 1, 0, 1, 0, 1]

})

X = df.drop(columns="Subscribed")

y = df["Subscribed"]

The numerical columns are Age and Income, while City is categorical. These columns require different preprocessing operations.

Creating the Pipeline

from sklearn.compose import ColumnTransformer

from sklearn.pipeline import Pipeline

from sklearn.preprocessing import OneHotEncoder, StandardScaler

from sklearn.impute import SimpleImputer

from sklearn.linear_model import LogisticRegression

from sklearn.model_selection import train_test_split

numeric_features = ["Age", "Income"]

categorical_features = ["City"]

numeric_pipeline = Pipeline([

("imputer", SimpleImputer(strategy="median")),

("scaler", StandardScaler())

])

categorical_pipeline = Pipeline([

("imputer", SimpleImputer(strategy="most_frequent")),

("encoder", OneHotEncoder(handle_unknown="ignore"))

])

preprocessor = ColumnTransformer([

("numeric", numeric_pipeline, numeric_features),

("categorical", categorical_pipeline, categorical_features)

])

model = Pipeline([

("preprocessor", preprocessor),

("classifier", LogisticRegression())

])

X_train, X_test, y_train, y_test = train_test_split(

X, y, test_size=0.33, random_state=42, stratify=y

)

model.fit(X_train, y_train)

print("Test accuracy:", model.score(X_test, y_test))

The pipeline performs the following operations:

1. Fills missing numerical values using the training median.
2. Standardizes numerical features.
3. Fills missing categorical values using the most frequent category.
4. One-hot encodes categorical values.
5. Trains a logistic regression classifier.

Because preprocessing is included in the model pipeline, the transformations are fitted on the training data and then reused for testing and prediction. This helps prevent data leakage.

The dataset here is intentionally small for demonstration, so its test accuracy is not a reliable estimate of real-world performance.

Why Reusable Pipelines Matter

Pipelines make data preparation easier to maintain, test, and deploy. They reduce the risk of applying different transformations during training and prediction and can be integrated with cross-validation and hyperparameter tuning.

Feature engineering should always be guided by the problem being solved. Useful features, appropriate encoding, careful scaling, and leakage-free preprocessing can improve model reliability, but their impact should be measured using a suitable validation strategy.