Data Science Lifecycle and End-to-End Workflow
A data science project involves more than selecting an algorithm and fitting a model. Production-oriented systems require reliable data pipelines, reproducible transformations, controlled experimentation, model evaluation, and monitoring after deployment.
The data science lifecycle is a structured process that converts a business or research objective into a validated analytical solution.
A standard workflow includes six stages:
- Problem definition
- Data collection and ingestion
- Data cleaning and preprocessing
- Exploratory data analysis and feature engineering
- Model development and evaluation
- Deployment and monitoring
Stage 1 Define the Analytical Problem
Every data science project begins with a clearly specified objective.
Consider a subscription platform that wants to predict customer churn.
The problem must be translated into a measurable prediction task.
Business objective: Identify customers likely to cancel their subscriptions within the next 30 days.
Machine learning task: Binary classification.
Target variable:
[
y =
\begin{cases}
1 & \text{Customer cancels within 30 days}\
0 & \text{Customer does not cancel within 30 days}
\end{cases}
]
The observation date and prediction horizon must be defined before extracting training data.
For example, a model may use customer activity recorded through September 30 to predict cancellation between October 1 and October 30.
Features recorded after September 30 must not be used for that prediction because doing so could introduce target leakage.
Potential input variables include:
| Feature | Data Type | Description |
|---|---|---|
| account_age_days | Integer | Days since subscription began |
| monthly_usage_hours | Float | Usage during the observation period |
| payment_failures_30d | Integer | Failed payment count |
| support_tickets_30d | Integer | Recent support requests |
| subscription_plan | Categorical | Current subscription category |
| churn | Binary | Target variable |
Success criteria should also be defined.
Possible metrics include recall, precision, PR-AUC, and the expected cost or benefit of retention interventions.
Stage 2 Collect and Ingest Data
Data collection involves extracting records from databases, application logs, APIs, or external datasets.
Relational databases are frequently accessed using SQL.
SELECT
customer_id,
account_age_days,
monthly_usage_hours,
payment_failures_30d,
support_tickets_30d,
subscription_plan,
churn
FROM customer_features
WHERE observation_date = '2026-09-30';This query retrieves a snapshot of customer features and the subsequently assigned churn label from a prepared analytical table.
In Python, the resulting data can be loaded using Pandas.
import pandas as pd
df = pd.read_csv("customer_churn.csv")
print(df.shape)
print(df.dtypes)shape returns the number of rows and columns, while dtypes reports the stored data types.
What Is the Difference Between ETL and ELT?
Data pipelines commonly use two processing patterns.
| Process | Sequence | Typical Application |
|---|---|---|
| ETL | Extract, Transform, Load | Transformation before loading into the destination |
| ELT | Extract, Load, Transform | Transformation inside a warehouse or analytical platform |
Both patterns help standardise data ingestion and support repeatable analytical workflows.
Stage 3 Clean and Preprocess the Dataset
Raw datasets may contain missing values, duplicate observations, inconsistent categories, and invalid numerical ranges.
Initial inspection can be performed using:
print(df.isnull().sum())
print(df.duplicated().sum())
print(df.describe())Handling Missing Values
Numerical missing values can sometimes be replaced using a median estimate.
df["monthly_usage_hours"] = (
pd.to_numeric(
df["monthly_usage_hours"],
errors="coerce"
)
)However, the imputation statistic must be learned from training data only.
Calculating imputation values before splitting the dataset can introduce information from validation or test records.
Handling Categorical Variables
Machine learning algorithms generally require a suitable numerical representation of categorical variables.
One-hot encoding transforms categories into binary indicator columns.
For example:
| subscription_plan | plan_basic | plan_premium |
|---|---|---|
| Basic | 1 | 0 |
| Premium | 0 | 1 |
Scikit-learn provides preprocessing transformers that can apply encoding consistently to training and inference data.
Stage 4 Perform EDA and Feature Engineering
Exploratory Data Analysis helps identify relationships, distributions, class imbalance, and possible data quality problems.
A common starting point is to examine the target distribution.
print(
df["churn"].value_counts(normalize=True)
)This reports the proportion of observations belonging to each target class.
Class imbalance matters because a model can achieve high accuracy by predicting the majority class while failing to identify customers who actually churn.
Creating Predictive Features
Feature engineering converts available information into variables that may improve predictive performance.
Examples include:
- Usage change over the previous four weeks
- Number of failed payments during the observation window
- Days since the most recent login
- Ratio of support tickets to account age
A derived feature can be created using Pandas.
df["tickets_per_30_account_days"] = (
df["support_tickets_30d"]
/ (df["account_age_days"].clip(lower=1) / 30)
)The denominator is clipped to avoid division by zero for new accounts.
Feature engineering must respect the prediction timestamp. Variables containing future events cannot be included in historical training examples.
Stage 5 Train and Evaluate the Model
A reproducible machine learning workflow should separate training data from evaluation data and apply preprocessing consistently.
The following example uses Scikit-learn.
from sklearn.model_selection import train_test_split
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
X = df.drop(columns=["churn", "customer_id"])
y = df["churn"]
numeric_features = [
"account_age_days",
"monthly_usage_hours",
"payment_failures_30d",
"support_tickets_30d",
"tickets_per_30_account_days"
]
categorical_features = ["subscription_plan"]
numeric_pipeline = Pipeline([
("imputer", SimpleImputer(strategy="median"))
])
categorical_pipeline = Pipeline([
("imputer", SimpleImputer(strategy="most_frequent")),
("encoder", OneHotEncoder(handle_unknown="ignore"))
])
preprocessor = ColumnTransformer([
("numeric", numeric_pipeline, numeric_features),
("categorical", categorical_pipeline, categorical_features)
])
pipeline = Pipeline([
("preprocessor", preprocessor),
("classifier", LogisticRegression(max_iter=1000))
])
X_train, X_test, y_train, y_test = train_test_split(
X,
y,
test_size=0.2,
random_state=42,
stratify=y
)
pipeline.fit(X_train, y_train)
predictions = pipeline.predict(X_test)
print(classification_report(y_test, predictions))This example demonstrates a preprocessing and classification pipeline on a tabular dataset.
The imputation and encoding operations are fitted only on the training subset.
A random stratified split is appropriate for an introductory example when observations can reasonably be treated as independent and identically distributed.
For actual customer churn forecasting, chronological validation may be more appropriate. Records belonging to the same customer must also be separated carefully to avoid leakage.
Understanding Classification Metrics
| Metric | Formula | Interpretation |
|---|---|---|
| Precision | TP / (TP + FP) | Proportion of predicted churners who actually churn |
| Recall | TP / (TP + FN) | Proportion of actual churners correctly identified |
| F1-score | 2PR / (P + R) | Harmonic mean of precision and recall |
| Accuracy | (TP + TN) / Total | Overall proportion of correct classifications |
Here, TP represents true positives, FP represents false positives, TN represents true negatives, and FN represents false negatives.
Accuracy alone may be misleading when churn is uncommon.
The choice of metric should reflect the intervention cost and the business impact of missed churners.
Stage 6 Deploy and Monitor the Model
After validation, a trained model can be serialised and integrated into an application.
import joblib
joblib.dump(
pipeline,
"churn_model.joblib"
)The saved pipeline contains the preprocessing steps and trained classifier.
A serving application can load this model and generate predictions for incoming customer records.
model = joblib.load("churn_model.joblib")
probabilities = model.predict_proba(
new_customer_data
)[:, 1]Only trusted model files should be loaded because Python object deserialisation can execute arbitrary code.
What Should Be Monitored After Deployment?
Important monitoring dimensions include:
- Data drift: Changes in feature distributions compared with reference data.
- Concept drift: Changes in the relationship between inputs and target outcomes.
- Prediction quality: Changes in precision, recall, calibration, or other relevant metrics when labels become available.
- Operational performance: API latency, error rates, and throughput.
- Business impact: Retention outcomes and the cost-effectiveness of interventions.
Training performance does not guarantee production performance.
A reliable data science lifecycle therefore includes ongoing validation, version control, monitoring, and retraining decisions based on observed evidence.
Data Science Tutorial for Beginners
Vaishali Ardhana








