Feature Engineering and Feature Selection
Machine learning models learn relationships between input variables and target outcomes. However, raw datasets do not always contain the most informative variables in a directly usable form.
For example, an e-commerce dataset may contain individual transaction timestamps, but a customer churn model may benefit more from knowing how many days have passed since the customer's most recent purchase.
Feature engineering and feature selection help improve the representation of information provided to a model.
Feature engineering is the process of creating or transforming input variables to represent relevant patterns. Feature selection identifies a subset of existing features that are useful for a particular machine learning task.
Both techniques aim to improve the quality and efficiency of predictive modelling.
What Is a Feature in Machine Learning?
A feature is an input variable used by a model to generate predictions.
Consider a customer churn prediction dataset.
| Customer ID | Account Age | Monthly Usage | Failed Payments | Churn |
|---|---|---|---|---|
| C101 | 24 months | 40 hours | 0 | 0 |
| C102 | 6 months | 5 hours | 2 | 1 |
| C103 | 18 months | 30 hours | 0 | 0 |
The input features are:
- Account age
- Monthly usage
- Failed payment count
The target variable is Churn.
For binary classification:
\[ y= \begin{cases} 1 & \text{Customer churned}\\ 0 & \text{Customer retained} \end{cases} \]
Feature engineering may create additional variables from the existing information.
For example, recent usage decline could be more informative than total monthly usage alone.
What Is Feature Engineering?
Feature engineering transforms raw information into variables that better represent the analytical problem.
Common feature engineering methods include:
- Mathematical transformations
- Date and time feature extraction
- Aggregation
- Categorical encoding
- Interaction features
- Domain-specific feature construction
The best approach depends on the dataset and the prediction objective.
1. Creating Mathematical Features
Mathematical feature engineering combines existing variables through calculations.
Consider an e-commerce dataset containing product price and quantity.
Revenue can be calculated as:
\[ Revenue=Price\times Quantity \]
Python implementation:
import pandas as pddf = pd.DataFrame({ "price": [1000, 1500, 2000], "quantity": [2, 3, 1]})df["revenue"] = df["price"] * df["quantity"]print(df)Output:
price quantity revenue
0 1000 2 2000
1 1500 3 4500
2 2000 1 2000The new revenue feature captures the combined effect of price and quantity.
Derived variables may provide useful predictive information, although they do not automatically improve every model.
2. Creating Date and Time Features
Datetime variables often contain information about seasonal and behavioural patterns.
For example, purchase timestamps can be used to derive the month, weekday, or hour of purchase.
import pandas as pddf = pd.DataFrame({ "purchase_date": [ "2026-01-15", "2026-02-20", "2026-03-10" ]})df["purchase_date"] = pd.to_datetime( df["purchase_date"])df["month"] = df["purchase_date"].dt.monthdf["day_of_week"] = df["purchase_date"].dt.dayofweekprint(df)The derived variables can help identify recurring purchasing patterns.
Pandas represents Monday as 0 and Sunday as 6 through dayofweek.
Why Are Time-Based Features Useful?
Retail demand may vary across weekdays, holidays, and seasons.
Transportation models may require departure-hour information because travel duration changes throughout the day.
However, future timestamps or outcomes cannot be used when constructing features for earlier predictions.
3. Creating Aggregated Features
Aggregated features summarise multiple records into customer-level, product-level, or time-window-level measurements.
Consider a transaction dataset.
| Customer ID | Order Amount |
|---|---|
| C101 | 1500 |
| C101 | 2500 |
| C102 | 3000 |
| C101 | 2000 |
| C102 | 1200 |
A customer-level dataset may require:
- Total spending
- Number of orders
- Average order value
Pandas can calculate these values using groupby().
import pandas as pddf = pd.DataFrame({ "customer_id": ["C101", "C101", "C102", "C101", "C102"], "order_amount": [1500, 2500, 3000, 2000, 1200]})customer_features = df.groupby("customer_id").agg( total_spending=("order_amount", "sum"), order_count=("order_amount", "count"), average_order_value=("order_amount", "mean"))print(customer_features)Output:
total_spending order_count average_order_value
customer_id
C101 6000 3 2000.0
C102 4200 2 2100.0These aggregated variables provide a summary of historical customer behaviour.
In predictive applications, aggregations must use only records available before the prediction timestamp.
4. Creating Interaction Features
Interaction features represent relationships between two or more input variables.
For example, advertising expenditure may influence revenue differently across product categories.
A numerical interaction feature can be created by multiplying variables.
\[ x_{interaction}=x_1x_2 \]
Consider:
df = pd.DataFrame({ "advertising_spend": [1000, 2000, 3000], "discount_rate": [0.05, 0.10, 0.15]})df["ad_discount_interaction"] = ( df["advertising_spend"] * df["discount_rate"])print(df)The interaction feature represents the product of advertising expenditure and discount rate.
In linear models, interaction terms allow the estimated effect of one predictor to vary with another predictor.
However, the usefulness of an interaction should be evaluated rather than assumed.
5. Creating Domain-Specific Features
Domain knowledge helps identify variables that represent meaningful real-world behaviour.
For example:
| Industry | Raw Data | Engineered Feature |
|---|---|---|
| E-commerce | Purchase timestamps | Days since last purchase |
| Banking | Transaction history | Transactions in the previous hour |
| Healthcare | Laboratory records | Change in biomarker level |
| Logistics | GPS observations | Average route speed |
| Manufacturing | Sensor readings | Rolling vibration variance |
| Telecommunications | Network usage | Percentage change in monthly usage |
Domain-specific features can be valuable because they translate raw observations into information directly related to the prediction problem.
What Is Feature Selection?
Feature selection identifies a subset of available input variables for model development.
A dataset may contain hundreds of columns, but some variables may be redundant, irrelevant, or unavailable during production inference.
Removing unnecessary features may reduce computational requirements and simplify model interpretation.
However, fewer features do not automatically produce better predictions.
Feature selection should be evaluated using an appropriate validation strategy.
What Are the Main Feature Selection Methods?
Feature selection techniques are commonly divided into three categories.
| Method | How It Works | Examples |
|---|---|---|
| Filter Methods | Evaluate features using statistical criteria | Correlation, mutual information |
| Wrapper Methods | Compare subsets through model performance | Recursive Feature Elimination |
| Embedded Methods | Perform feature selection during model training | L1 regularisation, tree-based models |
Each approach has different computational requirements and assumptions.
1. Filter Methods
Filter methods evaluate features using statistical measures independently of the final predictive model.
Common techniques include correlation analysis, mutual information, and statistical tests.
Correlation-Based Feature Screening
Pearson correlation measures linear association between numerical variables.
\[ r= \frac{Cov(X,Y)} {\sigma_X\sigma_Y} \]
A correlation coefficient near +1 indicates a strong positive linear relationship.
A coefficient near -1 indicates a strong negative linear relationship.
A value near zero indicates weak or absent linear association.
However, low correlation does not necessarily mean a feature has no predictive value.
A feature may have a nonlinear relationship with the target or become informative when combined with other variables.
Mutual Information
Mutual information measures statistical dependence between variables.
Unlike Pearson correlation, it can capture certain nonlinear relationships.
Mutual information can be useful for feature screening, although estimates depend on the data and estimation method.
2. Wrapper Methods
Wrapper methods evaluate feature subsets using a predictive model.
A common approach is Recursive Feature Elimination (RFE).
RFE repeatedly fits a model and removes features according to an importance criterion until the desired number of features remains.
Recursive Feature Elimination With Python
import numpy as npfrom sklearn.feature_selection import RFEfrom sklearn.linear_model import LogisticRegressionX = np.array([ [25, 5000, 2], [30, 7000, 5], [22, 3000, 1], [40, 9000, 7], [35, 6000, 4], [28, 4500, 2]])y = np.array([0, 0, 1, 0, 1, 1])model = LogisticRegression(max_iter=1000)selector = RFE( estimator=model, n_features_to_select=2)selector.fit(X, y)print(selector.support_)The output identifies which two features were selected.
This example illustrates the API using a very small dataset. Meaningful feature selection requires substantially more representative observations and proper validation.
RFE can become computationally expensive because the model is fitted repeatedly.
3. Embedded Feature Selection
Embedded methods perform feature selection as part of model training.
L1 regularisation is a common example.
The regularised objective may include:
\[ L_{total}= L_{prediction}+ \lambda\sum_j|\beta_j| \]
The penalty can shrink some coefficients to exactly zero.
Features corresponding to zero coefficients may be excluded from the resulting linear model.
Tree-based models can also provide feature importance measures, although importance does not necessarily imply causal relevance.
What Is the Difference Between Feature Engineering and Feature Selection?
| Aspect | Feature Engineering | Feature Selection |
|---|---|---|
| Primary Objective | Create or transform useful variables | Identify relevant existing variables |
| Typical Operation | Derive new features | Retain or remove features |
| Example | Calculate purchase frequency | Select the most informative predictors |
| Main Benefit | Improve information representation | Reduce complexity |
| Potential Risk | Introduce leakage or noisy features | Remove useful predictors |
Both techniques can be used together in a machine learning pipeline.
For example, a churn model may first generate usage trends and purchase-frequency features, then select a subset based on predictive performance.
What Is Data Leakage in Feature Engineering?
Data leakage occurs when information unavailable at the prediction time influences the model.
Consider a customer churn prediction model designed to identify customers likely to cancel within 30 days.
A feature such as days_since_last_login can be useful when calculated from activity recorded before the prediction date.
However, a feature indicating whether the customer cancelled during the prediction period would directly reveal the target.
Including it would produce misleading performance estimates.
Common Sources of Feature Leakage
- Using future transactions in historical features
- Including target-derived variables
- Calculating aggregates across training and test periods
- Fitting feature selection methods before train-test splitting
- Using information unavailable during real-time inference
Feature engineering must therefore respect the intended prediction timestamp and validation boundaries.
How Do You Build a Feature Selection Pipeline?
Scikit-learn supports pipelines that combine preprocessing, feature selection, and model training.
from sklearn.pipeline import Pipelinefrom sklearn.preprocessing import StandardScalerfrom sklearn.feature_selection import SelectKBest, f_classiffrom sklearn.linear_model import LogisticRegressionpipeline = Pipeline([ ("scaler", StandardScaler()), ("selector", SelectKBest( score_func=f_classif, k=2 )), ("classifier", LogisticRegression( max_iter=1000 ))])pipeline.fit(X, y)This example scales numerical features, selects two features using ANOVA F-statistics, and trains a logistic regression model.
For reliable evaluation, the pipeline should be fitted on training data and evaluated on separate observations.
Feature selection should also occur inside cross-validation folds during hyperparameter tuning.
How Do You Evaluate Feature Engineering and Selection?
The effectiveness of feature engineering and selection should be measured using validation performance rather than the number of features alone.
For classification models, relevant metrics may include precision, recall, F1-score, and ROC-AUC.
For regression models, common metrics include MAE, RMSE, and R².
A practical comparison involves training a baseline model using the original features and comparing it with a model using engineered or selected features.
The evaluation should use consistent data splits and identical performance metrics.
A new feature is useful when it improves the desired objective without introducing unacceptable complexity, leakage, or operational cost.
Common Mistakes in Feature Engineering and Selection
- Creating features using future information.
- Assuming more features always improve model performance.
- Selecting features using the complete dataset before splitting.
- Removing variables solely because of low pairwise correlation.
- Confusing feature importance with causal influence.
- Creating complex features that cannot be calculated during production inference.
Data Science Tutorial for Beginners
Vaishali Ardhana








