Menu

Data Transformation in Data Science

Machine learning algorithms require data to be represented in a form suitable for numerical computation. Raw datasets may contain categorical values, variables measured on different scales, skewed distributions, and inconsistent formats.

Data transformation converts this information into representations that algorithms can process.

Data transformation in data science involves modifying the structure, scale, format, or representation of variables to support statistical analysis and machine learning.

Important techniques include categorical encoding, feature scaling, mathematical transformations, and the construction of reproducible preprocessing pipelines.

What Is Data Transformation?

Data transformation changes the representation of one or more variables.

For example, a dataset may contain:

Feature Example Value Potential Transformation
City Delhi One-hot encoding
Subscription Level Premium Ordinal encoding if order is meaningful
Monthly Income ₹75,000 Standardisation
Transaction Value ₹5,000 Logarithmic transformation
Purchase Date 2026-01-15 Extract month or day of week
Product Category Electronics Categorical encoding

The appropriate transformation depends on the nature of the variable and the requirements of the selected model.

Why Is Categorical Encoding Necessary?

Many machine learning algorithms require numerical feature matrices.

However, business datasets frequently contain categorical information.

Examples include city names, payment methods, product categories, and subscription types.

Categorical encoding converts these values into numerical representations.

Common methods include:

  • One-hot encoding
  • Ordinal encoding
  • Frequency encoding
  • Target encoding
  • Learned categorical embeddings

The method should reflect whether the categories have a meaningful order and how the resulting features will be used.

What Is One-Hot Encoding?

One-hot encoding represents each category using a separate indicator variable.

Consider a dataset containing three cities.

Customer City
C101 Delhi
C102 Mumbai
C103 Pune

After one-hot encoding:

Customer City_Delhi City_Mumbai City_Pune
C101 1 0 0
C102 0 1 0
C103 0 0 1

Each row identifies the customer's city through a binary indicator.

One-Hot Encoding With Pandas

import pandas as pddf = pd.DataFrame({    "city": ["Delhi", "Mumbai", "Pune", "Delhi"]})encoded = pd.get_dummies(    df,    columns=["city"],    dtype=int)print(encoded)

Output:

   city_Delhi  city_Mumbai  city_Pune
0           1            0          0
1           0            1          0
2           1            0          0
3           1            0          0

The encoding creates numerical columns representing individual categories.

One-hot encoding is useful for nominal categorical variables without a natural order.

However, a variable containing thousands of categories may produce a large number of additional columns.

What Is Ordinal Encoding?

Ordinal encoding assigns numerical values to categories with a meaningful order.

Consider customer satisfaction levels.

Satisfaction Level Encoded Value
Low 0
Medium 1
High 2

The numerical ordering reflects the category ranking.

Ordinal Encoding With Python

import pandas as pddf = pd.DataFrame({    "satisfaction": ["Low", "Medium", "High"]})mapping = {    "Low": 0,    "Medium": 1,    "High": 2}df["encoded_satisfaction"] = (    df["satisfaction"].map(mapping))print(df)

Output:

  satisfaction  encoded_satisfaction
0          Low                     0
1       Medium                     1
2         High                     2

Ordinal encoding is appropriate when the categories have a meaningful order.

However, the assigned numerical distances do not necessarily represent equal differences between categories.

For example, the difference between low and medium satisfaction may not be equivalent to the difference between medium and high satisfaction.

One-Hot Encoding vs Ordinal Encoding

Aspect One-Hot Encoding Ordinal Encoding
Category Type Nominal Ordinal
Numerical Representation Binary indicator columns Ordered numerical codes
Implied Ordering No Yes
Number of Output Features Usually multiple Usually one
Example City names Satisfaction levels
Main Limitation High dimensionality May imply artificial numerical distances

Selecting an inappropriate encoding method can introduce misleading relationships into a machine learning model.

What Is Feature Scaling?

Feature scaling transforms numerical variables to control differences in their magnitude or distribution.

Consider a dataset containing customer age and annual income.

Customer Age Annual Income (₹)
C101 25 3,00,000
C102 35 8,00,000
C103 45 15,00,000

Age values range between 25 and 45, while income values range between ₹3 lakh and ₹15 lakh.

Certain algorithms are sensitive to these differences.

For example, K-Nearest Neighbours uses distance calculations. A feature with much larger numerical values may dominate the distance metric if scaling is not applied.

Feature scaling helps place variables on more comparable scales.

1. What Is Min-Max Scaling?

Min-max scaling transforms numerical values into a specified interval.

The common transformation to the interval from 0 to 1 is:

\[ x_{scaled}= \frac{x-x_{min}}{x_{max}-x_{min}} \]

Here:

  • \(x\) represents the original value.
  • \(x_{min}\) represents the minimum training value.
  • \(x_{max}\) represents the maximum training value.

Consider:

\[ x=[10,20,30,40,50] \]

For the value 30:

\[ x_{scaled}=\frac{30-10}{50-10} \]

\[ x_{scaled}=0.5 \]

Min-Max Scaling in Python

import numpy as npfrom sklearn.preprocessing import MinMaxScalerdata = np.array([    [10],    [20],    [30],    [40],    [50]])scaler = MinMaxScaler()scaled = scaler.fit_transform(data)print(scaled)

Output:

[[0.  ]
 [0.25]
 [0.5 ]
 [0.75]
 [1.  ]]

Min-max scaling preserves the relative ordering of observations.

However, extreme values can compress the majority of observations into a narrow part of the transformed range.

New observations may also fall outside the original fitted interval.

2. What Is Standardisation?

Standardisation transforms values using the mean and standard deviation.

It is commonly called Z-score scaling.

\[ z=\frac{x-\mu}{\sigma} \]

Here:

  • \(x\) represents an observation.
  • \(\mu\) represents the fitted mean.
  • \(\sigma\) represents the fitted standard deviation.

Standardisation produces values with a mean close to zero and standard deviation close to one on the data used for fitting, subject to numerical precision and constant features.

Standardisation With Scikit-learn

import numpy as npfrom sklearn.preprocessing import StandardScalerdata = np.array([    [10],    [20],    [30],    [40],    [50]])scaler = StandardScaler()standardised = scaler.fit_transform(data)print(np.round(standardised, 2))

Output:

[[-1.41]
 [-0.71]
 [ 0.  ]
 [ 0.71]
 [ 1.41]]

Standardisation is commonly used with linear models, Support Vector Machines, Principal Component Analysis, and gradient-based optimisation.

However, standardisation does not automatically make a distribution normal.

3. What Is Robust Scaling?

Robust scaling uses statistics that are less sensitive to extreme observations.

A common formulation is:

\[ x_{scaled}= \frac{x-\text{Median}}{IQR} \]

Here:

\[ IQR=Q_3-Q_1 \]

Robust scaling centres the data using the median and scales it using the interquartile range.

Robust Scaling in Python

import numpy as npfrom sklearn.preprocessing import RobustScalerdata = np.array([    [10],    [20],    [30],    [40],    [1000]])scaler = RobustScaler()scaled = scaler.fit_transform(data)print(scaled)

Robust scaling can be useful when numerical features contain extreme values.

However, it does not remove outliers or guarantee that they will stop influencing the downstream model.

Feature Scaling Techniques Compared

Technique Mathematical Basis Main Advantage Limitation
Min-Max Scaling Minimum and maximum Maps fitted data to a chosen range Sensitive to extremes
Standardisation Mean and standard deviation Produces comparable standardised units Sensitive to extreme values
Robust Scaling Median and IQR Less influenced by extremes Does not eliminate outliers
Log Transformation Logarithmic function Compresses large positive values Requires an appropriate domain

The choice depends on the algorithm and the statistical characteristics of the feature.

What Is Logarithmic Transformation?

A logarithmic transformation compresses large numerical values relative to smaller ones.

It is frequently considered for positively skewed distributions such as transaction amounts, customer spending, and certain financial measurements.

For non-negative values:

\[ x_{transformed}=\ln(1+x) \]

The transformation can be implemented using NumPy.

import numpy as npamounts = np.array([    100, 500, 1000, 5000, 50000])transformed = np.log1p(amounts)print(np.round(transformed, 2))

Output:

[ 4.62  6.22  6.91  8.52 10.82]

The transformed values occupy a smaller numerical range.

However, logarithmic transformation is not automatically suitable for every numerical feature or model.

Which Algorithms Require Feature Scaling?

Different machine learning algorithms respond differently to feature magnitude.

Algorithm Importance of Scaling Reason
K-Nearest Neighbours High Uses distance calculations
Support Vector Machine Often high Depends on feature geometry and kernels
Logistic Regression Often useful Supports stable numerical optimisation and regularisation
Linear Regression Depends on implementation Can improve numerical conditioning
K-Means High Uses distance-based clustering
Principal Component Analysis Often important Large-variance features can dominate
Decision Trees Generally low Uses feature thresholds
Random Forests Generally low Uses tree-based splitting

Tree-based algorithms are generally less sensitive to monotonic numerical scaling than distance-based or gradient-sensitive methods.

What Is Data Leakage During Preprocessing?

Data leakage occurs when information unavailable at the intended prediction time influences model development or evaluation.

A common example involves applying a scaler to the complete dataset before splitting it into training and test subsets.

This allows test-set statistics to influence the transformation.

Incorrect Workflow

Complete Dataset → Fit Scaler → Split Into Training and Test Sets

Correct Workflow

Complete Dataset → Train-Test Split → Fit Scaler on Training Data → Transform Training and Test Data

Scikit-learn pipelines help enforce this sequence.

from sklearn.model_selection import train_test_splitfrom sklearn.preprocessing import StandardScalerX_train, X_test = train_test_split(    data,    test_size=0.2,    random_state=42)scaler = StandardScaler()X_train_scaled = scaler.fit_transform(X_train)X_test_scaled = scaler.transform(X_test)

The scaler learns the mean and standard deviation from training data only.

The same fitted parameters are then applied to test observations.

Common Mistakes in Data Transformation

  1. Applying ordinal encoding to unordered categories without justification.
  2. Scaling the complete dataset before train-test splitting.
  3. Assuming standardisation makes a feature normally distributed.
  4. Using logarithmic transformation without checking the input domain.
  5. Applying the same preprocessing method to every algorithm without considering its requirements.

Data Science Tutorial for Beginners

VA

Vaishali Ardhana



Learn via Video Course

undefined Logo

0 hrs



Get in Touch For Details