Data Transformation in Data Science
Machine learning algorithms require data to be represented in a form suitable for numerical computation. Raw datasets may contain categorical values, variables measured on different scales, skewed distributions, and inconsistent formats.
Data transformation converts this information into representations that algorithms can process.
Data transformation in data science involves modifying the structure, scale, format, or representation of variables to support statistical analysis and machine learning.
Important techniques include categorical encoding, feature scaling, mathematical transformations, and the construction of reproducible preprocessing pipelines.
What Is Data Transformation?
Data transformation changes the representation of one or more variables.
For example, a dataset may contain:
| Feature | Example Value | Potential Transformation |
|---|---|---|
| City | Delhi | One-hot encoding |
| Subscription Level | Premium | Ordinal encoding if order is meaningful |
| Monthly Income | ₹75,000 | Standardisation |
| Transaction Value | ₹5,000 | Logarithmic transformation |
| Purchase Date | 2026-01-15 | Extract month or day of week |
| Product Category | Electronics | Categorical encoding |
The appropriate transformation depends on the nature of the variable and the requirements of the selected model.
Why Is Categorical Encoding Necessary?
Many machine learning algorithms require numerical feature matrices.
However, business datasets frequently contain categorical information.
Examples include city names, payment methods, product categories, and subscription types.
Categorical encoding converts these values into numerical representations.
Common methods include:
- One-hot encoding
- Ordinal encoding
- Frequency encoding
- Target encoding
- Learned categorical embeddings
The method should reflect whether the categories have a meaningful order and how the resulting features will be used.
What Is One-Hot Encoding?
One-hot encoding represents each category using a separate indicator variable.
Consider a dataset containing three cities.
| Customer | City |
|---|---|
| C101 | Delhi |
| C102 | Mumbai |
| C103 | Pune |
After one-hot encoding:
| Customer | City_Delhi | City_Mumbai | City_Pune |
|---|---|---|---|
| C101 | 1 | 0 | 0 |
| C102 | 0 | 1 | 0 |
| C103 | 0 | 0 | 1 |
Each row identifies the customer's city through a binary indicator.
One-Hot Encoding With Pandas
import pandas as pddf = pd.DataFrame({ "city": ["Delhi", "Mumbai", "Pune", "Delhi"]})encoded = pd.get_dummies( df, columns=["city"], dtype=int)print(encoded)Output:
city_Delhi city_Mumbai city_Pune
0 1 0 0
1 0 1 0
2 1 0 0
3 1 0 0The encoding creates numerical columns representing individual categories.
One-hot encoding is useful for nominal categorical variables without a natural order.
However, a variable containing thousands of categories may produce a large number of additional columns.
What Is Ordinal Encoding?
Ordinal encoding assigns numerical values to categories with a meaningful order.
Consider customer satisfaction levels.
| Satisfaction Level | Encoded Value |
|---|---|
| Low | 0 |
| Medium | 1 |
| High | 2 |
The numerical ordering reflects the category ranking.
Ordinal Encoding With Python
import pandas as pddf = pd.DataFrame({ "satisfaction": ["Low", "Medium", "High"]})mapping = { "Low": 0, "Medium": 1, "High": 2}df["encoded_satisfaction"] = ( df["satisfaction"].map(mapping))print(df)Output:
satisfaction encoded_satisfaction
0 Low 0
1 Medium 1
2 High 2Ordinal encoding is appropriate when the categories have a meaningful order.
However, the assigned numerical distances do not necessarily represent equal differences between categories.
For example, the difference between low and medium satisfaction may not be equivalent to the difference between medium and high satisfaction.
One-Hot Encoding vs Ordinal Encoding
| Aspect | One-Hot Encoding | Ordinal Encoding |
|---|---|---|
| Category Type | Nominal | Ordinal |
| Numerical Representation | Binary indicator columns | Ordered numerical codes |
| Implied Ordering | No | Yes |
| Number of Output Features | Usually multiple | Usually one |
| Example | City names | Satisfaction levels |
| Main Limitation | High dimensionality | May imply artificial numerical distances |
Selecting an inappropriate encoding method can introduce misleading relationships into a machine learning model.
What Is Feature Scaling?
Feature scaling transforms numerical variables to control differences in their magnitude or distribution.
Consider a dataset containing customer age and annual income.
| Customer | Age | Annual Income (₹) |
|---|---|---|
| C101 | 25 | 3,00,000 |
| C102 | 35 | 8,00,000 |
| C103 | 45 | 15,00,000 |
Age values range between 25 and 45, while income values range between ₹3 lakh and ₹15 lakh.
Certain algorithms are sensitive to these differences.
For example, K-Nearest Neighbours uses distance calculations. A feature with much larger numerical values may dominate the distance metric if scaling is not applied.
Feature scaling helps place variables on more comparable scales.
1. What Is Min-Max Scaling?
Min-max scaling transforms numerical values into a specified interval.
The common transformation to the interval from 0 to 1 is:
\[ x_{scaled}= \frac{x-x_{min}}{x_{max}-x_{min}} \]
Here:
- \(x\) represents the original value.
- \(x_{min}\) represents the minimum training value.
- \(x_{max}\) represents the maximum training value.
Consider:
\[ x=[10,20,30,40,50] \]
For the value 30:
\[ x_{scaled}=\frac{30-10}{50-10} \]
\[ x_{scaled}=0.5 \]
Min-Max Scaling in Python
import numpy as npfrom sklearn.preprocessing import MinMaxScalerdata = np.array([ [10], [20], [30], [40], [50]])scaler = MinMaxScaler()scaled = scaler.fit_transform(data)print(scaled)Output:
[[0. ]
[0.25]
[0.5 ]
[0.75]
[1. ]]Min-max scaling preserves the relative ordering of observations.
However, extreme values can compress the majority of observations into a narrow part of the transformed range.
New observations may also fall outside the original fitted interval.
2. What Is Standardisation?
Standardisation transforms values using the mean and standard deviation.
It is commonly called Z-score scaling.
\[ z=\frac{x-\mu}{\sigma} \]
Here:
- \(x\) represents an observation.
- \(\mu\) represents the fitted mean.
- \(\sigma\) represents the fitted standard deviation.
Standardisation produces values with a mean close to zero and standard deviation close to one on the data used for fitting, subject to numerical precision and constant features.
Standardisation With Scikit-learn
import numpy as npfrom sklearn.preprocessing import StandardScalerdata = np.array([ [10], [20], [30], [40], [50]])scaler = StandardScaler()standardised = scaler.fit_transform(data)print(np.round(standardised, 2))Output:
[[-1.41]
[-0.71]
[ 0. ]
[ 0.71]
[ 1.41]]Standardisation is commonly used with linear models, Support Vector Machines, Principal Component Analysis, and gradient-based optimisation.
However, standardisation does not automatically make a distribution normal.
3. What Is Robust Scaling?
Robust scaling uses statistics that are less sensitive to extreme observations.
A common formulation is:
\[ x_{scaled}= \frac{x-\text{Median}}{IQR} \]
Here:
\[ IQR=Q_3-Q_1 \]
Robust scaling centres the data using the median and scales it using the interquartile range.
Robust Scaling in Python
import numpy as npfrom sklearn.preprocessing import RobustScalerdata = np.array([ [10], [20], [30], [40], [1000]])scaler = RobustScaler()scaled = scaler.fit_transform(data)print(scaled)Robust scaling can be useful when numerical features contain extreme values.
However, it does not remove outliers or guarantee that they will stop influencing the downstream model.
Feature Scaling Techniques Compared
| Technique | Mathematical Basis | Main Advantage | Limitation |
|---|---|---|---|
| Min-Max Scaling | Minimum and maximum | Maps fitted data to a chosen range | Sensitive to extremes |
| Standardisation | Mean and standard deviation | Produces comparable standardised units | Sensitive to extreme values |
| Robust Scaling | Median and IQR | Less influenced by extremes | Does not eliminate outliers |
| Log Transformation | Logarithmic function | Compresses large positive values | Requires an appropriate domain |
The choice depends on the algorithm and the statistical characteristics of the feature.
What Is Logarithmic Transformation?
A logarithmic transformation compresses large numerical values relative to smaller ones.
It is frequently considered for positively skewed distributions such as transaction amounts, customer spending, and certain financial measurements.
For non-negative values:
\[ x_{transformed}=\ln(1+x) \]
The transformation can be implemented using NumPy.
import numpy as npamounts = np.array([ 100, 500, 1000, 5000, 50000])transformed = np.log1p(amounts)print(np.round(transformed, 2))Output:
[ 4.62 6.22 6.91 8.52 10.82]The transformed values occupy a smaller numerical range.
However, logarithmic transformation is not automatically suitable for every numerical feature or model.
Which Algorithms Require Feature Scaling?
Different machine learning algorithms respond differently to feature magnitude.
| Algorithm | Importance of Scaling | Reason |
|---|---|---|
| K-Nearest Neighbours | High | Uses distance calculations |
| Support Vector Machine | Often high | Depends on feature geometry and kernels |
| Logistic Regression | Often useful | Supports stable numerical optimisation and regularisation |
| Linear Regression | Depends on implementation | Can improve numerical conditioning |
| K-Means | High | Uses distance-based clustering |
| Principal Component Analysis | Often important | Large-variance features can dominate |
| Decision Trees | Generally low | Uses feature thresholds |
| Random Forests | Generally low | Uses tree-based splitting |
Tree-based algorithms are generally less sensitive to monotonic numerical scaling than distance-based or gradient-sensitive methods.
What Is Data Leakage During Preprocessing?
Data leakage occurs when information unavailable at the intended prediction time influences model development or evaluation.
A common example involves applying a scaler to the complete dataset before splitting it into training and test subsets.
This allows test-set statistics to influence the transformation.
Incorrect Workflow
Complete Dataset → Fit Scaler → Split Into Training and Test Sets
Correct Workflow
Complete Dataset → Train-Test Split → Fit Scaler on Training Data → Transform Training and Test Data
Scikit-learn pipelines help enforce this sequence.
from sklearn.model_selection import train_test_splitfrom sklearn.preprocessing import StandardScalerX_train, X_test = train_test_split( data, test_size=0.2, random_state=42)scaler = StandardScaler()X_train_scaled = scaler.fit_transform(X_train)X_test_scaled = scaler.transform(X_test)The scaler learns the mean and standard deviation from training data only.
The same fitted parameters are then applied to test observations.
Common Mistakes in Data Transformation
- Applying ordinal encoding to unordered categories without justification.
- Scaling the complete dataset before train-test splitting.
- Assuming standardisation makes a feature normally distributed.
- Using logarithmic transformation without checking the input domain.
- Applying the same preprocessing method to every algorithm without considering its requirements.
Data Science Tutorial for Beginners
Vaishali Ardhana








