Data Cleaning in Data Science
Raw datasets frequently contain incomplete, inconsistent, or incorrect information. These problems can distort statistical calculations and reduce the reliability of machine learning predictions.
For example, a customer dataset may contain missing ages, duplicate customer records, transaction amounts stored as strings, and unusually large purchases.
Data cleaning addresses these issues before the dataset is used for statistical analysis or model development.
Data cleaning is the process of identifying, investigating, and correcting or appropriately handling missing, duplicate, inconsistent, and potentially erroneous observations.
The objective is to improve data reliability while preserving meaningful information.
Why Is Data Cleaning Important?
Machine learning algorithms learn relationships from the information provided during training.
Incorrect or inconsistent values may introduce misleading patterns.
Consider a retail dataset.
| Customer ID | Age | Purchase Amount | City |
|---|---|---|---|
| C101 | 25 | 2500 | Delhi |
| C102 | Missing | 4200 | Mumbai |
| C103 | 30 | 1800 | delhi |
| C103 | 30 | 1800 | delhi |
| C104 | 28 | 250000 | Pune |
This dataset contains several potential problems.
- One record contains a missing age.
- One record appears duplicated.
- The city field uses inconsistent capitalisation.
- One transaction is substantially larger than the others.
Each issue requires investigation before selecting a correction.
What Are Missing Values in Data Science?
Missing values occur when information is unavailable, was not recorded, or cannot be interpreted.
Common causes include incomplete forms, system failures, integration errors, and optional fields.
Pandas generally represents missing numerical values using NaN, although nullable data types may use pd.NA.
Identifying Missing Values
import pandas as pddf = pd.DataFrame({ "customer": ["Aarav", "Priya", "Rohan"], "age": [25, None, 30], "spending": [2500, 4200, 1800]})print(df.isnull().sum())Output:
customer 0
age 1
spending 0
dtype: int64The output indicates that the age column contains one missing value.
Calculating Missing-Value Percentages
missing_percentage = ( df.isnull().mean() * 100)print(missing_percentage)This calculates the proportion of missing observations in each column.
A missing-value percentage is useful for assessing completeness, but it does not explain why information is missing.
What Are the Different Types of Missing Data?
Statistics distinguishes among three common missing-data mechanisms.
| Type | Full Form | Explanation |
|---|---|---|
| MCAR | Missing Completely at Random | Missingness is independent of observed and unobserved data |
| MAR | Missing at Random | Missingness can be explained by observed variables |
| MNAR | Missing Not at Random | Missingness depends on unobserved information or missing values themselves |
For example, income values may be missing more frequently among customers belonging to particular recorded age groups.
Such a pattern may be consistent with MAR under appropriate assumptions.
Understanding missingness matters because different mechanisms can introduce different biases into statistical analysis.
How Do You Handle Missing Values?
Several techniques are available.
1. Remove Incomplete Records
Records containing missing values can be removed using dropna().
cleaned_df = df.dropna()print(cleaned_df)This retains rows containing non-missing values across the selected columns.
However, deleting incomplete records reduces the sample size and may introduce bias if the missingness mechanism is not suitable.
2. Mean Imputation
Mean imputation replaces missing numerical values with the arithmetic average of available observations.
df["age"] = df["age"].fillna( df["age"].mean())Mean imputation is straightforward but can distort variance and relationships between variables.
3. Median Imputation
Median imputation replaces missing numerical values with the median.
df["age"] = df["age"].fillna( df["age"].median())The median is less sensitive to extreme values than the mean.
However, median imputation can still alter the statistical distribution and should not be treated as universally appropriate.
4. Mode Imputation
Mode imputation is commonly used for categorical variables.
df["city"] = df["city"].fillna( df["city"].mode()[0])The mode represents the most frequent observed category.
5. Advanced Imputation
More advanced methods include K-Nearest Neighbours imputation, iterative imputation, and model-based approaches.
These methods estimate missing values using information from related variables.
They may preserve relationships more effectively in some circumstances but require stronger assumptions and validation.
How Do You Detect Duplicate Records?
Duplicate records occur when the same observation is represented more than once.
Pandas provides duplicated().
df = pd.DataFrame({ "customer_id": [101, 102, 102, 103], "amount": [1500, 2000, 2000, 2500]})print(df.duplicated().sum())Output:
1The dataset contains one row that duplicates an earlier row.
Removing Duplicate Records
df = df.drop_duplicates()print(df)This removes duplicate rows.
However, identical customer IDs do not automatically indicate duplicate transactions.
A customer may legitimately place several orders.
Duplicate detection should consider identifiers such as transaction IDs and event timestamps.
Removing Duplicates Based on Specific Columns
df = df.drop_duplicates( subset=["customer_id"], keep="first")This retains only the first record for each customer ID.
Such an operation is appropriate only when the intended dataset should contain one row per customer.
It would be unsuitable for a transaction-level dataset containing legitimate repeat purchases.
What Are Outliers in Data Science?
Outliers are observations that differ substantially from the rest of a dataset.
They may represent unusual but valid events, measurement errors, data-entry mistakes, or rare operational conditions.
Consider the following purchase amounts:
₹500, ₹800, ₹1,000, ₹1,200, ₹1,500, and ₹50,000.
The final value is substantially larger than the others.
However, it may represent a legitimate high-value order rather than an error.
How Are Outliers Detected?
Interquartile Range Method
The Interquartile Range (IQR) measures the spread of the middle 50% of observations.
\[ IQR=Q_3-Q_1 \]
Potential outliers may be identified using:
\[ Lower\ Bound=Q_1-1.5(IQR) \]
\[ Upper\ Bound=Q_3+1.5(IQR) \]
Python implementation:
import pandas as pdvalues = pd.Series([ 500, 800, 1000, 1200, 1500, 50000])Q1 = values.quantile(0.25)Q3 = values.quantile(0.75)IQR = Q3 - Q1lower = Q1 - 1.5 * IQRupper = Q3 + 1.5 * IQRoutliers = values[ (values < lower) | (values > upper)]print(outliers)The output identifies observations outside the calculated thresholds.
The IQR method is useful for exploratory analysis but does not prove that flagged values are incorrect.
Z-Score Method
A Z-score measures how many standard deviations an observation lies above or below the mean.
\[ z=\frac{x-\mu}{\sigma} \]
Here:
- \(x\) represents the observation.
- \(\mu\) represents the mean.
- \(\sigma\) represents the standard deviation.
A large absolute Z-score indicates that an observation is far from the mean relative to the standard deviation.
Thresholds such as \(|z|>3\) are sometimes used for screening, particularly in approximately normal distributions.
However, the method is sensitive to extreme values and may be unsuitable for highly skewed datasets.
How Should Outliers Be Handled?
Possible approaches include:
| Method | Description | Consideration |
|---|---|---|
| Investigation | Verify the unusual observation | Preferred initial step |
| Correction | Fix confirmed data-entry errors | Requires reliable source information |
| Removal | Exclude observations under justified rules | May introduce bias |
| Capping | Limit values to selected thresholds | Changes the original distribution |
| Transformation | Apply logarithmic or other transformations | Depends on data characteristics |
| Robust Methods | Use techniques less sensitive to outliers | May preserve valid extremes |
Removing every unusual observation can eliminate important information.
For example, fraud detection models may depend on rare transactions that differ substantially from ordinary customer activity.
How Do You Standardise Inconsistent Data Formats?
Datasets collected from multiple sources may contain inconsistent text, dates, and numerical representations.
Consider a city column containing:
Delhi, delhi, DELHI, and Delhi.
These values may refer to the same location but appear as separate categories.
Pandas string operations can standardise them.
df = pd.DataFrame({ "city": ["Delhi", "delhi", "DELHI", " Delhi "]})df["city"] = ( df["city"] .str.strip() .str.title())print(df)Output:
city
0 Delhi
1 Delhi
2 Delhi
3 DelhiThe strip() operation removes surrounding whitespace.
The title() operation standardises capitalisation.
However, general title-casing may be unsuitable for names, abbreviations, and identifiers where exact capitalisation matters.
How Do You Correct Data Types?
A numerical column may be imported as text due to formatting inconsistencies.
For example:
df = pd.DataFrame({ "amount": ["2500", "4200", "1800"]})df["amount"] = pd.to_numeric( df["amount"], errors="coerce")print(df.dtypes)Output:
amount int64
dtype: objectThe pd.to_numeric() function converts compatible strings into numerical values.
The errors="coerce" option converts invalid entries into missing values.
These missing values must then be examined and handled appropriately.
How Does Data Cleaning Affect Machine Learning?
Poor-quality data can reduce model performance in several ways.
Missing Values: Some algorithms cannot process missing numerical inputs directly.
Duplicate Records: Repeated observations can distort training distributions and evaluation results.
Outliers: Extreme values can strongly influence certain statistical estimates and loss functions.
Incorrect Data Types: Numerical features stored as strings may prevent calculations.
Inconsistent Categories: Different representations of the same category can create unnecessary encoded features.
Data cleaning procedures that learn statistics from the dataset must be fitted using training data only.
For example, calculating the median of a feature using the complete dataset before the train-test split can introduce information from the test set.
Common Mistakes in Data Cleaning
- Removing missing observations without evaluating the missing-data mechanism.
- Applying mean imputation to highly skewed variables without investigation.
- Treating all unusual values as errors.
- Removing repeated customer records that represent legitimate transactions.
- Fitting preprocessing transformations on the complete dataset before evaluation.
Data Science Tutorial for Beginners
Vaishali Ardhana








