Menu

Data Cleaning in Data Science

Raw datasets frequently contain incomplete, inconsistent, or incorrect information. These problems can distort statistical calculations and reduce the reliability of machine learning predictions.

For example, a customer dataset may contain missing ages, duplicate customer records, transaction amounts stored as strings, and unusually large purchases.

Data cleaning addresses these issues before the dataset is used for statistical analysis or model development.

Data cleaning is the process of identifying, investigating, and correcting or appropriately handling missing, duplicate, inconsistent, and potentially erroneous observations.

The objective is to improve data reliability while preserving meaningful information.

Why Is Data Cleaning Important?

Machine learning algorithms learn relationships from the information provided during training.

Incorrect or inconsistent values may introduce misleading patterns.

Consider a retail dataset.

Customer ID Age Purchase Amount City
C101 25 2500 Delhi
C102 Missing 4200 Mumbai
C103 30 1800 delhi
C103 30 1800 delhi
C104 28 250000 Pune

This dataset contains several potential problems.

  • One record contains a missing age.
  • One record appears duplicated.
  • The city field uses inconsistent capitalisation.
  • One transaction is substantially larger than the others.

Each issue requires investigation before selecting a correction.

What Are Missing Values in Data Science?

Missing values occur when information is unavailable, was not recorded, or cannot be interpreted.

Common causes include incomplete forms, system failures, integration errors, and optional fields.

Pandas generally represents missing numerical values using NaN, although nullable data types may use pd.NA.

Identifying Missing Values

import pandas as pddf = pd.DataFrame({    "customer": ["Aarav", "Priya", "Rohan"],    "age": [25, None, 30],    "spending": [2500, 4200, 1800]})print(df.isnull().sum())

Output:

customer    0
age         1
spending    0
dtype: int64

The output indicates that the age column contains one missing value.

Calculating Missing-Value Percentages

missing_percentage = (    df.isnull().mean() * 100)print(missing_percentage)

This calculates the proportion of missing observations in each column.

A missing-value percentage is useful for assessing completeness, but it does not explain why information is missing.

What Are the Different Types of Missing Data?

Statistics distinguishes among three common missing-data mechanisms.

Type Full Form Explanation
MCAR Missing Completely at Random Missingness is independent of observed and unobserved data
MAR Missing at Random Missingness can be explained by observed variables
MNAR Missing Not at Random Missingness depends on unobserved information or missing values themselves

For example, income values may be missing more frequently among customers belonging to particular recorded age groups.

Such a pattern may be consistent with MAR under appropriate assumptions.

Understanding missingness matters because different mechanisms can introduce different biases into statistical analysis.

How Do You Handle Missing Values?

Several techniques are available.

1. Remove Incomplete Records

Records containing missing values can be removed using dropna().

cleaned_df = df.dropna()print(cleaned_df)

This retains rows containing non-missing values across the selected columns.

However, deleting incomplete records reduces the sample size and may introduce bias if the missingness mechanism is not suitable.

2. Mean Imputation

Mean imputation replaces missing numerical values with the arithmetic average of available observations.

df["age"] = df["age"].fillna(    df["age"].mean())

Mean imputation is straightforward but can distort variance and relationships between variables.

3. Median Imputation

Median imputation replaces missing numerical values with the median.

df["age"] = df["age"].fillna(    df["age"].median())

The median is less sensitive to extreme values than the mean.

However, median imputation can still alter the statistical distribution and should not be treated as universally appropriate.

4. Mode Imputation

Mode imputation is commonly used for categorical variables.

df["city"] = df["city"].fillna(    df["city"].mode()[0])

The mode represents the most frequent observed category.

5. Advanced Imputation

More advanced methods include K-Nearest Neighbours imputation, iterative imputation, and model-based approaches.

These methods estimate missing values using information from related variables.

They may preserve relationships more effectively in some circumstances but require stronger assumptions and validation.

How Do You Detect Duplicate Records?

Duplicate records occur when the same observation is represented more than once.

Pandas provides duplicated().

df = pd.DataFrame({    "customer_id": [101, 102, 102, 103],    "amount": [1500, 2000, 2000, 2500]})print(df.duplicated().sum())

Output:

1

The dataset contains one row that duplicates an earlier row.

Removing Duplicate Records

df = df.drop_duplicates()print(df)

This removes duplicate rows.

However, identical customer IDs do not automatically indicate duplicate transactions.

A customer may legitimately place several orders.

Duplicate detection should consider identifiers such as transaction IDs and event timestamps.

Removing Duplicates Based on Specific Columns

df = df.drop_duplicates(    subset=["customer_id"],    keep="first")

This retains only the first record for each customer ID.

Such an operation is appropriate only when the intended dataset should contain one row per customer.

It would be unsuitable for a transaction-level dataset containing legitimate repeat purchases.

What Are Outliers in Data Science?

Outliers are observations that differ substantially from the rest of a dataset.

They may represent unusual but valid events, measurement errors, data-entry mistakes, or rare operational conditions.

Consider the following purchase amounts:

₹500, ₹800, ₹1,000, ₹1,200, ₹1,500, and ₹50,000.

The final value is substantially larger than the others.

However, it may represent a legitimate high-value order rather than an error.

How Are Outliers Detected?

Interquartile Range Method

The Interquartile Range (IQR) measures the spread of the middle 50% of observations.

\[ IQR=Q_3-Q_1 \]

Potential outliers may be identified using:

\[ Lower\ Bound=Q_1-1.5(IQR) \]

\[ Upper\ Bound=Q_3+1.5(IQR) \]

Python implementation:

import pandas as pdvalues = pd.Series([    500, 800, 1000, 1200, 1500, 50000])Q1 = values.quantile(0.25)Q3 = values.quantile(0.75)IQR = Q3 - Q1lower = Q1 - 1.5 * IQRupper = Q3 + 1.5 * IQRoutliers = values[    (values < lower) | (values > upper)]print(outliers)

The output identifies observations outside the calculated thresholds.

The IQR method is useful for exploratory analysis but does not prove that flagged values are incorrect.

Z-Score Method

A Z-score measures how many standard deviations an observation lies above or below the mean.

\[ z=\frac{x-\mu}{\sigma} \]

Here:

  • \(x\) represents the observation.
  • \(\mu\) represents the mean.
  • \(\sigma\) represents the standard deviation.

A large absolute Z-score indicates that an observation is far from the mean relative to the standard deviation.

Thresholds such as \(|z|>3\) are sometimes used for screening, particularly in approximately normal distributions.

However, the method is sensitive to extreme values and may be unsuitable for highly skewed datasets.

How Should Outliers Be Handled?

Possible approaches include:

Method Description Consideration
Investigation Verify the unusual observation Preferred initial step
Correction Fix confirmed data-entry errors Requires reliable source information
Removal Exclude observations under justified rules May introduce bias
Capping Limit values to selected thresholds Changes the original distribution
Transformation Apply logarithmic or other transformations Depends on data characteristics
Robust Methods Use techniques less sensitive to outliers May preserve valid extremes

Removing every unusual observation can eliminate important information.

For example, fraud detection models may depend on rare transactions that differ substantially from ordinary customer activity.

How Do You Standardise Inconsistent Data Formats?

Datasets collected from multiple sources may contain inconsistent text, dates, and numerical representations.

Consider a city column containing:

Delhi, delhi, DELHI, and Delhi.

These values may refer to the same location but appear as separate categories.

Pandas string operations can standardise them.

df = pd.DataFrame({    "city": ["Delhi", "delhi", "DELHI", " Delhi "]})df["city"] = (    df["city"]    .str.strip()    .str.title())print(df)

Output:

    city
0  Delhi
1  Delhi
2  Delhi
3  Delhi

The strip() operation removes surrounding whitespace.

The title() operation standardises capitalisation.

However, general title-casing may be unsuitable for names, abbreviations, and identifiers where exact capitalisation matters.

How Do You Correct Data Types?

A numerical column may be imported as text due to formatting inconsistencies.

For example:

df = pd.DataFrame({    "amount": ["2500", "4200", "1800"]})df["amount"] = pd.to_numeric(    df["amount"],    errors="coerce")print(df.dtypes)

Output:

amount    int64
dtype: object

The pd.to_numeric() function converts compatible strings into numerical values.

The errors="coerce" option converts invalid entries into missing values.

These missing values must then be examined and handled appropriately.

How Does Data Cleaning Affect Machine Learning?

Poor-quality data can reduce model performance in several ways.

Missing Values: Some algorithms cannot process missing numerical inputs directly.

Duplicate Records: Repeated observations can distort training distributions and evaluation results.

Outliers: Extreme values can strongly influence certain statistical estimates and loss functions.

Incorrect Data Types: Numerical features stored as strings may prevent calculations.

Inconsistent Categories: Different representations of the same category can create unnecessary encoded features.

Data cleaning procedures that learn statistics from the dataset must be fitted using training data only.

For example, calculating the median of a feature using the complete dataset before the train-test split can introduce information from the test set.

Common Mistakes in Data Cleaning

  1. Removing missing observations without evaluating the missing-data mechanism.
  2. Applying mean imputation to highly skewed variables without investigation.
  3. Treating all unusual values as errors.
  4. Removing repeated customer records that represent legitimate transactions.
  5. Fitting preprocessing transformations on the complete dataset before evaluation.

Data Science Tutorial for Beginners

VA

Vaishali Ardhana



Learn via Video Course

undefined Logo

0 hrs



Get in Touch For Details