Menu

Exploratory Data Analysis Using Pandas

Raw datasets frequently contain thousands of observations distributed across numerical, categorical, and temporal variables. Examining individual records rarely provides enough information to understand the dataset's overall characteristics.

Exploratory Data Analysis helps data scientists identify distributions, unusual observations, missing values, and relationships before developing predictive models.

Exploratory Data Analysis (EDA) is the systematic process of examining datasets using statistical summaries, data inspection, and visualisation to understand their structure, quality, variability, and relationships.

EDA is usually performed after initial data collection and continues throughout preprocessing and feature development.

For example, an e-commerce company may analyse customer transactions to determine which product categories generate the most revenue, whether purchasing behaviour varies across regions, and whether unusually large orders affect average sales.

What Are the Main Objectives of Exploratory Data Analysis?

EDA helps answer several important analytical questions.

Objective Technical Question Common Method
Data Structure How many observations and variables exist? shape, info()
Data Quality Are values missing or duplicated? isnull(), duplicated()
Distribution Analysis How are numerical values distributed? describe(), histograms
Categorical Analysis How frequently does each category occur? value_counts()
Relationship Analysis Are variables statistically associated? corr(), scatter plots
Outlier Detection Which observations differ substantially? IQR, box plots
Group Comparisons How do metrics differ between categories? groupby()

These findings help determine suitable preprocessing and modelling techniques.

For example, a highly skewed revenue variable may require further investigation, while a classification dataset containing very few positive cases may require special attention to class imbalance.

How Do You Create a Dataset for Exploratory Data Analysis?

Consider an online retail company with the following transaction data.

Customer ID Category Quantity Revenue City
C101 Electronics 2 12000 Delhi
C102 Clothing 3 4500 Mumbai
C103 Electronics 1 8000 Delhi
C104 Home Decor 2 3000 Pune
C105 Clothing 4 6000 Mumbai

This dataset can be represented using a Pandas DataFrame.

import pandas as pddf = pd.DataFrame({    "customer_id": ["C101", "C102", "C103", "C104", "C105"],    "category": [        "Electronics",        "Clothing",        "Electronics",        "Home Decor",        "Clothing"    ],    "quantity": [2, 3, 1, 2, 4],    "revenue": [12000, 4500, 8000, 3000, 6000],    "city": ["Delhi", "Mumbai", "Delhi", "Pune", "Mumbai"]})print(df.head())

Output:

  customer_id     category  quantity  revenue    city
0        C101  Electronics         2    12000   Delhi
1        C102     Clothing         3     4500  Mumbai
2        C103  Electronics         1     8000   Delhi
3        C104   Home Decor         2     3000    Pune
4        C105     Clothing         4     6000  Mumbai

The DataFrame contains five transaction records and five variables.

Each row represents an observation, while each column represents an attribute.

How Do You Inspect the Structure of a Dataset?

Initial inspection helps determine whether a dataset has been loaded correctly.

Checking Dataset Dimensions

The shape attribute returns the number of rows and columns.

print(df.shape)

Output:

(5, 5)

The dataset contains five observations and five variables.

Checking Column Names

print(df.columns.tolist())

Output:

['customer_id', 'category', 'quantity', 'revenue', 'city']

Examining column names helps identify the variables available for analysis.

Checking Data Types

print(df.dtypes)

Output:

customer_id    object
category       object
quantity        int64
revenue         int64
city           object
dtype: object

Numerical variables are represented using integer types, while text columns commonly use the object dtype in this example.

Data types matter because mathematical operations require compatible numerical values.

A revenue column imported as text may require conversion before calculating totals or averages.

Using info()

df.info()

The info() method reports column names, non-null counts, data types, and approximate memory usage.

It is particularly useful when working with larger datasets containing many variables.

How Do You Identify Missing Values During EDA?

Missing values can affect statistical summaries and predictive model development.

Pandas provides isnull() and isna() to identify missing observations.

print(df.isnull().sum())

Output:

customer_id    0
category       0
quantity       0
revenue        0
city           0
dtype: int64

The example dataset contains no missing values.

For larger datasets, missing-value percentages can be calculated.

missing_percentage = (    df.isnull().mean() * 100)print(missing_percentage)

The resulting values indicate the percentage of missing observations within each column.

A column containing 200 missing values across 10,000 rows has a missing-value rate of 2%.

However, the percentage alone does not indicate whether missingness is random or related to particular observations.

How Do You Detect Duplicate Records?

Duplicate records may inflate totals or distort statistical results.

duplicate_count = df.duplicated().sum()print(duplicate_count)

Output:

0

No duplicate rows are present.

For transactional datasets, duplicate detection should consider the actual observation unit.

For example, two purchases by the same customer are not duplicates simply because they share a customer ID.

A transaction identifier is generally more appropriate for investigating repeated transaction records.

How Do You Generate Descriptive Statistics in Pandas?

The describe() method generates numerical summaries.

print(df.describe())

For numerical columns, the output includes:

Statistic Description
Count Number of non-missing observations
Mean Arithmetic average
Standard Deviation Sample standard deviation
Minimum Smallest observed value
25% First quartile
50% Median
75% Third quartile
Maximum Largest observed value

For the revenue column, the mean is:

\[ \bar{x}=\frac{12000+4500+8000+3000+6000}{5} \]

\[ \bar{x}=6700 \]

The average transaction revenue is ₹6,700.

The median revenue is ₹6,000 because it is the middle observation after sorting the values.

Why Compare Mean and Median?

The mean is sensitive to extreme observations, whereas the median is less affected.

For example, adding a transaction worth ₹2 lakh would substantially increase the mean.

Comparing mean and median can help identify potential asymmetry in the revenue distribution.

However, the difference between these statistics is only one indicator and should be considered alongside the complete distribution.

How Do You Analyse Categorical Variables?

Categorical analysis examines the frequency and distribution of groups.

The value_counts() function calculates the number of observations belonging to each category.

print(df["category"].value_counts())

Output:

category
Electronics    2
Clothing       2
Home Decor     1
Name: count, dtype: int64

Electronics and clothing each appear in two transactions.

Relative frequencies can also be calculated.

category_percentage = (    df["category"].value_counts(        normalize=True    ) * 100)print(category_percentage)

Output:

category
Electronics    40.0
Clothing       40.0
Home Decor     20.0
Name: proportion, dtype: float64

These proportions describe transaction frequency rather than revenue contribution.

A category with fewer transactions may still generate more revenue.

How Do You Perform Grouped Analysis Using Pandas?

The groupby() function allows observations to be grouped according to one or more variables.

Consider calculating total revenue by product category.

category_revenue = (    df.groupby("category")["revenue"]    .sum())print(category_revenue)

Output:

category
Clothing       10500
Electronics    20000
Home Decor      3000
Name: revenue, dtype: int64

Electronics generates the highest total revenue in this sample.

However, this does not necessarily mean it has the highest average transaction value across larger datasets.

Applying Multiple Aggregations

Pandas supports multiple aggregation functions.

summary = df.groupby("category")["revenue"].agg(    ["count", "mean", "median", "sum"])print(summary)

Output:

             count     mean   median    sum
category
Clothing         2   5250.0   5250.0  10500
Electronics      2  10000.0  10000.0  20000
Home Decor       1   3000.0   3000.0   3000

This comparison explains transaction frequency and revenue distribution across categories.

Group-level analysis is commonly used in customer segmentation, business reporting, and performance monitoring.

How Do You Identify Outliers During EDA?

Outliers are observations that differ considerably from the majority of a dataset.

One common method uses the Interquartile Range.

\[ IQR=Q_3-Q_1 \]

The conventional outlier thresholds are:

\[ Lower\ Bound=Q_1-1.5(IQR) \]

\[ Upper\ Bound=Q_3+1.5(IQR) \]

Python implementation:

Q1 = df["revenue"].quantile(0.25)Q3 = df["revenue"].quantile(0.75)IQR = Q3 - Q1lower = Q1 - 1.5 * IQRupper = Q3 + 1.5 * IQRoutliers = df[    (df["revenue"] < lower) |    (df["revenue"] > upper)]print(outliers)

The method flags observations outside the calculated thresholds.

However, a flagged transaction may represent a legitimate large purchase rather than an error.

Outlier detection should therefore lead to investigation rather than automatic deletion.

How Do You Examine Numerical Relationships During EDA?

Correlation analysis measures associations between numerical variables.

For example, a retail company may investigate whether transaction quantity is associated with revenue.

correlation = df[    ["quantity", "revenue"]].corr()print(correlation)

The resulting matrix contains pairwise Pearson correlation coefficients by default.

A positive coefficient indicates that higher values of one variable tend to be associated with higher values of the other in a linear relationship.

A negative coefficient indicates the opposite direction.

A coefficient near zero indicates weak or absent linear association, but nonlinear relationships may still exist.

Correlation does not establish causation.

How Does EDA Support Machine Learning?

EDA helps identify potential preprocessing and modelling requirements.

EDA Finding Possible Next Step
Missing numerical observations Evaluate suitable imputation
Highly skewed values Consider transformation
Imbalanced classification labels Select appropriate evaluation metrics
Strongly related predictors Investigate redundancy
Unexpected extreme observations Validate source records
Temporal trends Consider chronological validation
High-cardinality categories Evaluate suitable encoding

EDA is not a one-time activity.

Data scientists often return to exploratory analysis after cleaning records, creating features, or evaluating model errors.

Common Mistakes in Exploratory Data Analysis

  1. Calculating statistics without checking data types.
  2. Treating every extreme observation as an error.
  3. Comparing group totals without considering group sizes.
  4. Assuming correlation proves causation.
  5. Drawing strong conclusions from very small samples.
  6. Ignoring missing values and duplicate records.

Data Science Tutorial for Beginners

VA

Vaishali Ardhana



Learn via Video Course

undefined Logo

0 hrs



Get in Touch For Details