Exploratory Data Analysis Using Pandas
Raw datasets frequently contain thousands of observations distributed across numerical, categorical, and temporal variables. Examining individual records rarely provides enough information to understand the dataset's overall characteristics.
Exploratory Data Analysis helps data scientists identify distributions, unusual observations, missing values, and relationships before developing predictive models.
Exploratory Data Analysis (EDA) is the systematic process of examining datasets using statistical summaries, data inspection, and visualisation to understand their structure, quality, variability, and relationships.
EDA is usually performed after initial data collection and continues throughout preprocessing and feature development.
For example, an e-commerce company may analyse customer transactions to determine which product categories generate the most revenue, whether purchasing behaviour varies across regions, and whether unusually large orders affect average sales.
What Are the Main Objectives of Exploratory Data Analysis?
EDA helps answer several important analytical questions.
| Objective | Technical Question | Common Method |
|---|---|---|
| Data Structure | How many observations and variables exist? | shape, info() |
| Data Quality | Are values missing or duplicated? | isnull(), duplicated() |
| Distribution Analysis | How are numerical values distributed? | describe(), histograms |
| Categorical Analysis | How frequently does each category occur? | value_counts() |
| Relationship Analysis | Are variables statistically associated? | corr(), scatter plots |
| Outlier Detection | Which observations differ substantially? | IQR, box plots |
| Group Comparisons | How do metrics differ between categories? | groupby() |
These findings help determine suitable preprocessing and modelling techniques.
For example, a highly skewed revenue variable may require further investigation, while a classification dataset containing very few positive cases may require special attention to class imbalance.
How Do You Create a Dataset for Exploratory Data Analysis?
Consider an online retail company with the following transaction data.
| Customer ID | Category | Quantity | Revenue | City |
|---|---|---|---|---|
| C101 | Electronics | 2 | 12000 | Delhi |
| C102 | Clothing | 3 | 4500 | Mumbai |
| C103 | Electronics | 1 | 8000 | Delhi |
| C104 | Home Decor | 2 | 3000 | Pune |
| C105 | Clothing | 4 | 6000 | Mumbai |
This dataset can be represented using a Pandas DataFrame.
import pandas as pddf = pd.DataFrame({ "customer_id": ["C101", "C102", "C103", "C104", "C105"], "category": [ "Electronics", "Clothing", "Electronics", "Home Decor", "Clothing" ], "quantity": [2, 3, 1, 2, 4], "revenue": [12000, 4500, 8000, 3000, 6000], "city": ["Delhi", "Mumbai", "Delhi", "Pune", "Mumbai"]})print(df.head())Output:
customer_id category quantity revenue city
0 C101 Electronics 2 12000 Delhi
1 C102 Clothing 3 4500 Mumbai
2 C103 Electronics 1 8000 Delhi
3 C104 Home Decor 2 3000 Pune
4 C105 Clothing 4 6000 MumbaiThe DataFrame contains five transaction records and five variables.
Each row represents an observation, while each column represents an attribute.
How Do You Inspect the Structure of a Dataset?
Initial inspection helps determine whether a dataset has been loaded correctly.
Checking Dataset Dimensions
The shape attribute returns the number of rows and columns.
print(df.shape)Output:
(5, 5)The dataset contains five observations and five variables.
Checking Column Names
print(df.columns.tolist())Output:
['customer_id', 'category', 'quantity', 'revenue', 'city']Examining column names helps identify the variables available for analysis.
Checking Data Types
print(df.dtypes)Output:
customer_id object
category object
quantity int64
revenue int64
city object
dtype: objectNumerical variables are represented using integer types, while text columns commonly use the object dtype in this example.
Data types matter because mathematical operations require compatible numerical values.
A revenue column imported as text may require conversion before calculating totals or averages.
Using info()
df.info()The info() method reports column names, non-null counts, data types, and approximate memory usage.
It is particularly useful when working with larger datasets containing many variables.
How Do You Identify Missing Values During EDA?
Missing values can affect statistical summaries and predictive model development.
Pandas provides isnull() and isna() to identify missing observations.
print(df.isnull().sum())Output:
customer_id 0
category 0
quantity 0
revenue 0
city 0
dtype: int64The example dataset contains no missing values.
For larger datasets, missing-value percentages can be calculated.
missing_percentage = ( df.isnull().mean() * 100)print(missing_percentage)The resulting values indicate the percentage of missing observations within each column.
A column containing 200 missing values across 10,000 rows has a missing-value rate of 2%.
However, the percentage alone does not indicate whether missingness is random or related to particular observations.
How Do You Detect Duplicate Records?
Duplicate records may inflate totals or distort statistical results.
duplicate_count = df.duplicated().sum()print(duplicate_count)Output:
0No duplicate rows are present.
For transactional datasets, duplicate detection should consider the actual observation unit.
For example, two purchases by the same customer are not duplicates simply because they share a customer ID.
A transaction identifier is generally more appropriate for investigating repeated transaction records.
How Do You Generate Descriptive Statistics in Pandas?
The describe() method generates numerical summaries.
print(df.describe())For numerical columns, the output includes:
| Statistic | Description |
|---|---|
| Count | Number of non-missing observations |
| Mean | Arithmetic average |
| Standard Deviation | Sample standard deviation |
| Minimum | Smallest observed value |
| 25% | First quartile |
| 50% | Median |
| 75% | Third quartile |
| Maximum | Largest observed value |
For the revenue column, the mean is:
\[ \bar{x}=\frac{12000+4500+8000+3000+6000}{5} \]
\[ \bar{x}=6700 \]
The average transaction revenue is ₹6,700.
The median revenue is ₹6,000 because it is the middle observation after sorting the values.
Why Compare Mean and Median?
The mean is sensitive to extreme observations, whereas the median is less affected.
For example, adding a transaction worth ₹2 lakh would substantially increase the mean.
Comparing mean and median can help identify potential asymmetry in the revenue distribution.
However, the difference between these statistics is only one indicator and should be considered alongside the complete distribution.
How Do You Analyse Categorical Variables?
Categorical analysis examines the frequency and distribution of groups.
The value_counts() function calculates the number of observations belonging to each category.
print(df["category"].value_counts())Output:
category
Electronics 2
Clothing 2
Home Decor 1
Name: count, dtype: int64Electronics and clothing each appear in two transactions.
Relative frequencies can also be calculated.
category_percentage = ( df["category"].value_counts( normalize=True ) * 100)print(category_percentage)Output:
category
Electronics 40.0
Clothing 40.0
Home Decor 20.0
Name: proportion, dtype: float64These proportions describe transaction frequency rather than revenue contribution.
A category with fewer transactions may still generate more revenue.
How Do You Perform Grouped Analysis Using Pandas?
The groupby() function allows observations to be grouped according to one or more variables.
Consider calculating total revenue by product category.
category_revenue = ( df.groupby("category")["revenue"] .sum())print(category_revenue)Output:
category
Clothing 10500
Electronics 20000
Home Decor 3000
Name: revenue, dtype: int64Electronics generates the highest total revenue in this sample.
However, this does not necessarily mean it has the highest average transaction value across larger datasets.
Applying Multiple Aggregations
Pandas supports multiple aggregation functions.
summary = df.groupby("category")["revenue"].agg( ["count", "mean", "median", "sum"])print(summary)Output:
count mean median sum
category
Clothing 2 5250.0 5250.0 10500
Electronics 2 10000.0 10000.0 20000
Home Decor 1 3000.0 3000.0 3000This comparison explains transaction frequency and revenue distribution across categories.
Group-level analysis is commonly used in customer segmentation, business reporting, and performance monitoring.
How Do You Identify Outliers During EDA?
Outliers are observations that differ considerably from the majority of a dataset.
One common method uses the Interquartile Range.
\[ IQR=Q_3-Q_1 \]
The conventional outlier thresholds are:
\[ Lower\ Bound=Q_1-1.5(IQR) \]
\[ Upper\ Bound=Q_3+1.5(IQR) \]
Python implementation:
Q1 = df["revenue"].quantile(0.25)Q3 = df["revenue"].quantile(0.75)IQR = Q3 - Q1lower = Q1 - 1.5 * IQRupper = Q3 + 1.5 * IQRoutliers = df[ (df["revenue"] < lower) | (df["revenue"] > upper)]print(outliers)The method flags observations outside the calculated thresholds.
However, a flagged transaction may represent a legitimate large purchase rather than an error.
Outlier detection should therefore lead to investigation rather than automatic deletion.
How Do You Examine Numerical Relationships During EDA?
Correlation analysis measures associations between numerical variables.
For example, a retail company may investigate whether transaction quantity is associated with revenue.
correlation = df[ ["quantity", "revenue"]].corr()print(correlation)The resulting matrix contains pairwise Pearson correlation coefficients by default.
A positive coefficient indicates that higher values of one variable tend to be associated with higher values of the other in a linear relationship.
A negative coefficient indicates the opposite direction.
A coefficient near zero indicates weak or absent linear association, but nonlinear relationships may still exist.
Correlation does not establish causation.
How Does EDA Support Machine Learning?
EDA helps identify potential preprocessing and modelling requirements.
| EDA Finding | Possible Next Step |
|---|---|
| Missing numerical observations | Evaluate suitable imputation |
| Highly skewed values | Consider transformation |
| Imbalanced classification labels | Select appropriate evaluation metrics |
| Strongly related predictors | Investigate redundancy |
| Unexpected extreme observations | Validate source records |
| Temporal trends | Consider chronological validation |
| High-cardinality categories | Evaluate suitable encoding |
EDA is not a one-time activity.
Data scientists often return to exploratory analysis after cleaning records, creating features, or evaluating model errors.
Common Mistakes in Exploratory Data Analysis
- Calculating statistics without checking data types.
- Treating every extreme observation as an error.
- Comparing group totals without considering group sizes.
- Assuming correlation proves causation.
- Drawing strong conclusions from very small samples.
- Ignoring missing values and duplicate records.
Data Science Tutorial for Beginners
Vaishali Ardhana








