Pandas for Data Science
Real-world datasets frequently contain thousands of records distributed across multiple variables. These datasets may include missing values, duplicate transactions, inconsistent formats, and information collected from different sources.
Before performing statistical analysis or developing machine learning models, data scientists must organise and transform this information into a suitable structure.
Pandas is an open-source Python library used for data manipulation, cleaning, transformation, aggregation, and exploratory data analysis through labelled data structures.
Its primary data structures, Series and DataFrame, provide flexible methods for working with tabular information.
What Is Pandas in Python?
Pandas provides high-level operations for processing structured datasets.
It supports reading files, selecting records, filtering observations, calculating summary statistics, joining tables, and creating derived features.
A retail company may use Pandas to analyse customer transactions, calculate monthly revenue, identify high-value customers, and examine purchase patterns.
Pandas can process data from CSV files, Excel workbooks, SQL databases, and several other supported formats.
What Are Pandas Series and DataFrames?
Pandas provides two primary data structures.
Pandas Series
A Series is a one-dimensional labelled data structure.
It contains values associated with an index.
import pandas as pdsales = pd.Series([1200, 1500, 1800, 2200])print(sales)Output:
0 1200
1 1500
2 1800
3 2200
dtype: int64The numbers on the left represent index labels.
The values on the right represent the stored observations.
Pandas DataFrame
A DataFrame is a two-dimensional labelled data structure containing rows and columns.
It resembles a spreadsheet or relational database table.
df = pd.DataFrame({ "customer": ["Aarav", "Priya", "Rohan"], "spending": [2500, 4200, 1800], "orders": [2, 4, 1]})print(df)Output:
customer spending orders
0 Aarav 2500 2
1 Priya 4200 4
2 Rohan 1800 1Each column represents a variable, while each row represents an observation.
A DataFrame can contain columns with different data types.
How Do You Load Data Using Pandas?
Pandas provides functions for importing datasets from different file formats.
| Function | Purpose |
|---|---|
pd.read_csv() |
Reads CSV files |
pd.read_excel() |
Reads Excel files |
pd.read_json() |
Reads JSON data |
pd.read_sql() |
Reads SQL query results or tables through a database connection |
pd.read_parquet() |
Reads Parquet files |
For example:
import pandas as pddf = pd.read_csv("customer_data.csv")print(df.head())The read_csv() function loads the dataset into a DataFrame.
The head() function displays the first five observations.
File paths must refer to files accessible from the current Python environment.
How Do You Inspect a DataFrame?
Before analysis, data scientists inspect the structure and completeness of the dataset.
Pandas provides several useful methods.
| Operation | Purpose |
|---|---|
head() |
Displays initial rows |
tail() |
Displays final rows |
info() |
Reports column types and non-null counts |
describe() |
Generates statistical summaries |
shape |
Returns row and column counts |
dtypes |
Displays column data types |
isnull() |
Identifies missing values |
duplicated() |
Identifies duplicate rows |
Example:
print(df.head())print(df.info())print(df.shape)print(df.isnull().sum())These operations help identify incorrect types, incomplete records, and other potential data quality issues.
For example, a column containing sales revenue may be imported as a string if the source contains currency symbols or inconsistent formatting.
How Do You Select Rows and Columns in Pandas?
Pandas supports column selection and row indexing.
Selecting a Column
customers = pd.DataFrame({ "name": ["Aarav", "Priya", "Rohan"], "spending": [2500, 4200, 1800]})print(customers["name"])This returns a Series containing the customer names.
Selecting Multiple Columns
print(customers[["name", "spending"]])This returns a DataFrame containing the specified columns.
Using loc and iloc
The loc indexer selects data using labels.
The iloc indexer selects data using integer positions.
print(customers.loc[0, "name"])print(customers.iloc[1, 1])Output:
Aarav
4200loc refers to index labels and column names.
iloc refers to positional indices.
How Do You Filter Data Using Pandas?
Filtering selects records that satisfy specific conditions.
Consider a retailer identifying customers who spent more than ₹3,000.
import pandas as pddf = pd.DataFrame({ "customer": ["Aarav", "Priya", "Rohan"], "spending": [2500, 4200, 1800]})high_spenders = df[df["spending"] > 3000]print(high_spenders)Output:
customer spending
1 Priya 4200Code Explanation
pd.DataFrame()creates a tabular dataset.df["spending"] > 3000generates a Boolean condition.- Boolean indexing selects records satisfying the condition.
Filtering With Multiple Conditions
Pandas can combine conditions using & and |.
For example:
df = pd.DataFrame({ "customer": ["Aarav", "Priya", "Rohan"], "spending": [2500, 4200, 1800], "orders": [2, 4, 1]})selected = df[ (df["spending"] > 2000) & (df["orders"] >= 2)]print(selected)Output:
customer spending orders
0 Aarav 2500 2
1 Priya 4200 4Parentheses are important because Pandas uses bitwise operators for element-wise Boolean conditions.
How Do You Handle Missing Values in Pandas?
Missing values may occur because of incomplete records, system errors, or unavailable information.
Pandas typically represents missing numerical values using NaN, although nullable data types may use pd.NA.
Identifying Missing Values
print(df.isnull().sum())The operation calculates the number of missing values in each column.
Replacing Missing Values
Consider:
df = pd.DataFrame({ "customer": ["Aarav", "Priya", "Rohan"], "spending": [2500, None, 1800]})median_spending = df["spending"].median()df["spending"] = df["spending"].fillna( median_spending)print(df)Output:
customer spending
0 Aarav 2500.0
1 Priya 2150.0
2 Rohan 1800.0The missing value is replaced using the median of the available observations.
Median imputation can be useful for some numerical variables, but it is not universally appropriate.
The choice depends on the distribution, missing-data mechanism, and analytical objective.
In machine learning, imputation values must be learned from training data to avoid leakage.
How Do You Remove Duplicate Records?
Duplicate records can inflate counts and distort aggregate statistics.
Pandas provides duplicated() and drop_duplicates().
df = pd.DataFrame({ "transaction_id": [101, 102, 102, 103], "amount": [1500, 2000, 2000, 2500]})print(df.duplicated().sum())Output:
1Duplicate rows can be removed using:
df = df.drop_duplicates()print(df)However, repeated customer IDs or transaction amounts do not necessarily represent duplicate transactions.
Deduplication rules should reflect the underlying business identifiers and data structure.
How Do You Group and Aggregate Data in Pandas?
The groupby() method groups observations according to specified variables and applies aggregation functions.
Consider a sales dataset.
df = pd.DataFrame({ "category": [ "Electronics", "Clothing", "Electronics", "Clothing" ], "revenue": [12000, 4500, 8000, 6000]})category_revenue = df.groupby( "category")["revenue"].sum()print(category_revenue)Output:
category
Clothing 10500
Electronics 20000
Name: revenue, dtype: int64This operation calculates total revenue for each product category.
Applying Multiple Aggregations
summary = df.groupby("category")["revenue"].agg( ["count", "mean", "sum"])print(summary)The result contains transaction counts, average revenue, and total revenue for each category.
Grouping and aggregation are useful for customer segmentation, sales reporting, and exploratory data analysis.
How Do You Create New Features in Pandas?
Feature engineering transforms existing columns into variables that provide additional analytical information.
Consider an order dataset containing product prices and quantities.
df = pd.DataFrame({ "product": ["Laptop", "Mouse", "Keyboard"], "price": [50000, 800, 1500], "quantity": [2, 5, 3]})df["revenue"] = ( df["price"] * df["quantity"])print(df)Output:
product price quantity revenue
0 Laptop 50000 2 100000
1 Mouse 800 5 4000
2 Keyboard 1500 3 4500The derived revenue column combines price and quantity into a useful business metric.
Feature engineering can also create:
- Customer purchase frequency
- Average order value
- Days since the previous purchase
- Monthly transaction counts
- Revenue per customer
These derived variables can be used in statistical analysis and predictive modelling.
How Do You Merge DataFrames?
Business information is often stored across multiple tables.
For example, one dataset may contain customer details, while another contains purchase records.
Pandas provides merge() to combine related datasets.
customers = pd.DataFrame({ "customer_id": [101, 102], "name": ["Aarav", "Priya"]})orders = pd.DataFrame({ "customer_id": [101, 102], "amount": [2500, 4200]})merged = pd.merge( customers, orders, on="customer_id", how="inner")print(merged)Output:
customer_id name amount
0 101 Aarav 2500
1 102 Priya 4200The customer_id column acts as the common key.
Pandas supports inner, left, right, and outer joins.
The appropriate join depends on which records should be retained.
Why Is Pandas Important in Machine Learning?
Machine learning models require datasets that have been validated and transformed into suitable numerical representations.
Pandas supports important preprocessing tasks such as:
- Selecting relevant variables
- Filtering invalid observations
- Identifying missing values
- Calculating summary statistics
- Combining datasets
- Creating predictive features
- Converting data types
- Preparing model input tables
For example, a customer churn model may require account age, transaction frequency, payment failures, and recent activity levels.
Pandas can combine these attributes into a feature table before the numerical preprocessing and model training stages.
However, Pandas itself is primarily a data manipulation library. Machine learning algorithms are commonly implemented using libraries such as Scikit-learn, XGBoost, and PyTorch.
Common Mistakes Beginners Should Avoid
- Ignoring missing values: Incomplete records may affect calculations and modelling results.
- Confusing loc and iloc: One selects using labels, while the other uses integer positions.
- Using incorrect Boolean syntax: Multiple conditions require appropriate operators and parentheses.
- Removing legitimate repeated records: Duplicate detection must consider business identifiers.
- Introducing data leakage: Preprocessing and feature engineering must respect training and evaluation boundaries.
Data Science Tutorial for Beginners
Vaishali Ardhana








