Processing Large Datasets Efficiently

Large datasets can consume significant memory when loaded entirely into a DataFrame. Python provides several approaches to handle larger files more efficiently.

Reading a CSV File in Chunks

Pandas allows you to process a CSV file in smaller batches using the chunksize parameter.

import pandas as pd

total_sales = 0

for chunk in pd.read_csv("large_sales.csv", chunksize=10000):

total_sales += chunk["Sales"].sum()

print("Total sales:", total_sales)

Each iteration processes up to 10,000 rows at a time. The example calculates total sales without keeping the entire CSV dataset in one DataFrame.

Chunk processing is useful for aggregations and transformations that can be performed independently on each batch. However, operations that require comparing every record or maintaining global state may need additional logic.

Specifying Data Types

Pandas normally infers column types while reading a file. You can explicitly specify types when you know how the data should be represented.

df = pd.read_csv(

"sales.csv",

dtype={

"Product": "string",

"Region": "string",

"Units": "int64"

}

)

print(df.dtypes)

This can improve consistency and sometimes reduce memory usage. Ensure that the actual file values match the specified types; invalid values may cause conversion errors.

Loading Only Necessary Rows

You can limit the number of rows loaded while exploring an unfamiliar dataset.

df = pd.read_csv(

"large_sales.csv",

nrows=1000

)

print(df.shape)

The nrows parameter is useful for quickly inspecting a sample of a large file. Remember that the sample may not represent every pattern in the complete dataset.