Optimizing Pandas Performance

As data grows, slow code and high memory use become real problems. This topic shows practical ways to make Pandas faster and lighter.

a. Why Pandas Slows Down

Pandas loads your whole dataset into memory. With large files, this can fill the memory and slow your computer. Slow code is also often caused by the way operations are written.

Common causes are looping through rows one by one, using the wrong data types, and keeping unused columns. Each of these adds extra work or memory use.

Before fixing anything, measure. Use df.info(memory_usage="deep") to see memory use, and %timeit in Jupyter to time your code. This shows where the real problem is.

b. Vectorization vs Loops

Vectorization means applying an operation to a whole column at once instead of row by row. Pandas and NumPy run these operations in fast, optimized code.

# Slow
for i in range(len(df)):
df.loc[i, "total"] = df.loc[i, "price"] * df.loc[i, "qty"]
# Fast
df["total"] = df["price"] * df["qty"]

The second version is shorter and can be many times faster on large data. Avoid iterrows() and loops for column math whenever you can.

When you need conditions, use np.where() or boolean masks instead of apply() with a Python function. The apply() method is flexible, but it is slower because it runs Python code on each row.

c. Memory-Efficient Data Types

By default, Pandas uses large data types. You can save a lot of memory by choosing smaller ones. For example, if a column holds small whole numbers, a small integer type is enough.

df["age"] = pd.to_numeric(df["age"], downcast="integer")
df["city"] = df["city"].astype("category")

The category type is very effective for text columns with few unique values, like city or gender. It stores each unique value once and uses small codes for the rows.

Also, load only the columns you need with usecols. Check your savings by comparing df.memory_usage(deep=True).sum() before and after the changes.

d. Reading Data in Chunks

If a file is too big to fit in memory, read it in chunks. The chunksize option returns small pieces one at a time.

total = 0
for chunk in pd.read_csv("big_file.csv", chunksize=100000):
total += chunk["sales"].sum()
print(total)

You process each chunk, keep only the results you need, and move on. This keeps memory use low at any moment.

Chunking works best for tasks like totals, counts, and filtering. For tasks that need the whole dataset at once, like sorting everything, consider tools like Dask or Polars.