Working with Text Data

A lot of useful information is stored as text, like names, addresses, and reviews. This topic shows how to clean and extract it using Pandas.

a. String Methods in Pandas

Pandas has a set of string tools under the .str accessor. They work on a whole column at once, so you do not need loops. Most of them match normal Python string methods.

df["name"] = df["name"].str.title()
df["email"] = df["email"].str.lower()
df["name_length"] = df["name"].str.len()

Other helpful methods include strip(), replace(), split(), startswith(), and contains(). You can chain them together to do several steps in one line.

Note that .str methods return NaN for missing values instead of causing an error. This makes them safe to use on real data.

b. Cleaning Text Data

Text data is often messy. It can have extra spaces, mixed capital letters, symbols, and typing errors. Cleaning it makes grouping and searching much more accurate.

df["review"] = (
df["review"]
.str.lower()
.str.strip()
.str.replace("[^a-z0-9 ]", "", regex=True)
)

The code above converts text to lowercase, removes extra spaces, and removes all special characters. This is a common first step before counting words or analyzing reviews.

You can also filter rows using text. For example, df[df["product"].str.contains("phone", case=False)] finds all rows where the product name has the word phone, no matter the capital letters.

c. Introduction to Regular Expressions

A regular expression, or regex, is a pattern used to find text. It is like a smart search. Instead of searching for one fixed word, you can search for a shape, like "any 10 digit number."

Some common symbols are \d for a digit, \w for a letter or number, + for one or more, and ^ for the start of a text. For example, \d{10} matches exactly 10 digits in a row.

df[df["phone"].str.contains(r"^\d{10}$")]

This keeps only rows with a valid 10 digit phone number. Regex looks hard at first, but you only need a few patterns for most tasks. Test patterns on small samples before using them on the full data.

d. Extracting Information from Text

The str.extract() method pulls out a part of the text that matches a pattern. You put the part you want inside round brackets.

df["year"] = df["title"].str.extract(r"(\d{4})")

This finds a four digit year inside a movie title like "Inception (2010)". The result goes into a new column. It is a quick way to build structured data from messy text.

You can also use str.split() to break a column into parts. For example, split a full name into first name and last name with expand=True. These methods turn one text column into several useful columns.