Menu

Checking for Duplicate Reviews

Checking for Duplicate Reviews

Sometimes the same review appears multiple times in a dataset. Duplicate reviews can bias the model by giving extra importance to repeated examples.

We will identify duplicate rows and remove them if necessary.

Code

# Check duplicate rows
df.duplicated().sum()

Remove Duplicate Reviews

df.drop_duplicates(inplace=True)

Verify Dataset Size

df.shape