{"id":56506,"date":"2024-07-19T11:35:06","date_gmt":"2024-07-19T06:05:06","guid":{"rendered":"https:\/\/www.guvi.in\/blog\/?p=56506"},"modified":"2026-10-12T18:50:05","modified_gmt":"2026-10-12T13:20:05","slug":"data-cleaning-in-data-science","status":"publish","type":"post","link":"https:\/\/www.guvi.in\/blog\/data-cleaning-in-data-science\/","title":{"rendered":"Data Cleaning in Data Science: A Comprehensive Guide"},"content":{"rendered":"\n<p>Data cleaning is the process of finding and fixing errors, duplicates, and missing values in a dataset before it&#8217;s used for analysis or model building. It typically involves:<\/p>\n\n\n\n<ul>\n<li>Removing duplicate or irrelevant records<\/li>\n\n\n\n<li>Fixing missing or incomplete values<\/li>\n\n\n\n<li>Correcting formatting and typing errors<\/li>\n\n\n\n<li>Standardizing units, dates, and categories<\/li>\n\n\n\n<li>Flagging or treating outliers<\/li>\n<\/ul>\n\n\n\n<p>Without this step, even the most advanced analysis will produce misleading results because a model or dashboard is only as reliable as the data it&#8217;s fed.<\/p>\n\n\n\n<p><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>TL;DR Summary<\/strong><\/h2>\n\n\n\n<ul>\n<li>Data Cleaning is the process of finding and fixing errors, duplicates, and missing values before analysis or model building.<\/li>\n\n\n\n<li>Wrangling shapes the data, cleaning corrects it, and preprocessing tunes it for your model.<\/li>\n\n\n\n<li>The Data Cleaning process follows 8 steps, from understanding your data to validating the result.<\/li>\n\n\n\n<li>Missing data, duplicates, inconsistent formatting, typos, outliers, and structural errors are the most common issues.<\/li>\n\n\n\n<li>Tools like Pandas, OpenRefine, Excel, SQL, and Trifacta help, and habits like documenting every change prevent common Data Cleaning mistakes.<\/li>\n<\/ul>\n\n\n\n<p><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>What is Data Cleaning in Data Science?<\/strong><\/h2>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" width=\"1200\" height=\"585\" src=\"https:\/\/www.guvi.in\/blog\/wp-content\/uploads\/2026\/08\/ChatGPT-Image-Aug-12-2026-08_55_40-AM-1200x585.webp\" alt=\"What is Data Cleaning in Data Science?\" class=\"wp-image-131805\" srcset=\"https:\/\/www.guvi.in\/blog\/wp-content\/uploads\/2026\/08\/ChatGPT-Image-Aug-12-2026-08_55_40-AM-1200x585.webp 1200w, https:\/\/www.guvi.in\/blog\/wp-content\/uploads\/2026\/08\/ChatGPT-Image-Aug-12-2026-08_55_40-AM-300x146.webp 300w, https:\/\/www.guvi.in\/blog\/wp-content\/uploads\/2026\/08\/ChatGPT-Image-Aug-12-2026-08_55_40-AM-768x375.webp 768w, https:\/\/www.guvi.in\/blog\/wp-content\/uploads\/2026\/08\/ChatGPT-Image-Aug-12-2026-08_55_40-AM-1536x749.webp 1536w, https:\/\/www.guvi.in\/blog\/wp-content\/uploads\/2026\/08\/ChatGPT-Image-Aug-12-2026-08_55_40-AM-150x73.webp 150w, https:\/\/www.guvi.in\/blog\/wp-content\/uploads\/2026\/08\/ChatGPT-Image-Aug-12-2026-08_55_40-AM.webp 1796w\" sizes=\"(max-width: 1200px) 100vw, 1200px\" title=\"\"><\/figure>\n\n\n\n<p>Data cleaning, also called data cleansing or scrubbing, turns raw, messy data into a dataset you can trust for analysis. <\/p>\n\n\n\n<p>Raw data collected from forms, sensors, APIs, or logs is rarely ready to use.<\/p>\n\n\n\n<p>You&#8217;ll often find blank fields, duplicate rows, mismatched date formats, or nonsensical values, like a negative age or a future birth date. Cleaning resolves these problems before they quietly distort your results.<\/p>\n\n\n\n<p><\/p>\n\n\n\n<p><strong><em>Still Saying \u201cI\u2019ll Start Next Month\u201d? We\u2019ve all said it. Meanwhile, someone else just started. HCL GUVI\u2019s Advanced Data Science &amp; Generative AI Program takes you from zero to job-ready with live online classes, 10+ real projects, 1:1 doubt sessions, and an IITM Pravartak certification. Learn in your own language, with placement support and 1000+ hiring partners behind you. Your \u201csomeday\u201d can begin today. Apply now!<\/em><\/strong><\/p>\n\n\n<div class=\"wp-block-guvi-cta-button\" style=\"justify-content:center;\"><a href=\"https:\/\/www.guvi.in\/zen-class\/data-science-course\/?utm_source=blog&#038;utm_medium=hyperlink&#038;utm_campaign=data-cleaning-in-data-science\" class=\"guvi-btn-enquire\" target=\"_blank\" rel=\"noopener noreferrer\">Enroll in Data Science Course<\/a><\/div>\n\n\n<p><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Data Cleaning vs Data Preprocessing vs Data Wrangling<\/strong><\/h2>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" width=\"1200\" height=\"668\" src=\"https:\/\/www.guvi.in\/blog\/wp-content\/uploads\/2026\/08\/ChatGPT-Image-Aug-12-2026-08_57_03-AM-1200x668.webp\" alt=\"Data Cleaning vs Data Preprocessing vs Data Wrangling\" class=\"wp-image-131804\" srcset=\"https:\/\/www.guvi.in\/blog\/wp-content\/uploads\/2026\/08\/ChatGPT-Image-Aug-12-2026-08_57_03-AM-1200x668.webp 1200w, https:\/\/www.guvi.in\/blog\/wp-content\/uploads\/2026\/08\/ChatGPT-Image-Aug-12-2026-08_57_03-AM-300x167.webp 300w, https:\/\/www.guvi.in\/blog\/wp-content\/uploads\/2026\/08\/ChatGPT-Image-Aug-12-2026-08_57_03-AM-768x428.webp 768w, https:\/\/www.guvi.in\/blog\/wp-content\/uploads\/2026\/08\/ChatGPT-Image-Aug-12-2026-08_57_03-AM-1536x855.webp 1536w, https:\/\/www.guvi.in\/blog\/wp-content\/uploads\/2026\/08\/ChatGPT-Image-Aug-12-2026-08_57_03-AM-150x84.webp 150w, https:\/\/www.guvi.in\/blog\/wp-content\/uploads\/2026\/08\/ChatGPT-Image-Aug-12-2026-08_57_03-AM.webp 1681w\" sizes=\"(max-width: 1200px) 100vw, 1200px\" title=\"\"><\/figure>\n\n\n\n<p>People use these three terms interchangeably, but they aren&#8217;t quite the same. Here&#8217;s how they actually differ.<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><thead><tr><th><strong>Aspect<\/strong><\/th><th><strong>Data Cleaning<\/strong><\/th><th><strong>Data Preprocessing<\/strong><\/th><th><strong>Data Wrangling<\/strong><\/th><\/tr><\/thead><tbody><tr><td>Definition<\/td><td>Fixing errors, duplicates, missing values<\/td><td>Preparing cleaned data for modeling<\/td><td>Reshaping raw data into a usable structure<\/td><\/tr><tr><td>Primary goal<\/td><td>Accuracy and consistency<\/td><td>Model readiness<\/td><td>Structure and accessibility<\/td><\/tr><tr><td>When it happens<\/td><td>Right after data collection<\/td><td>After cleaning, before modeling<\/td><td>Can happen before or alongside cleaning<\/td><\/tr><tr><td>Example task<\/td><td>Removing duplicate customer records<\/td><td>Scaling numerical features<\/td><td>Merging three CSV files into one table<\/td><\/tr><tr><td>Typical output<\/td><td>A clean, error-free dataset<\/td><td>A model-ready dataset<\/td><td>A structured, unified dataset<\/td><\/tr><\/tbody><\/table><figcaption class=\"wp-element-caption\"><strong>Data Cleaning vs Data Preprocessing vs Data Wrangling<\/strong><\/figcaption><\/figure>\n\n\n\n<p>In short, wrangling shapes the data, cleaning corrects it, and <a href=\"https:\/\/www.guvi.in\/blog\/what-is-data-preprocessing-in-data-science\/\" data-type=\"link\" data-id=\"https:\/\/www.guvi.in\/blog\/what-is-data-preprocessing-in-data-science\/\" target=\"_blank\" rel=\"noreferrer noopener\">preprocessing<\/a> tunes it for your model.<\/p>\n\n\n\n<p><\/p>\n\n\n\n<div style=\"background-color: #099f4e; border: 3px solid #110053; border-radius: 12px; padding: 18px 22px; color: #FFFFFF; font-size: 18px; font-family: Montserrat, Helvetica, sans-serif; line-height: 1.6; box-shadow: 0 4px 12px rgba(0, 0, 0, 0.15); max-width: 750px;\">\n  <strong style=\"font-size: 22px; color: #FFFFFF;\">\ud83d\udca1 Did You Know?<\/strong> \n  <br \/><br \/> \n  According to multiple industry reports on data science workflows, professionals spend nearly 60 to 80 percent of their project time cleaning and preparing data, far more than they spend on modeling or analysis.\n<\/div>\n\n\n\n<p><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>The Data Cleaning Process, Step by Step<\/strong><\/h2>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" width=\"1200\" height=\"666\" src=\"https:\/\/www.guvi.in\/blog\/wp-content\/uploads\/2026\/08\/ChatGPT-Image-Aug-12-2026-08_58_56-AM-1200x666.webp\" alt=\"The Data Cleaning Process, Step by Step\" class=\"wp-image-131803\" srcset=\"https:\/\/www.guvi.in\/blog\/wp-content\/uploads\/2026\/08\/ChatGPT-Image-Aug-12-2026-08_58_56-AM-1200x666.webp 1200w, https:\/\/www.guvi.in\/blog\/wp-content\/uploads\/2026\/08\/ChatGPT-Image-Aug-12-2026-08_58_56-AM-300x166.webp 300w, https:\/\/www.guvi.in\/blog\/wp-content\/uploads\/2026\/08\/ChatGPT-Image-Aug-12-2026-08_58_56-AM-768x426.webp 768w, https:\/\/www.guvi.in\/blog\/wp-content\/uploads\/2026\/08\/ChatGPT-Image-Aug-12-2026-08_58_56-AM-1536x852.webp 1536w, https:\/\/www.guvi.in\/blog\/wp-content\/uploads\/2026\/08\/ChatGPT-Image-Aug-12-2026-08_58_56-AM-150x83.webp 150w, https:\/\/www.guvi.in\/blog\/wp-content\/uploads\/2026\/08\/ChatGPT-Image-Aug-12-2026-08_58_56-AM.webp 1684w\" sizes=\"(max-width: 1200px) 100vw, 1200px\" title=\"\"><\/figure>\n\n\n\n<p>Data Cleaning gets much easier when you follow a proven workflow instead of starting from scratch on every project. Here are the steps most <a href=\"https:\/\/www.guvi.in\/blog\/role-blog\/data-scientist-roles\/\" target=\"_blank\" rel=\"noreferrer noopener\">data scientists<\/a> rely on:<\/p>\n\n\n\n<p><strong>Note:<\/strong> All code snippets in this section use Python (with the pandas library).<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Step 1: Understand Your Data<\/strong><\/h4>\n\n\n\n<p>Before you change a single value, find out what you are actually holding. Read the data dictionary, check the data source, and review what each column should contain.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import pandas as pd\n\ndf = pd.read_csv(\"customers.csv\")\ndf.info()<\/code><\/pre>\n\n\n\n<p><strong>Code Explanation:<\/strong><\/p>\n\n\n\n<ul>\n<li><code>read_csv<\/code> loads the file into a DataFrame.<\/li>\n\n\n\n<li><code>df.info()<\/code> shows every column, its data type, and how many non-null values it has.<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Step 2: Keep a Raw Copy<\/strong><\/h4>\n\n\n\n<p>Never clean your only copy of the data. Save the original untouched, and do all your work on a duplicate. If something goes wrong, you can always start over.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>df_raw = df.copy()\ndf_raw.to_csv(\"customers_raw_backup.csv\", index=False)<\/code><\/pre>\n\n\n\n<p><strong>Code Explanation:<\/strong><\/p>\n\n\n\n<ul>\n<li><code>df.copy()<\/code> creates an independent copy, so changes to <code>df<\/code> never touch <code>df_raw<\/code>.<\/li>\n\n\n\n<li><code>to_csv<\/code> saves the backup to disk, so it survives even if your session ends.<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Step 3: Explore First<\/strong><\/h4>\n\n\n\n<p>Imagine buying a used car without opening the hood. That is what cleaning without exploring looks like. Run quick summaries and simple charts to spot the obvious problems early, like impossible ages, strange categories, or columns full of blanks.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>print(df.describe())\nprint(df.isnull().sum())<\/code><\/pre>\n\n\n\n<p><strong>Code Explanation:<\/strong><\/p>\n\n\n\n<ul>\n<li><code>describe()<\/code> gives the min, max, mean, and spread of numeric columns, which makes odd values easy to catch.<\/li>\n\n\n\n<li><code>isnull().sum()<\/code> counts the missing values in each column.<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Step 4: Handle Missing Values<\/strong><\/h4>\n\n\n\n<p>Not every blank deserves the same treatment. If only a tiny share of rows is missing, deleting them is usually fine. If a lot is missing, fill the gaps sensibly. If the fact that data is missing is itself meaningful, flag it so the information is not lost.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>df&#91;\"missing_income\"] = df&#91;\"income\"].isnull()\ndf&#91;\"income\"] = df&#91;\"income\"].fillna(df&#91;\"income\"].median())\ndf = df.dropna(subset=&#91;\"customer_id\"])<\/code><\/pre>\n\n\n\n<p><strong>Code Explanation:<\/strong><\/p>\n\n\n\n<ul>\n<li>The first line creates a flag column that records which rows were originally missing.<\/li>\n\n\n\n<li>The second line fills missing income with the median, which is safer than the mean when values are skewed.<\/li>\n\n\n\n<li>The third line drops rows where <code>customer_id<\/code> is missing, since those rows are unusable.<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Step 5: Remove Duplicates<\/strong><\/h4>\n\n\n\n<p>Duplicates quietly inflate your counts and skew your results. First decide what \u201cduplicate\u201d means for your dataset. Is it an identical row, or just a repeated customer ID? Exact matches are easy to remove, while near matches like \u201cJon Smith\u201d and \u201cJohn Smith\u201d need fuzzy matching tools.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>print(df.duplicated().sum())\ndf = df.drop_duplicates(subset=\"customer_id\", keep=\"first\")<\/code><\/pre>\n\n\n\n<p><strong>Code Explanation:<\/strong><\/p>\n\n\n\n<ul>\n<li><code>duplicated().sum()<\/code> tells you how many fully identical rows exist.<\/li>\n\n\n\n<li><code>drop_duplicates<\/code> removes repeats based on <code>customer_id<\/code> and keeps the first record.<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Step 6: Treat Outliers<\/strong><\/h4>\n\n\n\n<p>Here is the golden rule: investigate before you delete. A salary of 10 million might be a typing error, or it might be your most valuable customer. Check the context, then decide whether to remove, cap, or keep it.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>q1 = df&#91;\"income\"].quantile(0.25)\nq3 = df&#91;\"income\"].quantile(0.75)\niqr = q3 - q1\n\noutliers = df&#91;(df&#91;\"income\"] &lt; q1 - 1.5 * iqr) | (df&#91;\"income\"] &gt; q3 + 1.5 * iqr)]\nprint(len(outliers))<\/code><\/pre>\n\n\n\n<p><strong>Code Explanation:<\/strong><\/p>\n\n\n\n<ul>\n<li><code>q1<\/code> and <code>q3<\/code> mark the middle 50% of your data, and <code>iqr<\/code> is the gap between them.<\/li>\n\n\n\n<li>Values beyond 1.5 times the IQR from either end are flagged as potential outliers.<\/li>\n\n\n\n<li>Printing the count lets you review them before deciding what to do.<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Step 7: Standardize Formats<\/strong><\/h4>\n\n\n\n<p>Computers treat \u201cMumbai\u201d, \u201cmumbai\u201d, and \u201c MUMBAI \u201c as three different cities. Make dates, units, capitalization, and category labels consistent so that your analysis groups things correctly.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>df&#91;\"signup_date\"] = pd.to_datetime(df&#91;\"signup_date\"], errors=\"coerce\")\ndf&#91;\"city\"] = df&#91;\"city\"].str.strip().str.title()<\/code><\/pre>\n\n\n\n<p><strong>Code Explanation:<\/strong><\/p>\n\n\n\n<ul>\n<li><code>to_datetime<\/code> converts messy date strings into a proper date type, and <code>errors=\"coerce\"<\/code> turns unreadable dates into blanks instead of crashing.<\/li>\n\n\n\n<li><code>str.strip()<\/code> removes extra spaces, and <code>str.title()<\/code> makes capitalization uniform.<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Step 8: Validate the Result<\/strong><\/h4>\n\n\n\n<p>Cleaning is only finished once you have proven it worked. Compare the cleaned data against the raw copy and confirm that you removed the problems without losing anything important.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>print(df_raw.shape, df.shape)\nprint(df_raw&#91;\"income\"].describe())\nprint(df&#91;\"income\"].describe())<\/code><\/pre>\n\n\n\n<p><strong>Code Explanation:<\/strong><\/p>\n\n\n\n<ul>\n<li><code>shape<\/code> shows how many rows and columns you started with versus what remains.<\/li>\n\n\n\n<li>Comparing <code>describe()<\/code> output for both versions shows whether key statistics shifted more than expected.<\/li>\n<\/ul>\n\n\n\n<p><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Common Data Quality Issues You&#8217;ll Run Into<\/strong><\/h2>\n\n\n\n<p>Data Cleaning starts with recognizing what is actually wrong, and most messy datasets share the same six problems. For each one, you&#8217;ll see the mistake in code first, then the correct way to handle it.<\/p>\n\n\n\n<p><strong>Note:<\/strong> All code snippets in this section use Python (with the pandas library).<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>1. Missing Data<\/strong><\/h4>\n\n\n\n<p>Blanks appear when a form field is skipped, a sensor fails, or a system never records a value. The danger is that they hide quietly, and the usual quick fix, dropping them, can wipe out far more data than you expect.<\/p>\n\n\n\n<p><strong>The problem:<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>df = df.dropna()\nprint(len(df))  # 10,000 rows shrink to 3,200<\/code><\/pre>\n\n\n\n<p>One blank in any column removes the entire row, so a few scattered gaps can destroy most of your dataset.<\/p>\n\n\n\n<p><strong>How to fix it:<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>print(df.isnull().mean().mul(100).round(1))   # check how much is missing first\ndf&#91;\"income\"] = df&#91;\"income\"].fillna(df&#91;\"income\"].median())\ndf = df.dropna(subset=&#91;\"customer_id\"])        # drop only where it truly matters<\/code><\/pre>\n\n\n\n<p>Check the percentage per column, fill what can be filled sensibly, and drop rows only when a critical column is empty.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>2. Duplicate Records<\/strong><\/h4>\n\n\n\n<p>These usually appear when you merge files from different sources or when a user submits a form twice. They inflate your counts, so ten real customers can look like fifteen.<\/p>\n\n\n\n<p><strong>The problem:<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>df = df.drop_duplicates()\nprint(df&#91;\"customer_id\"].duplicated().sum())  # still shows 120 repeats<\/code><\/pre>\n\n\n\n<p>Rows are only removed when every single column matches. If one copy has a different phone number or timestamp, it slips through.<\/p>\n\n\n\n<p><strong>How to fix it:<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>df = df.sort_values(\"updated_at\")\ndf = df.drop_duplicates(subset=\"customer_id\", keep=\"last\")\nprint(df&#91;\"customer_id\"].duplicated().sum())  # 0<\/code><\/pre>\n\n\n\n<p>Define duplicates by the column that identifies a record, and keep the most recent version instead of a random one.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>3.<\/strong> <strong>Inconsistent Formatting<\/strong><\/h4>\n\n\n\n<p>Is 04\/05\/2026 the fourth of May or the fifth of April? When two sources use different date styles, pandas may guess wrong without telling you.<\/p>\n\n\n\n<p><strong>The problem:<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>df&#91;\"signup_date\"] = pd.to_datetime(df&#91;\"signup_date\"])\n# 04\/05\/2026 is read as April 5th, but your source meant May 4th<\/code><\/pre>\n\n\n\n<p>There is no error and no warning, only wrong dates that quietly corrupt your analysis.<\/p>\n\n\n\n<p><strong>How to fix it:<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>df&#91;\"signup_date\"] = pd.to_datetime(df&#91;\"signup_date\"], format=\"%d\/%m\/%Y\", errors=\"coerce\")\nprint(df&#91;\"signup_date\"].isnull().sum())  # rows that did not match the format<\/code><\/pre>\n\n\n\n<p>State the format explicitly, and let failures show up as blanks you can inspect instead of silent mistakes.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>4. Typos and Spelling Errors<\/strong><\/h4>\n\n\n\n<p>Free-text fields are full of &#8220;Delhi&#8221;, &#8220;delhi&#8221;, and &#8220;Dehli&#8221;. To a computer, those are three different cities, so your results split into pieces.<\/p>\n\n\n\n<p><strong>The problem:<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>print(df.groupby(\"city\")&#91;\"sales\"].sum())\n# Delhi     540000\n# delhi      32000\n# Dehli       8000<\/code><\/pre>\n\n\n\n<p>One city is reported as three, and every total and chart built on this is wrong.<\/p>\n\n\n\n<p><strong>How to fix it:<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>df&#91;\"city\"] = df&#91;\"city\"].str.strip().str.title()\ndf&#91;\"city\"] = df&#91;\"city\"].replace({\"Dehli\": \"Delhi\"})\nprint(df.groupby(\"city\")&#91;\"sales\"].sum())\n# Delhi     580000<\/code><\/pre>\n\n\n\n<p>Standardize the casing first, then map the remaining typos to the correct label. Keeping the mapping in your code makes the fix repeatable.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>5. Outliers<\/strong><\/h4>\n\n\n\n<p>An age of 250 is clearly a typing error. A purchase of 5 lakh rupees might be a real, valuable customer. Both look strange, but only one is wrong.<\/p>\n\n\n\n<p><strong>The problem:<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>df = df&#91;df&#91;\"income\"] &lt; df&#91;\"income\"].quantile(0.95)]<\/code><\/pre>\n\n\n\n<p>This automatically deletes the top 5% of earners, whether they are errors or your best customers.<\/p>\n\n\n\n<p><strong>How to fix it:<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>q1, q3 = df&#91;\"income\"].quantile(&#91;0.25, 0.75])\niqr = q3 - q1\ndf&#91;\"is_outlier\"] = (df&#91;\"income\"] &lt; q1 - 1.5 * iqr) | (df&#91;\"income\"] &gt; q3 + 1.5 * iqr)\nprint(df&#91;df&#91;\"is_outlier\"]]&#91;&#91;\"customer_id\", \"income\"]])<\/code><\/pre>\n\n\n\n<p>Flag the unusual values, review them one by one, and only then decide to remove, cap, or keep them.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>6. Structural Errors<\/strong><\/h4>\n\n\n\n<p>These are layout problems rather than value problems, such as hidden spaces in headers or the same column named three different ways.<\/p>\n\n\n\n<p><strong>The problem:<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>print(df&#91;\"income\"])\n# KeyError: 'income'  (the real column name is \"Income \" with a trailing space)<\/code><\/pre>\n\n\n\n<p>The data is there, but the code cannot find it, and the error message gives no hint about why.<\/p>\n\n\n\n<p><strong>How to fix it:<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>df.columns = df.columns.str.strip().str.lower().str.replace(\" \", \"_\")\nprint(df.columns.tolist())\nprint(df&#91;\"income\"])<\/code><\/pre>\n\n\n\n<p>Clean the column names first so they all follow a single style. Every later step depends on referring to them correctly.<\/p>\n\n\n\n<p><\/p>\n\n\n\n<p>If you&#8217;re weighing a data science path against analytics, <a href=\"https:\/\/www.guvi.in\/blog\/data-science-vs-data-analytics-career\/\" target=\"_blank\" rel=\"noreferrer noopener\">Data Science vs Data Analytics<\/a> breaks down where this skill fits in each role.<\/p>\n\n\n\n<p><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Data Cleaning Tools Compared<\/strong><\/h2>\n\n\n\n<p>Your tool choice usually depends on dataset size and how comfortable you are writing code.<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><thead><tr><th><strong>Tool<\/strong><\/th><th><strong>Best For<\/strong><\/th><th><strong>Coding Required<\/strong><\/th><th><strong>Handles Large Data<\/strong><\/th><\/tr><\/thead><tbody><tr><td>Pandas (Python)<\/td><td>Custom, repeatable cleaning pipelines<\/td><td>Yes<\/td><td>Yes<\/td><\/tr><tr><td><a href=\"https:\/\/openrefine.org\/\" target=\"_blank\" rel=\"noreferrer noopener nofollow\">OpenRefine<\/a><\/td><td>Exploratory cleaning of messy text data<\/td><td>No<\/td><td>Moderate<\/td><\/tr><tr><td>Excel \/ Power Query<\/td><td>Small datasets, quick fixes<\/td><td>No<\/td><td>No<\/td><\/tr><tr><td>SQL<\/td><td>Cleaning inside a database<\/td><td>Yes<\/td><td>Yes<\/td><\/tr><tr><td>Trifacta \/ Alteryx<\/td><td>Visual, drag-and-drop workflows<\/td><td>No<\/td><td>Yes<\/td><\/tr><\/tbody><\/table><figcaption class=\"wp-element-caption\"><strong>Data Cleaning Tools Compared<\/strong><\/figcaption><\/figure>\n\n\n\n<p>Pandas remains the industry default because you can reuse scripts across projects. If you&#8217;re new to it, HCL GUVI&#8217;s guide on <a href=\"https:\/\/www.guvi.in\/blog\/useful-python-libraries-tools-for-data-science\/\" target=\"_blank\" rel=\"noreferrer noopener\">useful Python libraries for data science<\/a> is a good next stop.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Best Practices Worth Following<\/strong><\/h2>\n\n\n\n<ul>\n<li>Always work on a copy, never the original file<\/li>\n\n\n\n<li>Document every transformation you make and why<\/li>\n\n\n\n<li>Start with a small sample before cleaning the full dataset<\/li>\n\n\n\n<li>Automate repeatable steps, but keep human judgment for edge cases<\/li>\n\n\n\n<li>Re-validate the data after every major cleaning step<\/li>\n<\/ul>\n\n\n\n<p><strong><em>Also Read: <\/em><\/strong><em><a href=\"https:\/\/www.guvi.in\/blog\/best-datasets-for-data-science-projects\/\" target=\"_blank\" data-type=\"link\" data-id=\"https:\/\/www.guvi.in\/blog\/best-datasets-for-data-science-projects\/\" rel=\"noreferrer noopener\">Best Datasets for Data Science Projects<\/a><\/em><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Common Mistakes to Avoid<\/strong><\/h2>\n\n\n\n<ol>\n<li><strong>Cleaning before exploring.<\/strong> Jumping straight into fixes without understanding the data first often causes you to &#8220;fix&#8221; things that weren&#8217;t broken.<\/li>\n\n\n\n<li><strong>Deleting missing data by default.<\/strong> Deletion is the easiest option, but it can quietly remove useful information. Imputation is often the better call.<\/li>\n\n\n\n<li><strong>Ignoring outliers without investigation.<\/strong> Not every <a href=\"https:\/\/www.guvi.in\/blog\/outliers-in-data-science\/\" target=\"_blank\" data-type=\"link\" data-id=\"https:\/\/www.guvi.in\/blog\/outliers-in-data-science\/\" rel=\"noreferrer noopener\">outlier<\/a> is an error. Some represent real, important events.<\/li>\n\n\n\n<li><strong>Skipping documentation.<\/strong> Undocumented cleaning steps make your work impossible to reproduce or audit later.<\/li>\n<\/ol>\n\n\n\n<p><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Conclusion<\/strong><\/h2>\n\n\n\n<p>Data cleaning determines whether you can trust any analysis or model results. Even the best algorithm produces unreliable output when the underlying data is inaccurate, inconsistent, or incomplete. That is why clean data leads to sharper insights, more dependable models, and better decisions. Ultimately, Data Cleaning isn&#8217;t a side task; it&#8217;s one of the most important skills in data science.<\/p>\n\n\n\n<p><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>FAQs<\/strong><\/h2>\n\n\n<div id=\"rank-math-faq\" class=\"rank-math-block\">\n<div class=\"rank-math-list \">\n<div id=\"faq-question-1791810776413\" class=\"rank-math-list-item\">\n<h3 class=\"rank-math-question \">1. What is data cleaning in data science?<\/h3>\n<div class=\"rank-math-answer \">\n\n<p>Data cleaning is the process of finding and fixing errors, duplicates, and missing values in a dataset before using it for analysis or model building.<\/p>\n\n<\/div>\n<\/div>\n<div id=\"faq-question-1791810778936\" class=\"rank-math-list-item\">\n<h3 class=\"rank-math-question \">2. What are the steps of data cleaning?<\/h3>\n<div class=\"rank-math-answer \">\n\n<p>The steps are to understand your data, keep a raw copy, explore, handle missing values, remove duplicates, treat outliers, standardize formats, and validate the result.<\/p>\n\n<\/div>\n<\/div>\n<div id=\"faq-question-1791810779651\" class=\"rank-math-list-item\">\n<h3 class=\"rank-math-question \">3. Why is data cleaning important in data science?<\/h3>\n<div class=\"rank-math-answer \">\n\n<p>A model or dashboard is only as reliable as the data it is fed, so skipping data cleaning leads to misleading results.<\/p>\n\n<\/div>\n<\/div>\n<div id=\"faq-question-1791810780744\" class=\"rank-math-list-item\">\n<h3 class=\"rank-math-question \">4. What is the difference between data cleaning and data preprocessing?<\/h3>\n<div class=\"rank-math-answer \">\n\n<p>Data cleaning fixes errors and inconsistencies in raw data, while preprocessing prepares the cleaned data for modeling.<\/p>\n\n<\/div>\n<\/div>\n<div id=\"faq-question-1791810781550\" class=\"rank-math-list-item\">\n<h3 class=\"rank-math-question \">5. Which tools are used for data cleaning?<\/h3>\n<div class=\"rank-math-answer \">\n\n<p>Common data cleaning tools include Pandas, OpenRefine, Excel or Power Query, SQL, and Trifacta or Alteryx.<\/p>\n\n<\/div>\n<\/div>\n<\/div>\n<\/div>","protected":false},"excerpt":{"rendered":"<p>Data cleaning is the process of finding and fixing errors, duplicates, and missing values in a dataset before it&#8217;s used for analysis or model building. It typically involves: Without this step, even the most advanced analysis will produce misleading results because a model or dashboard is only as reliable as the data it&#8217;s fed. TL;DR [&hellip;]<\/p>\n","protected":false},"author":64,"featured_media":131806,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[16],"tags":[],"views":"20297","authorinfo":{"name":"Abhishek Pati","url":"https:\/\/www.guvi.in\/blog\/author\/abhishek-pati\/"},"thumbnailURL":"https:\/\/www.guvi.in\/blog\/wp-content\/uploads\/2024\/07\/data-cleaning-in-data-science-1-300x116.webp","_links":{"self":[{"href":"https:\/\/www.guvi.in\/blog\/wp-json\/wp\/v2\/posts\/56506"}],"collection":[{"href":"https:\/\/www.guvi.in\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.guvi.in\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.guvi.in\/blog\/wp-json\/wp\/v2\/users\/64"}],"replies":[{"embeddable":true,"href":"https:\/\/www.guvi.in\/blog\/wp-json\/wp\/v2\/comments?post=56506"}],"version-history":[{"count":40,"href":"https:\/\/www.guvi.in\/blog\/wp-json\/wp\/v2\/posts\/56506\/revisions"}],"predecessor-version":[{"id":143191,"href":"https:\/\/www.guvi.in\/blog\/wp-json\/wp\/v2\/posts\/56506\/revisions\/143191"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.guvi.in\/blog\/wp-json\/wp\/v2\/media\/131806"}],"wp:attachment":[{"href":"https:\/\/www.guvi.in\/blog\/wp-json\/wp\/v2\/media?parent=56506"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.guvi.in\/blog\/wp-json\/wp\/v2\/categories?post=56506"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.guvi.in\/blog\/wp-json\/wp\/v2\/tags?post=56506"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}