Welcome

LIVE Classes

Courses

Practice Platforms

Leaderboard

Rewards

Referral

Profile

Finish

Welcome LIVE Classes Courses Practice Platforms Leaderboard Rewards Referral Profile Finish

Welcome to HCL GUVI

Hey there! Welcome to HCL GUVI—Grab Your Vernacular Imprint—where tech learning is easy, fun, and curated specially for you. Incubated by IIT Madras & IIM Ahmedabad in 2014 and now part of HCL Group, we're making quality tech education accessible to all.

Join 3M+ learners breaking barriers and upskilling for a brighter future. We're here to guide you every step of the way! 🚀

LIVE Classes

Zen Classes are HCL GUVI's most refined and flagship product—live, expert-led tech programs for beginners and pros. With IITM Pravartak affiliations, master Full-Stack, Data Science, DevOps, UI/UX, and more in multiple languages!

Explore More

Courses

Looking for flexibility? HCL GUVI's 200+ self-paced courses let you learn anytime, anywhere! From free lessons to IIT-M & Autodesk-certified programs, gain in-demand skills in your preferred language.

Explore More

Practice Platforms

Enhance your coding skills with HCL GUVI's Practice Platforms—interactive, structured, and designed to help you master programming effortlessly.

CodeKata:

A structured coding practice platform with 1500+ coding problems designed by industry experts. Ideal for beginners and professionals preparing for tech interviews with real-world coding challenges.

Try Now >

WebKata:

An interactive platform to master HTML, CSS, JavaScript, and Bootstrap with a live coding environment. Perfect for hands-on web development practice without any setup.

Try Now >

SQLKata:

A practice ground for mastering SQL queries used in real-world applications. Write, optimize, and refine your queries to build strong database skills.

Try Now >

Debugging:

Hone your bug-fixing skills with real-world debugging challenges in Python, C++, JavaScript, and Golang. More languages coming soon!

Try Now >

IDE:

A free online compiler supporting 20+ programming languages with auto-complete, debugging, and AI-powered code generation—all in the cloud!

Try Now >

Leaderboard

Climb the leaderboard as you earn Geekoins by learning and practicing! The top scorers get featured, making learning competitive and rewarding. Keep going—you could be next!

Explore More

Rewards

Earn Geekoins by watching videos and practicing problems, then redeem them for exciting rewards. The more you engage, the more you win!

Explore More

Referral

Love learning with HCL GUVI? Share it with friends! Invite them using your unique link or code and unlock exciting rewards—Amazon vouchers, iPhones, and more. A Win-Win.

Explore More

Profile

Your HCL GUVI profile is your digital portfolio! Track progress, showcase skills, add projects, and build a resume. Keep it updated—opportunities await!

Explore More

That's It! You Are Ready!

You're all set to dive into your learning journey with HCL GUVI. Explore, upskill, and make each step count—exciting possibilities awaits!

Home
Python 3
Scraping Website Data

Understanding Scraping Website Data

Scraping Product Names from ConsumerReports Website
importing bs4, requests and fake_useragent modules
initializing the UserAgent object
getting the reponse from the page using get method of requests module
storing the content of the page in a variable
creating BeautifulSoup object

Scraping Product Names from ConsumerReports Website

NOTE: This tutorial is just for educational purpose and we request the reader to not use the code to carry out harm to the website in any form whatsoever.

In this tutorial we will learn how to actually scrape data off any website. The website from which we will get the data is ConsumerReports website. We will be requesting data from this URL and then collect the product names list from it.

Let the scraping begin...

importing bs4, requests and fake_useragent modules

import bs4 import requests from fake_useragent import UserAgent

initializing the UserAgent object

user_agent = UserAgent() url = "https://www.consumerreports.org/cro/a-to-z-index/products/index.htm"

getting the reponse from the page using get method of requests module

page = requests.get(url, headers={"user-agent": user_agent.chrome})

storing the content of the page in a variable

html = page.content


By this step, we already have the complete source code for the webpage stored in our variable `html`. Now let's create BeautifulSoup object. You can even try and run the `prettify` method.

```python

creating BeautifulSoup object

soup = bs4.BeautifulSoup(html, "html.parser")


We have also created the BeautifulSoup object, now what? How do we know which tag to find and extract from the HTML code. Should we search HTML code for it? No way!

Remember in the first tutorial of this series when we introduced the term web scraping, we did share a technique with you, where we could use the **Chrome browser's Developer tool** to find the HTML code for any webpage element.(other browsers like Firefox etc too have there own developer tools which can also be used.)

Open the Developer Tools(in chrome browser) by pressing **F12** key if you are using Windows and **Option + Command + I** if you are a Mac user.

Click on the top-left corner button:

![](https://static.guvi.in/learner-hub/webscrapingtutorial/1768222949199.png)

And then hover your mouse cursor on the Product list entries to find their HTML tags:

![](https://static.guvi.in/learner-hub/webscrapingtutorial/1768222968013.png)

See, how simple it was. And with this we have successfully scraped data from a website.

Recommended Handbooks

4.7

C++ Handbook

Level up your programming skills with our C++ Tutorials hub, guiding you from beginner to advanced. Start your journey now!

English

6254

3 Hrs

4.7

Python Basics Handbook

Level up your programming skills with our Python Tutorials hub, guiding you from beginner to advanced. Start your journey now!

English

7323

3.5 Hrs

4.6

Javascript Handbook

Level up your programming skills with our JavaScript Tutorials hub, guiding you from beginner to advanced. Start your journey now!

English

6188

2 Hrs

ReactJS Projects Handbook

Learn ReactJS by building projects that mirror real-world applications. Strengthen your skills with step-by-step guidance and hands-on coding experience.

English

2.5 Hrs

Computer Networks Tutorial

A complete guide to computer networking, from fundamentals to protocols, routing, addressing, and real-world data communication.

English

1.5 Hrs

Operating System Tutorial

Your complete guide to Operating Systems, from fundamentals to advanced topics like memory management, scheduling, threads, and deadlock handling.

English

1 Hr

DBMS and SQL Tutorial

A complete handbook to guide you through DBMS fundamentals and SQL mastery, perfect for building data-driven applications, managing data systems, or preparing for database roles.

English

0.5 Hr

Java Tutorial

Beginner-friendly Java handbook covering core concepts, OOP principles, and hands-on programming examples.

English

2 Hrs

C Language Tutorial

A step-by-step C programming handbook for beginners. Understand C syntax, logic, memory, and hands-on coding to build solid programming foundations.

English

0.5 Hr

PHP Tutorial

Step-by-step PHP handbook for web developers. Master server-side scripting with practical code and concepts.

English

0.5 Hr

Android Tutorial

Beginner-friendly Android handbook covering app fundamentals, UI design, and hands-on development concepts.

English

1 Hr

Linux Guide Tutorial

A practical Linux handbook covering command-line basics, file management, and system operations.

English

2.5 Hrs

Data Structures and Algorithms Tutorial

Learn core data structures and algorithms with practical examples to improve coding efficiency and problem-solving skills.

English

0.5 Hr

Computer Architecture

A beginner-friendly guide to computer architecture covering processors, memory, and system-level concepts.

English

0.5 Hr

HTML 5 References Tutorial

A handy HTML5 reference guide covering modern tags, attributes, and semantic elements.

English

1.5 Hrs

Docker Tutorial

A hands-on Docker handbook covering containers, images, and modern application deployment basics.

English

0 Hr

GIT (Using Github) Tutorial

A hands-on Git and GitHub handbook for managing code, tracking changes, and collaborating on projects.

English

0.5 Hr

Go Language Tutorial

A beginner-friendly Go handbook covering core language concepts and modern backend programming.

English

0.5 Hr

GIT Guide

A practical Git guide covering version control basics, branching, and real project workflows.

English

1 Hr

CSS Tutorial

A beginner-friendly CSS handbook covering page styling, layouts, and responsive design basics.

English

1 Hr

Advanced Data Structures

A focused handbook covering advanced data structures for efficient and scalable problem solving.

English

0 Hr

Spring Framework Tutorial

A hands-on Spring Framework handbook covering core concepts and backend development fundamentals.

English

1 Hr

Spring Boot Tutorial

A practical Spring Boot handbook focused on building and running modern Java backend applications.

English

0.5 Hr

Kotlin Tutorial

A beginner-friendly Kotlin handbook covering modern language features and real-world development concepts.

English

1 Hr

Apache Cordova Tutorial

A hands-on Apache Cordova handbook for building cross-platform mobile apps with web technologies.

English

0 Hr

Python Tutorial

A beginner-friendly Python handbook covering core concepts and practical programming examples.

English

1.5 Hrs

SASS-SCSS Tutorial

A hands-on SASS / SCSS handbook for writing clean, reusable, and scalable stylesheets.

English

0.5 Hr

MongoDB Tutorial

A hands-on MongoDB handbook covering NoSQL concepts and modern database operations.

English

0.5 Hr

Numpy Tutorial

A hands-on NumPy handbook for fast numerical computation and data manipulation using Python.

English

1.5 Hrs

PL-SQL Tutorial

A hands-on PL/SQL handbook for writing procedural database programs and business logic.

English

0.5 Hr

Python Built-in Functions Tutorial

A handy reference guide to Python’s built-in functions for cleaner and faster coding.

English

0.5 Hr

Pandas Tutorial

A hands-on Pandas handbook for data manipulation, cleaning, and analysis using Python.

English

2.5 Hrs

Elasticsearch Tutorial

A hands-on Elasticsearch handbook covering indexing, searching, and data analysis concepts.

English

0 Hr

Matplotlib Tutorial

A hands-on Matplotlib handbook for creating charts and visualizing data using Python.

English

0.5 Hr

Networking with Python

A hands-on handbook for building network-enabled applications using Python.

English

0.5 Hr

Tkinter Tutorial

A hands-on Tkinter handbook for building desktop applications with Python.

English

0.5 Hr

Java Programs Tutorial

A hands-on Java programs handbook for practicing core concepts and problem-solving in Java.

English

2 Hrs

Understanding Scraping Website Data

Contents

Scraping Product Names from ConsumerReports Website

importing bs4, requests and fake_useragent modules

initializing the UserAgent object

getting the reponse from the page using get method of requests module

storing the content of the page in a variable

creating BeautifulSoup object

Web Scraping Tutorial

Recommended Handbooks

C++ Handbook

Python Basics Handbook

Javascript Handbook

ReactJS Projects Handbook

Computer Networks Tutorial

Operating System Tutorial

DBMS and SQL Tutorial

Java Tutorial

C Language Tutorial

PHP Tutorial

Android Tutorial

Linux Guide Tutorial

Data Structures and Algorithms Tutorial

Computer Architecture

HTML 5 References Tutorial

Docker Tutorial

GIT (Using Github) Tutorial

Go Language Tutorial

GIT Guide

CSS Tutorial

Advanced Data Structures

Spring Framework Tutorial

Spring Boot Tutorial

Kotlin Tutorial

Apache Cordova Tutorial

Python Tutorial

SASS-SCSS Tutorial

MongoDB Tutorial

Numpy Tutorial

PL-SQL Tutorial

Python Built-in Functions Tutorial

Pandas Tutorial

Elasticsearch Tutorial

Matplotlib Tutorial

Networking with Python

Tkinter Tutorial

Java Programs Tutorial