Search⌘ K
Home
Courses
Mastering Web Scraping Using Python: From Beginner to Advanced

Introduction to Requests

Explore the basics of the Python Requests library to send HTTP requests and receive server responses. Understand how to spoof headers like User-Agent and manage cookies to bypass restrictions. Learn to retrieve web content effectively for your scraping projects.

We'll cover the following...

We have covered how a browser communicates with a website server by sending an HTTP request and receiving an HTML response that includes the Document Object Model (DOM) structure. Now, we plan to implement the same procedure in our script to ensure that it accurately emulates the actions of a browser. Our primary objective is to replicate the behavior of a browser to accomplish our desired outcomes.

The requests library

It is a Python library that enables us to send HTTP requests to website servers and quickly receive the response objects.

Python 3.8
import requests
r = requests.get('https://books.toscrape.com/')
print("Request URL: ", r.url)
print("Request status code: ", r.status_code)
print("Response headers: ", r.headers)
# Prints the text chuck that holds the <title> tag in the HTML DOM returned.
print("Page's title: ", r.text[360:425])

The above code sends an HTTP request to the Books to Scrape website and retrieves the response object.

The response object has several attributes, such as:

  • object.URL: The address of the site being requested. ...