Mastering Web Scraping Using Python: From Beginner to Advanced/

...

/

Introduction to Requests

Introduction to Requests

Discover the Requests library and header spoofing.

We'll cover the following...

We have covered how a browser communicates with a website server by sending an HTTP request and receiving an HTML response that includes the Document Object Model (DOM) structure. Now, we plan to implement the same procedure in our script to ensure that it accurately emulates the actions of a browser. Our primary objective is to replicate the behavior of a browser to accomplish our desired outcomes.

The requests library

It is a Python library that enables us to send HTTP requests to website servers and quickly receive the response objects.

Press + to interact
import requests
r = requests.get('https://books.toscrape.com/')
print("Request URL: ", r.url)
print("Request status code: ", r.status_code)
print("Response headers: ", r.headers)
# Prints the text chuck that holds the <title> tag in the HTML DOM returned.
print("Page's title: ", r.text[360:425])

The above code sends an HTTP request to the Books to Scrape website and retrieves the response object.

The response object has several attributes, such as:

  • object.URL: The address of the site being requested. ...