Search⌘ K
Home
Courses
Web Scraping with Puppeteer

Debug Issues

Discover how to debug issues in Puppeteer by turning off headless mode to visually inspect pages, applying slowMo to slow down actions, adding informative log statements, and monitoring network requests. This lesson helps you identify root causes and improve the reliability of your web scraping scripts.

We'll cover the following...

Common approaches

Web scraping can be challenging due to the dynamic nature of websites, varying page structures, and potential network or runtime errors. Understanding how to identify and address these issues will enhance our web scraping skills and improve the reliability of our Puppeteer scripts.

Turn off headless

When running Puppeteer in headless mode, the browser window is not visible, which makes it challenging to visually inspect the page and see the actual rendering of the elements. By disabling headless mode, we can see the web page as it appears in a regular browser, allowing us to identify any visual issues or discrepancies.

We can configure headless when creating the browser instance in Puppeteer like below.

Node.js
const browser = await puppeteer.launch({headless: false});

The code snippet below shows this behavior when we turn off the headless mode.

{
  "name": "puppeteer-learn",
  "version": "1.0.0",
  "description": "",
  "type": "commonjs",
  "main": "index.js",
  "scripts": {
    "start": "node index.js"
  },
  "keywords": [],
  "author": "",
  "license": "MIT",
  "dependencies": {
    "puppeteer": "19.8.3"
  }
}
Execution when the headless mode is turned off

Here, in line 9, we have turned off the headless mode. It's important to note that while turning off headless mode can be helpful for debugging purposes, it may also affect the performance and speed of our web scraping ...