Navigate Through Categories

Now, let’s implement the navigation among the category pages. This is useful because we need to capture all the books under respective categories.

Approach

The first thing we need to figure out is how to automate this with Puppeteer. To answer that, we need to think through how we usually navigate among the categories on the webpage as users. We can see two approaches to this.

On the web page, clicking on a category brings us to the respective category page. If we can simulate that clicking behavior with Puppeteer, we can easily navigate among categories.

The other approach is that when we click a category, we are brought to a different page with a different URL. If we can extract that URL from the link, we can easily navigate to category pages. So, both options are possible approaches for this, and we can follow the easiest way to simplify the implementation.

Implementing the approach

We'll implement the navigation URL-based approach since it is easy to implement. First, weextract the URLs for each category page and then use them to navigate to the corresponding category page. In the HTML structure, we can observe that there are anchor elements, including the link to the respective category page.