Updating the Web Spider

Web spider v4

Now it’s time to adapt our web spider to implement a limited parallel execution flow using the TaskQueue class we’ve just created.

First, we need to split the spider() function into two functions, one simply initializing a new TaskQueue object and another actually executing the spidering task, which we’ll call the spiderTask() function. Then, we need to update the spiderLinks() function to invoke the newly created spiderTask() function and forward the task queue instance received as an input. Here’s what all of this looks like: