Difficulties of Asynchronous Programming: Web Spider

Losing control of asynchronous code in JavaScript is undoubtedly easy. Closures and in-place definitions of anonymous functions allow for a smooth programming experience that doesn’t require the developer to jump to other points in the codebase. This is perfectly in line with the KISS principle (Keep It Simple, Stupid); it’s simple, it keeps the code flowing, and we get it working in less time. Unfortunately, sacrificing qualities such as modularity, reusability, and maintainability will, sooner or later, lead to the uncontrolled proliferation of callback nesting, functions growing in size, and poor code organization. Most of the time, creating callbacks as in-place functions is not strictly required, so it’s more a matter of discipline than a problem related to asynchronous programming. Recognizing that our code is becoming unwieldy or, even better, knowing in advance that it might become unwieldy and then acting accordingly with the most adequate solution, is what differentiates a novice from an expert.

Creating a simple web spider

To explain this problem, we’ll create a little web spider (a command-line application that takes in a web URL as input and downloads its contents locally into a file). In the code presented in this chapter, we’re going to use a couple of npm dependencies:

• superagent: A library to streamline HTTP calls

• mkdirp: A small utility to create directories recursively

Also, we’ll often refer to a local module named ./utils.js, which contains some helpers that we’ll be using in our application. We’ll omit the contents of this file for brevity, but the full implementation, along with a package.json file containing the full list of dependencies, can be found in the official repository.

The core functionality of our application is contained inside a spider.js module. Let’s see how it looks. To start with, let’s load all the dependencies that we’re going to use.