Implement Worker Queues with Lists

Worker queues

Redis lists support blocking operations (BLPop, BRPop, etc.), which makes it possible to wait for a specified time duration if there is no item in the list. As soon as an item is added to the list—for example, by another application using the LPush method—that item can be retrieved and processed. This allows us to decouple applications and build asynchronous, event-driven solutions. This also known as the worker queue pattern, where the list acts as a queue; the producer applications add items to this queue and the worker applications (consumers) dequeue and process them.

Let’s learn how this works with the help of an application.

Email processor application overview

Sending emails is a common requirement for many websites. This is a perfect use case for leveraging background processing, because we don’t want the user or client to wait while the email is being sent. We’ll go over this a hypothetical solution based on the combination of a REST endpoint and a background job consumer implemented using a Redis list.

This application consists of two parts, outlined below.

Use cases

The following use cases have been implemented in the application:

• A REST endpoint to accept emails: A JSON payload representing the email information (recipient and message) is sent to a REST endpoint using HTTP POST, and the HTTP handler uses the LPush method to add this information to a Redis list.

• A background job to process email requests: A separate consumer application uses the BRPop method to retrieve and process email tasks from the job.

Note: The actual email sending process is not implemented as a part of this application.

Code explanation

Let’s walk through the important parts of the code. First, let’s take a look at the REST API that accepts email requests and sends them to the list.

Imports

Here are some of the package imports: