Pipelines and Competing Consumers in AMQP

Point-to-point communications and competing consumers

In a peer-to-peer configuration, a pipeline is a very straightforward concept to imagine. With a message broker in the middle, though, the relationships between the various nodes of the system are a little bit harder to understand: the broker itself acts as an intermediary for our communications and, often, we don’t really know who is on the other side listening for messages. For example, when we send a message using AMQP, we don’t deliver it to its destination directly, but instead to an exchange and then to a queue. Finally, it’ll be for the broker to decide where to route the message, based on the rules defined in the exchange, the bindings, and the destination queues.

If we want to implement a pipeline and a task distribution pattern using a system like AMQP, we have to make sure that each message is received by only one consumer, but this is impossible to guarantee if an exchange can potentially be bound to more than one queue. The solution, then, is to send a message directly to the destination queue, bypassing the exchange altogether. This way, we can make sure that only one queue will ever receive the message. This communication pattern is called point-to-point communication.

Once we’re able to send a set of messages to a single queue directly, we’re already halfway to implementing our task distribution pattern. In fact, the next step comes naturally: when multiple consumers are listening on the same queue, the messages will be distributed evenly across them, following a fanout distribution pattern. In the context of message brokers, this is better known as the Competing Consumers pattern.

Next, we’re going to reimplement our simple hashsum cracker using AMQP.

Implementing the hashsum cracker using AMQP

We just learned that exchanges are the point in a broker where a message is multicast to a set of consumers, while queues are the place where messages are load balanced. With this knowledge in mind, let’s now implement our brute-force hashsum cracker on top of an AMQP broker (which in our case is RabbitMQ). The following illustration gives us an overview of the system we want to implement.