Peer-To-Peer Publish/Subscribe with ZeroMQ

The presence of a broker can considerably simplify the architecture of a messaging system. However, in some circumstances, this might not be the best solution. This includes all the situations where a low latency is critically important, when scaling complex distributed systems, or when the presence of a single point of failure isn’t an option. The alternative to using a broker is, of course, implementing a peer-to-peer messaging system.

Introducing ZeroMQ

If our project is a good candidate for a peer-to-peer architecture, one of the best solutions to evaluate is certainly the ZeroMQ library(also known as zmq or ØMQ). ZeroMQ is a networking library that provides the basic tools to build a large variety of messaging patterns. It’s low-level, extremely fast, and has a minimalistic API, but it offers all the basic building blocks to create a solid messaging system, such as atomic messages, load balancing, queues, and many more. It supports many types of transport, such as in-process channels (inproc://), inter-process communication (ipc://), multicast using the PGM protocol (pgm:// or epgm://), and, of course, the classic TCP (tcp://).

Among the features of ZeroMQ, we can also find tools to implement a Publish/ Subscribe pattern, which is exactly what we need for our example. So, what we’re going to do now is remove the broker (Redis) from the architecture of our chat application and let the various nodes communicate in a peer-to-peer fashion, leveraging the publish/subscribe sockets of ZeroMQ.

Note: A ZeroMQ socket can be considered as an advanced network socket, which provides additional abstractions to help implement the most common messaging patterns. For example, we can find sockets designed to implement publish/subscribe, request/reply, or one-way push communications.

Designing a peer-to-peer architecture for the chat server

When we remove the broker from our architecture, each instance of the chat server has to connect to the other available instances directly in order to receive the messages they publish. In ZeroMQ, we have two types of sockets specifically designed for this purpose: the PUB and SUB sockets. The typical pattern is to bind a PUB socket to a local port where it’ll start listening for incoming subscription requests from the SUB type sockets.

A subscription can have a filter that specifies what messages are delivered to the connected SUB sockets. The filter is a simple binary buffer (so it can also be a string), which will be matched against the beginning of the message (which is also a binary buffer). When a message is sent through the PUB socket, it’s broadcast to all the connected SUB sockets, but only after their subscription filters are applied. The filters will be applied to the publisher side only if a connected protocol is used, such as, for example, TCP.

The following illustration shows the pattern applied to our distributed chat server architecture (with only two instances, for simplicity):