What Is a Message Broker?

Simply put, a message broker is a piece of middleware that facilitates (or brokers) communication between two systems. While this sounds minimalistic, this basic concept can be expounded on for use cases such as the producer-consumer pattern, among others. Given the versatility of this type of middleware, it can be tempting to add additional processing features. Knowing when to add functionalities, such as data enrichment or transformation, is key to preventing product lock-in with a particular broker.

By means of domain-driven design (DDD) standards, keeping the implementation flexible while staying prescriptive on the interface functionality will yield the most consistent results. While we might not encounter a domain project that has 10 different concrete implementations of a broker or even a repository, it’s not uncommon to have a few different options available. Often, we’ll see implementation classes for not only brokers but at least one type of persistence store, whether that’s a cache, a database, or even a flat file.

Terminology aside, there’s a vast array of message brokers out there. Some are free to use, while others are commercial software that requires licensing and payment. Regardless of pricing, though, brokers tend to fall into three main categories:

• Queue-based technology

• Cache-based technology

• Stream-based technology

Let’s look at each type of broker, along with some general examples that fall into each category.

Queue-based technology

First, we’ll examine brokers that are queue-based, meaning there’s a conceptual construct that defines a queue or list where the messages sit. Processing those queued messages (also known as dequeuing) pulls them from the current queue, performs some sort of action on them, and yields a result that can be placed in another queue for continued processing.

The components that perform this processing could be executables, scripts, web services, microservices, or even built-in functionality with the broker itself. While discussing product-specific capabilities is beyond the scope of this course, the following products and their related documentation pages discuss what’s possible with queue-based message brokers:

• RabbitMQ: This is an open-source message queue and broker that allows us to use regular queue-based messaging and advanced message queueing protocol (AMQP) message routing.

• ActiveMQ: This is a Java-based platform that uses AMQP message routing.

• Azure Service Bus: This is a cloud-native message queue and relay specifically implemented in Microsoft Azure.

• WebSphere Message Broker: This is a message broker and transformation engine authored by IBM. This is commonly used with IBM MQ.

• IBM MQ: This is a messaging platform that heavily utilizes queues and topics for message storage and routing, authored by IBM.

While queue-based technologies might be more immediately familiar to some, other storage options are available. Next, we’ll discuss the concept of caching and some related technologies that support it.

Cache-based technology

Cache-based brokers tend to rely on underlying technologies that are used to cache data. Using caching mechanisms for critical messages may not be the best option. By nature, the assumption is that a cache can be invalidated at a predetermined time or even at a random time. While it’s possible to associate persistent disk storage with a caching mechanism, it’s not normally a requirement. A typical use case for caching is to store temporary data as a convenience for another consuming component. Relying on an inherently transient data store does not allow for patterns, such as guaranteed delivery, to be completely enforceable.

Again, how the mechanism is applied will be driven by the needs of the application and the business specifications behind it. Some popular caching technologies are as follows:

• Redis

• Memcached

• Content delivery network (CDN)

Stream-based technology

Stream-based brokers, in many cases, are like queue-based brokers. For example, there are generally places for messages to be sent (queues vs. topics), some sort of computation or action to be taken, and a resulting message to be produced. A key difference between queue-based brokers and stream-based brokers is how the mechanism to receive, process, and retrieve data is handled. While queue-based technologies can be clustered to enable concurrency, stream-based systems provide concurrency and near–real-time visibility to data out of the box. This is facilitated by the lack of complex routing rules that can be found in a queue-based system. Stream-based systems allow raw information to be provided and the lack of potentially complex routing logic keeps touch time on messages very low. The following illustration shows the difference between a traditional queueing system and a streaming system in terms of concurrency, throughput, and resiliency: