Implementing Message Broker Technologies

As we saw in the previous chapter, standing up and using a minimal Kafka instance can be done relatively quickly. This is great for localized testing. However, it does not translate into a production-grade infrastructure that’s capable of handling the raw volume of events we might see with the application. While every configuration detail is not relevant to developing the domain code and the overall application, there are some points to keep in mind when we’re setting up and configuring Kafka that can impact how software components may process events.

Now, let’s walk through a high-level overview of the components that are needed to run Kafka, as well as relevant implementations and configurations that will enable resiliency and scalability.

Reviewing essential Kafka components

There are three primary components that we must have to establish a functioning Kafka instance. We’ve already talked about the broker, as well as topics. The final piece of the puzzle is the ZooKeeper component. ZooKeeper, according to the official Apache ZooKeeper site, “is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.” In short, it minds the brokers in our Kafka instance and helps facilitate event routing, configuration updates, replication, and leader election. It also has the following properties:

• Event routing ...