The Power of GenStage

Our applications must grow and adapt to changing requirements or lessons learned over time. We find ourselves in the best position to implement new requirements by changing very little code. The power of GenStage is to grow with our application. We start simple and add as needed over time.

Let’s look at a few examples of how our application may change over time. We’ll think about how GenStage can help us achieve these goals.

Enforce stricter delivery guarantees for messages

One of the most significant trade-offs with a fully in-memory approach is that a server crash or restart will lose all current in-memory work. We can use a data store such as a SQL database, Redis, or Kafka to store every outgoing message. This would reduce the data loss potential when a server restarts. GenStage helps us here because its core purpose is to handle data requesting and buffering from external data stores. We could adapt our pipeline’s Producer module to read from the data store.

We may get pretty far with an in-memory, no-persistence solution. An excellent saying in software architecture is: “Our software architecture is judged by the decisions we can defer.” This means that we’re able to tackle the critical things up-front and leave ourselves open to tackle currently less critical things in the future—such as pipeline persistence.

Augment an outgoing message with data from a database

Our GenStage-powered pipeline exposes a Worker module that can do anything we want it to do. For example, augmenting messages with data from an external API or database is possible. These external resources often have a maximum throughput, so the maximum concurrency option helps us avoid overwhelming these external data providers. We could also leverage the concept of a GenStage ProducerConsumer to achieve the goal of data augmentation.

Equitable dispatching between users

Our GenStage-based pipeline will currently send items on a first-come-first-served basis. This is great for most applications, but it could be problematic in an environment where a single user (or team of users) has significantly more messages than other users. In this scenario, all users would become slower due to the effect of a single user.