Introduction to the Broadway Library

The history of message brokers

Message brokers have been around for decades. They were initially developed for large enterprises and allowed software engineers to break complex systems into independent services. The message broker would then act as a middleware between those services, effectively decoupling components that previously depended on each other. This significantly improved maintainability and created systems that are easier to scale and can handle increased workloads.

Message brokers today

Nowadays, message brokers are popular with companies and projects of any size and offer many advanced features. Most message brokers are open source and support all major operating systems. Others are proprietary software but still available at a low cost.

The Broadway library

This chapter looks at Broadway, an Elixir library that makes working with message brokers a breeze. The Broadway library, just like Flow, is built on top of GenStage. Broadway makes it easy to build data-processing pipelines that consume events from external sources. This is why they are also called data-ingestion pipelines. The Broadway library supports the most popular message brokers and requires only a small amount of configuration to run.

Fault tolerance

Last but not least, the GenStage pipeline generated by Broadway is designed according to the current best practices. It is fault-tolerant and performs out-of-the-box graceful shutdowns to ensure a minimal loss of messages when something unexpected happens. It also has several other useful features, such as automatic acknowledgments, dynamic batching, rate limiting, and more. While we can certainly build this all by ourselves using GenStage, with Broadway, we get everything in just a few lines of configuration code.

Gear up

In this chapter, we’ll build a simple ticket-processing system. Along the way, we’ll learn about the Broadway behavior and the callbacks that are available to implement. We will then move to batch processing — another key feature that helps us process data efficiently and leverage concurrency. Finally, we’ll build our own Broadway producer using GenStage and will see how easy it is to use Broadway outside the context of message brokers.

---