Gain insights into data processing and concurrency in Elixir. Explore GenServer, GenStage, Flow, and Broadway to enhance performance, set up pipelines, and handle increased workloads efficiently.

Concurrent Data Processing in Elxir by The Pragmatic Programmer-01.png

myDokcer.tar.gz

ch1_1

ch_3

ch_3.1

ch1_1-task supervisor

ch_4.1

ch_5.1 RabbitMQ

Ch_5.2 RabbitMQ

Most engineers don’t think that concurrency is distinct from programming. Data processing includes transforming information in some way, like reporting, data aggregation, or analytics. On the other hand, concurrency involves significantly improving performance.

This course is designed to help you navigate the world of concurrency tools available in the Elixir ecosystem. You’ll explore popular modules and libraries like GenServer, GenStage, Flow, and Broadway. You’ll be able to ingest data, set up pipelines, and run concurrent processes with Elixir.

By the end of the course, you’ll have a wide range of new techniques to simplify your product, improve code performance, and make your application more resilient to increased workloads.

Concurrent Data Processing in Elixir

## The history of message brokers

**Message brokers** have been around for decades. They were initially developed for large enterprises and allowed software engineers to break complex systems into independent services. The message broker would then act as a middleware between those services, effectively decoupling components that previously depended on each other. This significantly improved maintainability and created systems that are easier to scale and can handle increased workloads.

## Message brokers today

Nowadays, message brokers are popular with companies and projects of any size and offer many advanced features. Most message brokers are open source and support all major operating systems. Others are proprietary software but still available at a low cost.

## The `Broadway` library

This chapter looks at `Broadway`, an Elixir library that makes working with message brokers a breeze. The `Broadway` library, just like `Flow`, is built on top of `GenStage`. `Broadway` makes it easy to build data-processing pipelines that consume events from external sources. This is why they are also called **data-ingestion pipelines**. The `Broadway` library supports the most popular message brokers and requires only a small amount of configuration to run.

## Fault tolerance

Last but not least, the `GenStage` pipeline generated by `Broadway` is designed according to the current best practices.  It is fault-tolerant and performs out-of-the-box graceful shutdowns to ensure a minimal loss of messages when something unexpected happens. It also has several other useful features, such as automatic acknowledgments, dynamic batching, rate limiting, and more. While we can certainly build this all by ourselves using `GenStage`, with `Broadway`, we get everything in just a few lines of configuration code.

## Gear up

In this chapter, we'll build a simple ticket-processing system. Along the way, we'll learn about the `Broadway` behavior and the callbacks that are available to implement. We will then move to batch processing — another key feature that helps us process data efficiently and leverage concurrency. Finally, we'll build our own `Broadway` producer using `GenStage` and will see how easy it is to use `Broadway` outside the context of message brokers.

---

# The history of message brokers

**Message brokers** have been around for decades. They were initially developed for large enterprises and allowed software engineers to break complex systems into independent services. The message broker would then act as a middleware between those services, effectively decoupling components that previously depended on each other. This significantly improved maintainability and created systems that are easier to scale and can handle increased workloads.

# Message brokers today

Nowadays, message brokers are popular with companies and projects of any size and offer many advanced features. Most message brokers are open source and support all major operating systems. Others are proprietary software but still available at a low cost.

# The `Broadway` library

This chapter looks at `Broadway`, an Elixir library that makes working with message brokers a breeze. The `Broadway` library, just like `Flow`, is built on top of `GenStage`. `Broadway` makes it easy to build data-processing pipelines that consume events from external sources. This is why they are also called **data-ingestion pipelines**. The `Broadway` library supports the most popular message brokers and requires only a small amount of configuration to run.

# Fault tolerance

Last but not least, the `GenStage` pipeline generated by `Broadway` is designed according to the current best practices.  It is fault-tolerant and performs out-of-the-box graceful shutdowns to ensure a minimal loss of messages when something unexpected happens. It also has several other useful features, such as automatic acknowledgments, dynamic batching, rate limiting, and more. While we can certainly build this all by ourselves using `GenStage`, with `Broadway`, we get everything in just a few lines of configuration code.

# Gear up

In this chapter, we'll build a simple ticket-processing system. Along the way, we'll learn about the `Broadway` behavior and the callbacks that are available to implement. We will then move to batch processing — another key feature that helps us process data efficiently and leverage concurrency. Finally, we'll build our own `Broadway` producer using `GenStage` and will see how easy it is to use `Broadway` outside the context of message brokers.

---

Get introduced to message brokers and data-ingestion pipelines.

Getting Started

Easy Concurrency with the Task Module

Long-Running Processes Using GenServer

Create a GenServer and Supervisor From Scratch

Data Processing Pipelines with GenStage

Process Collections with Flow

Data Ingestion Pipelines with Broadway

Concluding the Course

Introduction to the Broadway Library

The history of message brokers