Gain insights into data processing and concurrency in Elixir. Explore GenServer, GenStage, Flow, and Broadway to enhance performance, set up pipelines, and handle increased workloads efficiently.

Concurrent Data Processing in Elxir by The Pragmatic Programmer-01.png

myDokcer.tar.gz

ch1_1

ch_3

ch_3.1

ch1_1-task supervisor

ch_4.1

ch_5.1 RabbitMQ

Ch_5.2 RabbitMQ

Most engineers don’t think that concurrency is distinct from programming. Data processing includes transforming information in some way, like reporting, data aggregation, or analytics. On the other hand, concurrency involves significantly improving performance.

This course is designed to help you navigate the world of concurrency tools available in the Elixir ecosystem. You’ll explore popular modules and libraries like GenServer, GenStage, Flow, and Broadway. You’ll be able to ingest data, set up pipelines, and run concurrent processes with Elixir.

By the end of the course, you’ll have a wide range of new techniques to simplify your product, improve code performance, and make your application more resilient to increased workloads.

Concurrent Data Processing in Elixir

# The history of `GenStage`

`GenStage` was initially developed by José Valim, the creator of Elixir, and released in July 2016. As he described it in the official announcement:

>“GenStage is a new Elixir behaviour for exchanging events with back-pressure between Elixir processes.”

The previous chapter uses the `GenServer` behaviour to build long-running server processes. The `GenStage` behaviour, as its name suggests, builds stages. **Stages** are also Elixir processes, and they’re our building blocks for creating data processing pipelines.

# Stages

Stages are simple but very powerful. They receive events and use them to do useful work. They can also send events to the next stage in the pipeline. We can do that by connecting stages to each other and creating something like a chain, as we can see here:


Learn how the GenStage library in Elixir uses stages to build data processing pipelines.


Introduce GenStage

Learn how the GenStage library in Elixir uses stages to build data processing pipelines.

Getting Started

Easy Concurrency with the Task Module

Long-Running Processes Using GenServer

Data Processing Pipelines with GenStage

Process Collections with Flow

Data Ingestion Pipelines with Broadway

Concluding the Course

Introduce GenStage

The history of `GenStage`

Introduce GenStage

The history of GenStage

The history of `GenStage`