Gain insights into data processing and concurrency in Elixir. Explore GenServer, GenStage, Flow, and Broadway to enhance performance, set up pipelines, and handle increased workloads efficiently.

Concurrent Data Processing in Elxir by The Pragmatic Programmer-01.png

myDokcer.tar.gz

ch1_1

ch_3

ch_3.1

ch1_1-task supervisor

ch_4.1

ch_5.1 RabbitMQ

Ch_5.2 RabbitMQ

Most engineers don’t think that concurrency is distinct from programming. Data processing includes transforming information in some way, like reporting, data aggregation, or analytics. On the other hand, concurrency involves significantly improving performance.

This course is designed to help you navigate the world of concurrency tools available in the Elixir ecosystem. You’ll explore popular modules and libraries like GenServer, GenStage, Flow, and Broadway. You’ll be able to ingest data, set up pipelines, and run concurrent processes with Elixir.

By the end of the course, you’ll have a wide range of new techniques to simplify your product, improve code performance, and make your application more resilient to increased workloads.

Concurrent Data Processing in Elixir

So far, we have used two types of supervisors—`Task.Supervisor` and `DynamicSupervisor`. They’re great because they allow us to start processes on demand. However, they are both made with a specific goal in mind, and sometimes we need greater configurability over how the supervisor works.

# Implement a `Supervisor` from scratch

We will often opt for the `Supervisor` behaviour to create a supervisor from scratch. In this section, we create a supervisor using the `Supervisor` behaviour and put it in front of each `Job` process.

We already learned that supervisors could restart child processes and saw how the `:permanent` and `:transient` restart values work. The only setting we haven’t used yet is `:temporary`. This value tells the supervisor to ignore the process and not worry about restarting it even when the child process exits with an error.

Our goal is to make `JobRunner` start an intermediary supervisor for each job, which starts the actual `Job` process. This new supervisor will restart the `Job` process if needed but will have a restart value of `:temporary`. Therefore, it won’t cause further damage if it fails. This figure shows how the supervision tree is going to look after we make all the changes:

Learn to make our supervisor from scratch and add it to our project.

Getting Started

Easy Concurrency with the Task Module

Long-Running Processes Using GenServer

Data Processing Pipelines with GenStage

Process Collections with Flow

Data Ingestion Pipelines with Broadway

Concluding the Course

Implement a Supervisor

Implement a `Supervisor` from scratch

Implement a Supervisor

Implement a Supervisor from scratch

Implement a `Supervisor` from scratch