Gain insights into data processing and concurrency in Elixir. Explore GenServer, GenStage, Flow, and Broadway to enhance performance, set up pipelines, and handle increased workloads efficiently.

Concurrent Data Processing in Elxir by The Pragmatic Programmer-01.png

myDokcer.tar.gz

ch1_1

ch_3

ch_3.1

ch1_1-task supervisor

ch_4.1

ch_5.1 RabbitMQ

Ch_5.2 RabbitMQ

Most engineers don’t think that concurrency is distinct from programming. Data processing includes transforming information in some way, like reporting, data aggregation, or analytics. On the other hand, concurrency involves significantly improving performance.

This course is designed to help you navigate the world of concurrency tools available in the Elixir ecosystem. You’ll explore popular modules and libraries like GenServer, GenStage, Flow, and Broadway. You’ll be able to ingest data, set up pipelines, and run concurrent processes with Elixir.

By the end of the course, you’ll have a wide range of new techniques to simplify your product, improve code performance, and make your application more resilient to increased workloads.

Concurrent Data Processing in Elixir

# Dynamic batching

Static batching is great when we know how to group our messages in advance. However, often messages contain dynamic data, which means that static batching is not going to work. Thankfully, `Broadway` gives us the option to define batches at runtime using `:batch_key`. This is also called **dynamic batching**.

To use dynamic batching, we set `:batch_key` on the message using `Broadway.Message.put_batch_key/2` in our `handle_message/3` callback. The batch key then partitions the data within each batch for each batcher group.

We can demonstrate how dynamic batching works by improving our email notifications. Right now, if a user makes multiple bookings, they get multiple emails. This won’t be very customer-friendly and will also increase our bill from the email service provider.

# Batch ticket confirmations

Instead, let's batch ticket confirmations and send a single email to the customer. We group notifications by email address and prevent sending too many emails at once by introducing a ten-second window.

To accomplish this, we create another pipeline called `NotificationsPipeline`. We create the `notifications_pipeline.ex` file in the `lib` directory and add the following logic:


Learn about a batching technique called dynamic batching.

Getting Started

Easy Concurrency with the Task Module

Long-Running Processes Using GenServer

Data Processing Pipelines with GenStage

Process Collections with Flow

Data Ingestion Pipelines with Broadway

Concluding the Course

Use Dynamic Batching

Dynamic batching