Gain insights into data processing and concurrency in Elixir. Explore GenServer, GenStage, Flow, and Broadway to enhance performance, set up pipelines, and handle increased workloads efficiently.

Concurrent Data Processing in Elxir by The Pragmatic Programmer-01.png

myDokcer.tar.gz

ch1_1

ch_3

ch_3.1

ch1_1-task supervisor

ch_4.1

ch_5.1 RabbitMQ

Ch_5.2 RabbitMQ

Most engineers don’t think that concurrency is distinct from programming. Data processing includes transforming information in some way, like reporting, data aggregation, or analytics. On the other hand, concurrency involves significantly improving performance.

This course is designed to help you navigate the world of concurrency tools available in the Elixir ecosystem. You’ll explore popular modules and libraries like GenServer, GenStage, Flow, and Broadway. You’ll be able to ingest data, set up pipelines, and run concurrent processes with Elixir.

By the end of the course, you’ll have a wide range of new techniques to simplify your product, improve code performance, and make your application more resilient to increased workloads.

Concurrent Data Processing in Elixir

# Introduction

In addition to `map` and `filter`, the `reduce` function in the `Enum` module is another frequently used tool when working with data. It is also the backbone of other functions in this module, including `group_by`, `uniq_by`, and `take_sort`. This section shows how to use `Flow.reduce/3` to count the number of working airports in each country present in the dataset, in parallel. Then, we’ll move on to other convenience functions to group and sort data. 

# How `reduce` works 

Before we begin, it's useful to understand how `reduce` works in `Flow`. In `reduce`, we have something called an `accumulator` that holds the latest state. The `accumulator` could be of any data type. Each element in the list we want to process is given to a reducer function, followed by the `accumulator` value, often shortened to `acc` for brevity. Once all items in the list have been transformed using the reducer function, the last `accumulator` value results from the `reduce` operation. Here is the blueprint of `Enum.reduce/2`:

```elixir
Enum.reduce(enumerable, fn item, acc -> 
  # Inside the reducer function we can 
  # use the `item` event, and
  # return the `acc` value.
  acc 
end)
```


# Problems with `Enum.reduce`

Like with `Flow.map/2`, each reducer function runs concurrently by default. However, we'll see unexpected results if we directly replace `Enum.reduce/2` with `Flow.reduce/3` in our code. Since each reducer function runs in its process, it does not know other accumulator values. The final outcome will be the combined list of all results, which contains duplicates. Let's demonstrate the issue by adding `Flow.reduce/3` to `open_airports/0`, just before `Enum.to_list/1`:

Learn about the reduce function in the Flow module.

Getting Started

Easy Concurrency with the Task Module

Long-Running Processes Using GenServer

Data Processing Pipelines with GenStage

Process Collections with Flow

Data Ingestion Pipelines with Broadway

Concluding the Course

The reduce Function

Introduction