Gain insights into data processing and concurrency in Elixir. Explore GenServer, GenStage, Flow, and Broadway to enhance performance, set up pipelines, and handle increased workloads efficiently.

Concurrent Data Processing in Elxir by The Pragmatic Programmer-01.png

myDokcer.tar.gz

ch1_1

ch_3

ch_3.1

ch1_1-task supervisor

ch_4.1

ch_5.1 RabbitMQ

Ch_5.2 RabbitMQ

Most engineers don’t think that concurrency is distinct from programming. Data processing includes transforming information in some way, like reporting, data aggregation, or analytics. On the other hand, concurrency involves significantly improving performance.

This course is designed to help you navigate the world of concurrency tools available in the Elixir ecosystem. You’ll explore popular modules and libraries like GenServer, GenStage, Flow, and Broadway. You’ll be able to ingest data, set up pipelines, and run concurrent processes with Elixir.

By the end of the course, you’ll have a wide range of new techniques to simplify your product, improve code performance, and make your application more resilient to increased workloads.

Concurrent Data Processing in Elixir

# The `Task.async/1` task

To retrieve the result of a function, we use **`Task.async/1`**. It returns a `%Task{}` struct which we can assign to a variable for later use. We can give it a try in IEx like this:

This is the executable command:

task = Task.async(fn -> Sender.send_email("hello@world.com") end)

Here is the output we get:

```shell
iex> task = Task.async(fn -> Sender.send_email("hello@world.com") end)
%Task{
  owner: #PID<0.163.0>,
  pid: #PID<0.165.0>,
  ref: #Reference<0.1475641457.1291583489.28556>
} 
Email to hello@world.com sent
```
The `send_email/1` code now runs in the background. In the meantime, we are free to do other work we need. We can add more business logic or even start other tasks. When we need the actual result of the task, we can retrieve it using the `task` variable. Let’s have a closer look at what this variable contains:

* **`owner`**: This is the process’s PID that starts the `Task` process.

* **`pid`**: This is the `Task` process’s identifier.

* **`ref`**: This is the process monitor reference.

# The limitations of `Task.start/1`

The `Task.start/1` function has one limitation by design: it does not return the executed function’s result. This may be useful in some cases, but we could need the result for something else most of the time. It would be great to modify our code and return a meaningful result when all emails are successfully sent.


# Playground

The executable command mentioned above can be executed here: 

Learn to use Task.async, which has the property to return a struct containing the executed task’s result.

Getting Started

Easy Concurrency with the Task Module

Long-Running Processes Using GenServer

Data Processing Pipelines with GenStage

Process Collections with Flow

Data Ingestion Pipelines with Broadway

Concluding the Course

Retrieve a Task’s Results

The `Task.async/1` task

Retrieve a Task’s Results

The Task.async/1 task

The `Task.async/1` task