Gain insights into data processing and concurrency in Elixir. Explore GenServer, GenStage, Flow, and Broadway to enhance performance, set up pipelines, and handle increased workloads efficiently.

Concurrent Data Processing in Elxir by The Pragmatic Programmer-01.png

myDokcer.tar.gz

ch1_1

ch_3

ch_3.1

ch1_1-task supervisor

ch_4.1

ch_5.1 RabbitMQ

Ch_5.2 RabbitMQ

Most engineers don’t think that concurrency is distinct from programming. Data processing includes transforming information in some way, like reporting, data aggregation, or analytics. On the other hand, concurrency involves significantly improving performance.

This course is designed to help you navigate the world of concurrency tools available in the Elixir ecosystem. You’ll explore popular modules and libraries like GenServer, GenStage, Flow, and Broadway. You’ll be able to ingest data, set up pipelines, and run concurrent processes with Elixir.

By the end of the course, you’ll have a wide range of new techniques to simplify your product, improve code performance, and make your application more resilient to increased workloads.

Concurrent Data Processing in Elixir

# Stop a task

What happens if our task is stuck and never finishes?

While `await/1` stops the task, `yield/1` will leave it running. It is good to stop the task manually by calling `Task.shutdown(task)`. The **`shutdown/1`** function also accepts a timeout and gives the process the last chance to complete before stopping it. If it ends, we will receive the result as expected. We can also stop a process immediately by using the `:brutal_kill` atom as the second argument.


As we can see, using `yield/1` and `shutdown/1` is a bit more work than `await/1`. Deciding what to use depends on our use case. A task timeout often justifies an exception, in which case `await/1` will be more convenient to use. Whenever we require more control over the timeout and shutdown, we can switch to `yield/1` and `shutdown/1`. The important part is that after using `async/1`, we always process the result.  We can do this using either `await/1` or `yield/1` followed by `shutdown/1`.

# Sender example
We are going to update our sender example.

## Update `notify_all`

For our `notify_all/1` logic, we’ll use `await/1` for simplicity. Now we’re going to replace `Task.start/1` with `Task.async/1` and use `Enum.map/2` instead of `Enum.each/2`. This is because we want to map each email string to its corresponding `Task` struct. The struct is needed to retrieve the result from each process:


Learn to stop a task and modify our notify_all function with Task.async.

Getting Started

Easy Concurrency with the Task Module

Long-Running Processes Using GenServer

Data Processing Pipelines with GenStage

Process Collections with Flow

Data Ingestion Pipelines with Broadway

Concluding the Course

Stop a Task and Async Task

Stop a task