Gain insights into data processing and concurrency in Elixir. Explore GenServer, GenStage, Flow, and Broadway to enhance performance, set up pipelines, and handle increased workloads efficiently.

Concurrent Data Processing in Elxir by The Pragmatic Programmer-01.png

myDokcer.tar.gz

ch1_1

ch_3

ch_3.1

ch1_1-task supervisor

ch_4.1

ch_5.1 RabbitMQ

Ch_5.2 RabbitMQ

Most engineers don’t think that concurrency is distinct from programming. Data processing includes transforming information in some way, like reporting, data aggregation, or analytics. On the other hand, concurrency involves significantly improving performance.

This course is designed to help you navigate the world of concurrency tools available in the Elixir ecosystem. You’ll explore popular modules and libraries like GenServer, GenStage, Flow, and Broadway. You’ll be able to ingest data, set up pipelines, and run concurrent processes with Elixir.

By the end of the course, you’ll have a wide range of new techniques to simplify your product, improve code performance, and make your application more resilient to increased workloads.

Concurrent Data Processing in Elixir

Our `SendServer` implementation is far from ideal. Remember that `send_email/1` pauses the process for three seconds. If we try running `GenServer.cast(pid, {:send, "hello@email.com"})` several times in a row, GenServer acknowledges all messages by returning `:ok`. However, it doesn’t perform the work straight away if it is busy doing something else. After all, it’s just a single process doing all the work. All these messages are placed in the process’s message queue and executed one by one.

Even worse, if we try using `GenServer.call(pid, :get_state)` while the process is still busy, we get a timeout error. By default, the `GenServer.call/3` will give an error after 5000 ms. We can tweak this by providing an optional third argument. However, this is still not a practical solution.

# Use the `Task` module with `GenServer`

We fix this by building a significantly improved job processing system. It will be capable of performing any job concurrently, such as sending emails and retrying the jobs that fail. All we have to do is change our approach slightly.  

We can free up our `GenServer` process by using `Task.start/1` and `Task.async/2` to run the code concurrently. Upon completion, the `Task` process sends messages back to `GenServer`, which we can process with `handle_info/2`. For more information, see the [documentation](https://hexdocs.pm/elixir/Task.Supervisor.html#async_nolink/3) for `Task.Supervisor.async_nolink/3`, which contains some useful examples.

___


Learn some of the potential limitations of our GenServer and how we can go about correcting them.

Getting Started

Easy Concurrency with the Task Module

Long-Running Processes Using GenServer

Data Processing Pipelines with GenStage

Process Collections with Flow

Data Ingestion Pipelines with Broadway

Concluding the Course

Make Improvements to Our GenServer

Limitations of our `GenServer`

Make Improvements to Our GenServer

Limitations of our GenServer

Limitations of our `GenServer`