Gain insights into data processing and concurrency in Elixir. Explore GenServer, GenStage, Flow, and Broadway to enhance performance, set up pipelines, and handle increased workloads efficiently.

Concurrent Data Processing in Elxir by The Pragmatic Programmer-01.png

myDokcer.tar.gz

ch1_1

ch_3

ch_3.1

ch1_1-task supervisor

ch_4.1

ch_5.1 RabbitMQ

Ch_5.2 RabbitMQ

Most engineers don’t think that concurrency is distinct from programming. Data processing includes transforming information in some way, like reporting, data aggregation, or analytics. On the other hand, concurrency involves significantly improving performance.

This course is designed to help you navigate the world of concurrency tools available in the Elixir ecosystem. You’ll explore popular modules and libraries like GenServer, GenStage, Flow, and Broadway. You’ll be able to ingest data, set up pipelines, and run concurrent processes with Elixir.

By the end of the course, you’ll have a wide range of new techniques to simplify your product, improve code performance, and make your application more resilient to increased workloads.

Concurrent Data Processing in Elixir

# The `DynamicSupervisor` supervisor

**`DynamicSupervisor`** is another ready-to-use supervisor available to us. It can
start any `GenServer` process on demand. Let’s add it to the supervision tree:


#file path -> jobber/lib/jobber/application.ex
#add this code at the indicated place mentioned in comments of jobber/lib/jobber/application.ex
#in the playground widget

def start(_type, _args) do
  children = [
    {DynamicSupervisor, strategy: :one_for_one, name: Jobber.JobRunner},
  ]

  opts = [strategy: :one_for_one, name: Jobber.Supervisor]
  Supervisor.start_link(children, opts)
end

The `strategy` setting is required, and the only currently accepted strategy is `:one_for_one`. We talk about supervisor strategies [later](https://www.educative.io/courses/concurrent-data-processing-elixir/supervisor-strategies) in this chapter, so we don’t need to worry about this for now.

# The module-based `DynamicSupervisor` supervisor

We can also define a `DynamicSupervisor` module as we see in the later lessons. Instead of `use Supervisor`, we need to use `DynamicSupervisor` and call `DynamicSupervisor.init/1` with the required `:strategy` value. We don’t have to provide a list of children processes.

# Create helper functions

It’ll be useful to create some helpers for testing. Create `.iex.exs` at the top project directory with the following content:

Learn to use the dynamic supervisors to start GenServer processes.

Getting Started

Easy Concurrency with the Task Module

Long-Running Processes Using GenServer

Data Processing Pipelines with GenStage

Process Collections with Flow

Data Ingestion Pipelines with Broadway

Concluding the Course

Introduce DynamicSupervisor

The `DynamicSupervisor` supervisor

The module-based `DynamicSupervisor` supervisor

Introduce DynamicSupervisor

The DynamicSupervisor supervisor

The module-based DynamicSupervisor supervisor

The `DynamicSupervisor` supervisor

The module-based `DynamicSupervisor` supervisor