Gain insights into data processing and concurrency in Elixir. Explore GenServer, GenStage, Flow, and Broadway to enhance performance, set up pipelines, and handle increased workloads efficiently.

Concurrent Data Processing in Elxir by The Pragmatic Programmer-01.png

myDokcer.tar.gz

ch1_1

ch_3

ch_3.1

ch1_1-task supervisor

ch_4.1

ch_5.1 RabbitMQ

Ch_5.2 RabbitMQ

Most engineers don’t think that concurrency is distinct from programming. Data processing includes transforming information in some way, like reporting, data aggregation, or analytics. On the other hand, concurrency involves significantly improving performance.

This course is designed to help you navigate the world of concurrency tools available in the Elixir ecosystem. You’ll explore popular modules and libraries like GenServer, GenStage, Flow, and Broadway. You’ll be able to ingest data, set up pipelines, and run concurrent processes with Elixir.

By the end of the course, you’ll have a wide range of new techniques to simplify your product, improve code performance, and make your application more resilient to increased workloads.

Concurrent Data Processing in Elixir

# The `handle_continue` callback function

The `handle_continue/2` callback is a recent addition to `GenServer`. Often `GenServer` processes do complex work as soon as they start. Rather than blocking the whole application from starting, we return `{:ok, state, {:continue, term}}` from the `init/1` callback and use `handle_continue/2`.

## Return values 

Accepted return values for `handle_continue/2` include the following:

* `{:noreply, new_state}`

* `{:noreply, new_state, {:continue, term}}`
 
* `{:stop, reason, new_state}`

## `{:noreply, new_state}`

Since the `handle_continue/2` callback receives the latest state, we can use it to update the `GenServer` with new information by returning `{:noreply, new_state}`. For example, we can write something like this:

```Elixir
def handle_continue(:fetch_from_database, state) do 
    # get `users` from the database
    {:noreply, Map.put(state, :users, users)}
end
```
## `{:noreply, new_state, {:continue, term}}` and `{:stop, reason, new_state}`

The other return values are similar to the ones we already covered for `init/1`, but it’s interesting to note that `handle_continue/2` can also return `{:continue, term}`, which triggers another `handle_continue/2`. We can use this to break down work into several steps when needed. Although `handle_continue/2` is often used in conjunction with `init/1`, other callbacks can also return `{:continue, term}`.

# Send process messages

One of the highlights of `GenServer` processes is that we can interact with them while they run. This is done by sending messages to the process. If we want to get some information back from the process, we use **`GenServer.call/3`**. When we don’t need a result back, we use **`GenServer.cast/2`**. Both functions accept the process identifier as their first argument. The second argument is the message to send to the process. Messages could be any Elixir term.

When the `cast/2` and `call/3` functions are used, the `handle_cast/2` and `handle_call/3` callbacks are invoked, respectively. Let’s see how they work in practice.

# The `handle_call` callback function

Let’s implement an interface to get the current process state and start sending emails. We add the following code to `send_server.ex`:

Learn how the GenServer uses handle_continue and handle_call to divide work and interact with processes.

Getting Started

Easy Concurrency with the Task Module

Long-Running Processes Using GenServer

Data Processing Pipelines with GenStage

Process Collections with Flow

Data Ingestion Pipelines with Broadway

Concluding the Course

GenServer Callbacks: handle_continue and handle_call

The `handle_continue` callback function

Return values

GenServer Callbacks: handle_continue and handle_call

The handle_continue callback function

Return values

The `handle_continue` callback function