Gain insights into data processing and concurrency in Elixir. Explore GenServer, GenStage, Flow, and Broadway to enhance performance, set up pipelines, and handle increased workloads efficiently.

Concurrent Data Processing in Elxir by The Pragmatic Programmer-01.png

myDokcer.tar.gz

ch1_1

ch_3

ch_3.1

ch1_1-task supervisor

ch_4.1

ch_5.1 RabbitMQ

Ch_5.2 RabbitMQ

Most engineers don’t think that concurrency is distinct from programming. Data processing includes transforming information in some way, like reporting, data aggregation, or analytics. On the other hand, concurrency involves significantly improving performance.

This course is designed to help you navigate the world of concurrency tools available in the Elixir ecosystem. You’ll explore popular modules and libraries like GenServer, GenStage, Flow, and Broadway. You’ll be able to ingest data, set up pipelines, and run concurrent processes with Elixir.

By the end of the course, you’ll have a wide range of new techniques to simplify your product, improve code performance, and make your application more resilient to increased workloads.

Concurrent Data Processing in Elixir

# The `terminate` callback function

The **`terminate/2`** callback is usually invoked before the process exits, but only when the process itself is responsible for the exit. Most often, the exit results from explicitly returning `{:stop, reason, state}` from a callback (excluding `init/1`). It can also happen when an unhandled exception happens within the process.

# Caution when relying on `terminate/2`

There are cases when a `GenServer` process is forced to exit due to an external event—for example, when the whole application shuts down. In such cases, `terminate/2` is not invoked by default.

This is something we must keep in mind if we want to ensure that important business logic always runs before the process exits. For example, we may want to persist in-memory data (stored in the process’s state) to the database before the process dies. This is out of the scope of this course. However, if we want to ensure that `terminate/2` is always called, we can look into setting `Process.flag(:trap_exit, true)` on the process, or use `Process.monitor/1` to perform the required work in a separate process.

# Print a message before the process exits

Let’s implement `terminate/2` by printing a message before the process exits:


Learn how the GenServer ends running processes using the terminate callback.

Getting Started

Easy Concurrency with the Task Module

Long-Running Processes Using GenServer

Data Processing Pipelines with GenStage

Process Collections with Flow

Data Ingestion Pipelines with Broadway

Concluding the Course

GenServer Callbacks: terminate

The `terminate` callback function

GenServer Callbacks: terminate

The terminate callback function

The `terminate` callback function