Gain insights into data processing and concurrency in Elixir. Explore GenServer, GenStage, Flow, and Broadway to enhance performance, set up pipelines, and handle increased workloads efficiently.

Concurrent Data Processing in Elxir by The Pragmatic Programmer-01.png

myDokcer.tar.gz

ch1_1

ch_3

ch_3.1

ch1_1-task supervisor

ch_4.1

ch_5.1 RabbitMQ

Ch_5.2 RabbitMQ

Most engineers don’t think that concurrency is distinct from programming. Data processing includes transforming information in some way, like reporting, data aggregation, or analytics. On the other hand, concurrency involves significantly improving performance.

This course is designed to help you navigate the world of concurrency tools available in the Elixir ecosystem. You’ll explore popular modules and libraries like GenServer, GenStage, Flow, and Broadway. You’ll be able to ingest data, set up pipelines, and run concurrent processes with Elixir.

By the end of the course, you’ll have a wide range of new techniques to simplify your product, improve code performance, and make your application more resilient to increased workloads.

Concurrent Data Processing in Elixir

# The `handle_failed/3` callback function

Right now, when we return a message from `handle_message/3`, `Broadway` automatically acknowledges it since it reaches the end of the pipeline.

However, if an unhandled exception happens in our business logic, the message is marked as failed. We can also manually mark a message as failed using `Broadway.Message.failed/2`. This could be useful when we want to discard messages we don't want to process.

Remember that in `@producer_config` we configure the `:on_failure` setting with the `:reject_and_requeue` value. This means that failed messages will be sent back to RabbitMQ and redelivered. By default, messages will be redelivered indefinitely. We can change this setting to `:reject_and_requeue_once` or simply `:reject` to completely discard all failed messages.

We can also configure acknowledgments on a per-message basis, and the ideal place for that is `handle_failed/3`. The optional `handle_failed/3` callback allows us to check which messages have failed and why. We can then decide if we want to take further action.


# Update `tickets_available?`

Let's update the `tickets_available?/1` function and pattern match for cinema events. We return `false` to all of them for being unavailable:



Learn to handle failed messages through the handle_message callback function.

Getting Started

Easy Concurrency with the Task Module

Long-Running Processes Using GenServer

Data Processing Pipelines with GenStage

Process Collections with Flow

Data Ingestion Pipelines with Broadway

Concluding the Course

Implement handle_failed/3

The `handle_failed/3` callback function

Implement handle_failed/3

The handle_failed/3 callback function

The `handle_failed/3` callback function