Gain insights into data processing and concurrency in Elixir. Explore GenServer, GenStage, Flow, and Broadway to enhance performance, set up pipelines, and handle increased workloads efficiently.

Concurrent Data Processing in Elxir by The Pragmatic Programmer-01.png

myDokcer.tar.gz

ch1_1

ch_3

ch_3.1

ch1_1-task supervisor

ch_4.1

ch_5.1 RabbitMQ

Ch_5.2 RabbitMQ

Most engineers don’t think that concurrency is distinct from programming. Data processing includes transforming information in some way, like reporting, data aggregation, or analytics. On the other hand, concurrency involves significantly improving performance.

This course is designed to help you navigate the world of concurrency tools available in the Elixir ecosystem. You’ll explore popular modules and libraries like GenServer, GenStage, Flow, and Broadway. You’ll be able to ingest data, set up pipelines, and run concurrent processes with Elixir.

By the end of the course, you’ll have a wide range of new techniques to simplify your product, improve code performance, and make your application more resilient to increased workloads.

Concurrent Data Processing in Elixir

## Introduction
 
Operations like `Flow.reduce/3`, `group_by/2`, and take_sort/3 require all events to complete before they produce a result. After all, we cannot group or sort a list without all the data in the list being available to us. However, what if the data source keeps producing events for a long time or keeps producing events forever? This means we’ll have to wait for a result for either a very long time or indefinitely. To avoid this, `Flow` introduces windows and triggers. This section will discuss how windows and triggers can help us get progress updates with long-running flows.

## Windows

**Windows** group all events are going through `Flow`. We can think of windows as folders for events. They allow us to organize data conveniently. So far, we’ve used `Flow.Window.global/0`, which creates a global window for us by default. **Global windows** group all events together and close only when all events have been received. These are the other types of windows:

- The `Flow.Window.fixed/3` window groups events by a timestamp value on the event, using a specified time duration.

- The `Flow.Window.periodic/2` window is like `fixed/3` but groups events by processing time.

- The `Flow.Window.count/1` window groups events when they reach the given count.

Windows also influence how reducer operations work. At the beginning of a window, the accumulator is reset to collect data from scratch. Remember that `Flow.reduce/3` uses a function to create the initial accumulator value. This function is called at the beginning of each window.

## Triggers

We can also specify a trigger for each window. **Triggers** work like checkpoints, and all trigger functions take a window as their first argument. When a trigger <pagebreak/>event occurs, we have the opportunity to take data out of the flow by specifying what events to emit. We can even change the accumulator value on the fly. These are available triggers:

- `Flow.Window.trigger_every/2` triggers when we reach the given count of events.

- `Flow.Window.trigger_periodically/3` triggers at the specified time duration.

- `Flow.Window.trigger/3` is used to implement custom triggers.

By default, every flow has an internal trigger for an event called `:done`, which occurs when no data is left to process.


## Use different combinations of windows and triggers

Depending on the type of window and trigger we choose, we can get different insights into the flow. In a moment, we'll see how to use `trigger_every/2` with the default global window. We won't cover all possible combinations in this course, but the good news is that they are well documented online, and we can use them. After completing this chapter, we encourage you to check out the official documentation for the [`Flow.Window`](https://hexdocs.pm/flow/Flow.Window.html) module.

We've covered the basics of windows and triggers, but one important thing is left—detecting when a trigger occurs. Luckily, there is a function that makes this very easy.

## Catch trigger events with `on_trigger/2`



# Introduction
 
Operations like `Flow.reduce/3`, `group_by/2`, and take_sort/3 require all events to complete before they produce a result. After all, we cannot group or sort a list without all the data in the list being available to us. However, what if the data source keeps producing events for a long time or keeps producing events forever? This means we’ll have to wait for a result for either a very long time or indefinitely. To avoid this, `Flow` introduces windows and triggers. This section will discuss how windows and triggers can help us get progress updates with long-running flows.

# Windows

**Windows** group all events are going through `Flow`. We can think of windows as folders for events. They allow us to organize data conveniently. So far, we’ve used `Flow.Window.global/0`, which creates a global window for us by default. **Global windows** group all events together and close only when all events have been received. These are the other types of windows:

- The `Flow.Window.fixed/3` window groups events by a timestamp value on the event, using a specified time duration.

- The `Flow.Window.periodic/2` window is like `fixed/3` but groups events by processing time.

- The `Flow.Window.count/1` window groups events when they reach the given count.

Windows also influence how reducer operations work. At the beginning of a window, the accumulator is reset to collect data from scratch. Remember that `Flow.reduce/3` uses a function to create the initial accumulator value. This function is called at the beginning of each window.

# Triggers

We can also specify a trigger for each window. **Triggers** work like checkpoints, and all trigger functions take a window as their first argument. When a trigger <pagebreak/>event occurs, we have the opportunity to take data out of the flow by specifying what events to emit. We can even change the accumulator value on the fly. These are available triggers:

- `Flow.Window.trigger_every/2` triggers when we reach the given count of events.

- `Flow.Window.trigger_periodically/3` triggers at the specified time duration.

- `Flow.Window.trigger/3` is used to implement custom triggers.

By default, every flow has an internal trigger for an event called `:done`, which occurs when no data is left to process.


# Use different combinations of windows and triggers

Depending on the type of window and trigger we choose, we can get different insights into the flow. In a moment, we'll see how to use `trigger_every/2` with the default global window. We won't cover all possible combinations in this course, but the good news is that they are well documented online, and we can use them. After completing this chapter, we encourage you to check out the official documentation for the [`Flow.Window`](https://hexdocs.pm/flow/Flow.Window.html) module.

We've covered the basics of windows and triggers, but one important thing is left—detecting when a trigger occurs. Luckily, there is a function that makes this very easy.

# Catch trigger events with `on_trigger/2`



Learn how windows and triggers are used to group events in a flow and take data out of a flow, respectively.

Getting Started

Easy Concurrency with the Task Module

Long-Running Processes Using GenServer

Create a GenServer and Supervisor From Scratch

Data Processing Pipelines with GenStage

Process Collections with Flow

Data Ingestion Pipelines with Broadway

Concluding the Course

Windows and Triggers

Introduction

Windows