Pipelining Data Streams
Learn how to combine Elixir's pipe operator and streams.
We'll cover the following...
Introduction to pipelining data streams
In this lesson, we’ll combine Elixir pipe operators and streams to create a pipeline of tasks that consumes a data stream. We can do it in two ways: eager or lazy.
Each computation will process all the items with the eager strategy before sending them to the next computation. With the lazy approach, each computation can process a few elements and send them to the next computation. The effect is that the eager strategy will output a result only after all the items have been processed. The lazy strategy will start to produce a result after a small number of elements have been processed. Until now, we have used a lot of the eager approach, and it has worked very well.
This lesson will explore the benefits of the lazy strategy using lazy collections and Stream higher-order functions.
Understanding eager and lazy evaluation
To better understand how eager and lazy evaluation work, think of a mechanized assembly line: it has a tray we can put items on, the assembly line will start to run, and the items will be processed sequentially by the machines connected to the tray.
In the eager approach, all the elements on the tray must be processed by a machine before it lets them go to the next step. So, if we have three items on the tray at the beginning, all of them must be processed by the ...