Adding Concurrency and Channels

Concurrency

A scalable data pipeline must handle multiple items simultaneously; it must be concurrent. GenStage has a solution for adding concurrency to our pipeline with the ConsumerSupervisor module. This module allows us to focus on defining the pipeline and letting the library take care of how the concurrency will be managed.

ConsumerSupervisor is a type of GenStage consumer that spawns a child process for each item received. The concurrency is controlled via setup options, and it behaves exactly like a consumer. Every item spawns a new process; they’re not re-used, but this is cheap to do in Elixir.

Note: We make our system concurrent by creating processes that work. The BEAM then makes that system parallel by taking the concurrent work and running it over multiple cores simultaneously. How parallel execution happens is entirely handled by the BEAM.

Our final result in this section will look like this: