Concurrent Data Processing in Elixir/

...

/

Use GenStage Producers

Use GenStage Producers

Learn to use GenStage for processing data pipelines with dynamic batching.

We'll cover the following...

GenStage producers

The Broadway library is an excellent choice when working with popular message brokers. However, it’s not limited to just that. Broadway could be useful in a wide range of use cases where we need a data-processing pipeline with built-in dynamic batching. We only have to bring our own GenStage producer and Broadway will happily do the work for us. We’ll see how this works by revisiting the scraper project one last time. We already used GenStage and Flow to implement it, so it will be great to implement another version using Broadway, for comparison.

Replace :flow with :broadway

Let’s start by replacing :flow with :broadway in the dependencies’ list:

Press + to interact
# file path -> scraper/mix.exs
defp deps do
  [
    {:gen_stage, "~> 1.0"},
    {:broadway, "~> 0.6"}
  ]
end

We use the built-in processors in Broadway to do the heavy lifting, leverage concurrency, and provide fault tolerance. This means that we no longer need OnlinePageProducerConsumer, PageConsumerSupervisor, and PageConsumer. We can delete their files later, so they don’t get in the way. The brand new ScrapingPipeline ...