Batch Messages
Explore how batching enhances Broadway pipelines by grouping messages to improve efficiency and concurrency. Learn to configure batching, send test messages using the amqp package, and streamline operations like bulk database inserts. This lesson helps you optimize data ingestion and processing workflows in Elixir.
We'll cover the following...
Improve efficiency using batching
Using processors is great for many tasks, but sometimes we need another layer of stages. Our business logic in handle_message/3 could become a bottleneck and slow down the pipeline, especially under heavy workloads. We may also want to improve efficiency by grouping and running some operations together. This is where batching comes in to help.
Using batching in Broadway is an easy way to add another step in our data-ingestion pipeline for further processing. It allows us to leverage concurrency and group relevant messages to perform operations in bulk. This section configures batching for ...