Adjust Batch Size and Timeout
Explore how to adjust the batch size and batch timeout configurations in Broadway pipelines to improve data ingestion performance. This lesson helps you understand controlling message flow and balancing concurrency by tweaking settings like :batch_size and :batch_timeout for efficient pipeline processing.
Use batch size
If we inspect batch_info.size in the handle_batch/4 callbacks, we notice that our batches contain no more than fifty messages. The maximum number of messages received is also known as batch size and is typically controlled by the :batch_size value for each batcher in start_link/1. We can see this here:
...
batchers: [
...
default: [
batch_size: 100
]
]
However, the :batch_size default value for each batcher is already 100. It may seem like something is going wrong since we only get half the expected amount.
This is actually by design and is specific to the broadway_rabbitmq producer. The BroadwayRabbitMQ.Producer module ...