Add More Consumers and Buffer Events
Learn how adding more consumers and buffering events will affect the output of our scrape_pages function.
Add more consumers
Rather than working in batches within a single process, we can easily scale our data processing pipeline by running more than one consumer, each responsible for scraping one page at a time. To do this, let’s adjust the demand of PageConsumer:
Get hands-on with 1400+ tech skills courses.