Measuring our Pipeline

Pipeline

The ultimate question of running a software is, “How do I know if it’s working?” Our data pipeline is no different. We need to be able to answer questions about the health of our pipeline so that we can fix any problems that occur. We’ll achieve this by adding measurements for how long our Worker takes to process and how long it takes to broadcast our message.

We can use our stats knowledge to capture a timing event for our Worker process. We’ll trigger a manual timing event to measure the time between item generation and push delivery. Let’s jump right into our Worker process.

Replace our current Worker.start_link function with this new timed one: