Building Real-time Applications with Phoenix & Elixir/

...

/

Measuring our Pipeline

Measuring our Pipeline

Learn how to measure and monitor our pipelines.

We'll cover the following...

Pipeline

The ultimate question of running a software is, “How do I know if it’s working?” Our data pipeline is no different. We need to be able to answer questions about the health of our pipeline so that we can fix any problems that occur. We’ll achieve this by adding measurements for how long our Worker takes to process and how long it takes to broadcast our message.

We can use our stats knowledge to capture a timing event for our Worker process. We’ll trigger a manual timing event to measure the time between item generation and push delivery. Let’s jump right into our Worker process.

Replace our current Worker.start_link function with this new timed one:

Press + to interact
# hello_sockets/lib/hello_sockets/pipeline/worker.ex
  def start_link(item) do
    Task.start_link(fn ->
      HelloSockets.Statix.measure("pipeline.worker.process_time", fn ->
        process(item)
      end)
    end)
  end

That wasn’t very exciting; all we did was add a measure around our existing code! This is the beauty of taking measurements: it doesn’t have to be complicated. This simple two-line addition allows us to keep track of a critical part of our pipeline health. It does feel a bit low-value to capture this single-function timing. However, it quickly becomes essential as we add more logic or servers.

Adding a measurement of our total delivery time ...