Correlation and Causation

Correlation identifiers have been critical to monolithic application logging for as long as we can remember. This involves generating a unique identifier, which is then passed from function to function and included in any log outputs. That way, we can query lots of log entries using a single common identifier to correlate them together. This forms a trace within the logs so that we can observe a connected pattern of messaging from within the application code.

As we shift to microservices, this challenge goes beyond a single application process. Most commonly, correlation identifiers are propagated from one service to the next using HTTP for correlation.

When we move to event-driven architecture, we introduce a few new challenges to this correlation pattern.

Correlation identifiers need to persist across events by being included in the event message from the producer. Then, any consumer can reuse the correlation identifier in its own log output.

While that solves the cross-service trace challenge, we should remember that an event could have any number of subscribers, and they’ll all persist with the same correlation identifier. This means that when we query by that identifier, the resulting logs might become unwieldy, lacking enough granular trace threads to be efficiently interrogated.

To address the unwieldy logs, we can introduce causation identifiers. A causation identifier is the unique identifier of the preceding event that caused a method to fire. It very simply chains together the preceding event with the current event, so we know what caused what.

Application Insights supports this pattern, which is known as distributed tracing. It uses slightly different names for the identifiers:

• Instead of Log Item Id, it uses Id.

• Instead of Correlation Id, it uses operation_Id.

• Instead of Causation Id, it uses operation_ParentId.

The following is an example of these identifiers in action. Multiple consumers have been added to illustrate how the correlation/causation pattern works: