Design of a Distributed Logging Service

We’ll design the distributed logging system now. Our logging system should log all activities or messages (we’ll not incorporate sampling ability into our design).

Requirements

Let’s list the requirements for designing a distributed logging system:

Functional requirements

The functional requirements of our system are as follows:

• Writing logs: The services of the distributed system must be able to write into the logging system.

• Searchable logs: It should be effortless for a system to find logs. Similarly, the application’s flow from end-to-end should also be effortless.

• Storing logging: The logs should reside in distributed storage for easy access.

• Centralized logging visualizer: The system should provide a unified view of globally separated services.

Non-functional requirements

The non-functional requirements of our system are as follows:

• Low latency: Logging is an I/O-intensive operation that is often much slower than CPU operations. We need to design the system so that logging is not on an application’s critical path.

• Scalability: We want our logging system to be scalable. It should be able to handle the increasing amounts of logs over time and a growing number of concurrent users.

• Availability: The logging system should be highly available to log the data.

Building blocks we will use

The design of a distributed logging system will utilize the following building blocks:

• Pub-sub system: We’ll use a pub-sub- system to handle the huge size of logs.
• Distributed search: We’ll use distributed search to query the logs efficiently.