We’ll design the distributed logging system now. Our logging system should log all activities or messages (we’ll not incorporate sampling ability into our design).

Requirements

Let’s list the requirements for designing a distributed logging system:

Functional requirements

The functional requirements of our system are as follows:

  • Writing logs: The services of the distributed system must be able to write into the logging system.

  • Searchable logs: It should be effortless for a system to find logs. Similarly, the application’s flow from end-to-end should also be effortless.

  • Storing logging: The logs should reside in distributed storage for easy access.

  • Centralized logging visualizer: The system should provide a unified view of globally separated services.

Non-functional requirements

The non-functional requirements of our system are as follows:

  • Low latency: Logging is an I/O-intensive operation that is often much slower than CPU operations. We need to design the system so that logging is not on an application’s critical path.

  • Scalability: We want our logging system to be scalable. It should be able to handle the increasing amounts of logs over time and a growing number of concurrent users.

  • Availability: The logging system should be highly available to log the data.

Building blocks we will use

The design of a distributed logging system will utilize the following building blocks:

  • Pub-sub system: We’ll use a pub-sub- system to handle the huge size of logs.
  • Distributed search: We’ll use distributed search to query the logs efficiently.

Level up your interview prep. Join Educative to access 70+ hands-on prep courses.