Design of a Distributed Logging Service
Design a distributed logging service that meets strict requirements for scalability, availability, and low latency. Learn to use asynchronous log accumulators and a pub-sub architecture to handle massive log volumes without affecting application performance.
We will design a distributed logging system to capture all activities and messages. We will not incorporate sampling in this design.
Requirements
Let’s list the requirements for designing a distributed logging system:
Functional requirements
Writing logs: Services within the distributed system must be able to write to the logging system.
Searchable logs: Users must be able to search logs effortlessly and track the application flow end to end.
Storing logs: Logs must reside in distributed storage for easy access.
Centralized logging visualizer: The system must provide a unified view of globally distributed services.
Non-functional requirements
Low latency: Logging is an I/O-intensive operation. It must not block the application’s critical path.
Scalability: The system must handle increasing log volumes and a growing number of concurrent users.
Availability: The logging system must be highly available. ... ...