Introduction to Sharded Counters [backup]

Problem statement

Real-time applications like Facebook, Twitter, and YouTube have high user traffic. Users interact with the applications and perform multiple operations (view, like, comment, etc.) depending on the application’s structure. For instance, an image is posted on a Facebook page that has millions of followers, and the post likes rapidly increase after each millisecond. Here, counting the likes might be easy for this single image, but what will we do when thousands of such images or videos are uploaded simultaneously by many celebrities, each with millions of followers. This problem is known as the heavy hitter problem. The above scenario shows how a simple counting operation becomes challenging to manage with precision and performance. The following figure shows which of YouTube’s videos were viewed by millions of users in a 24 hours span as of August 2021.

On average, Six thousand tweets are sent on Twitter within one second, which equals 360 thousand tweets per minute and about 500 million tweets per day. A challenging task is to handle billions of likes on these 500 million tweets per day. The following table shows the most liked tweets in one day as of 2022.

How will we handle millions of write requests coming against the likes on thousands of tweets per minute? The challenge is that writing takes more time than reading and concurrent activity makes this problem harder. As the number of concurrent writes increases for some counter (which might be a variable residing in a node’s memory), the lock contention increases non-linearly. After some point, we might be spending most of the time acquiring the lock so ...