Introduction to Newsfeed System [backup]

What is a newsfeed?

A newsfeed of any social media platform—Twitter, Facebook, Instagram— is a list of stories generated by entitiesAn entity could be a page, group, friends, and followers of a user. that a user follows. It contains text, images, videos, and other activities such as likes, comments, shares, advertisements, etc. This list is continuously updated and presented to the relevant users on the user’s home page. Similarly, a newsfeed system also displays the newsfeed to users from friends, followers, groups, and other pages, including a user’s own posts.

A newsfeed is essential for social media platforms’ users because it keeps them informed about the latest industry developments, current affairs, and relevant information. It also provides them additional reasons to return to, visit, and connect with a platform on a regular basis. Billions of users use such platforms; the challenging task is how to provide a personalized newsfeed in real-time while keeping the system scalable and highly available.

This chapter will discuss the high-level and detailed design of a newsfeed system (for a social platform like Facebook, Twitter, Instagram, etc.). To limit the scope of the problem, we will focus on the following functional and non-functional requirements.

To limit the scope of the problem, we will focus on the following functional and non-functional requirements.

Requirements

We aim to design a newsfeed system having the following functional and non-functional requirements.

Functional requirements

• Newsfeed generation: The system will generate newsfeeds based on pages, groups, and followers that a user follows. A user may have many friends and followers; therefore, the system should be capable of generating feeds from all friends and followers. The challenge here is that potentially there is huge content out of which our system needs to decide which content to pick (that might be relevant/interesting) for the user and to rank it further to decide which one to show first.

• Newsfeed contents: The newsfeed may contain text, images, and videos.

• Newsfeed display: The system should affix new incoming posts to the newsfeed for all active users based on some ranking mechanism. Once ranked, we show content to a user with higher-ranked first.

Non-functional requirements

• Scalability: Our proposed system should be highly scalable to support the ever-increasing number of users on any platform (Twitter, Facebook, Instagram).

• Fault tolerance: Since the system should be handling a large amount of data; therefore, partition tolerance (system availability in the events of network failure between the system’s components) is necessary.

• Availability: The service should be highly available to keep the users engaged with the platform. The system can compromise strong consistency for availability and fault tolerance, according to the PACELC theoremThe PACELC theorem is an extension of the CAP theorem that states, in the event of network Partition, one should choose between Availability or Consistency; else, choose between Latency and Consistency..

The system can compromise strong consistency for availability and fault tolerance, as according to the

• Low latency: The system should provide newsfeeds in real-time hence the maximum latency should not be greater than 2 seconds.

Capacity estimation

Assume the platform for which the newsfeed system is designed has 1 Billion users per day, out of which, on average, 500 million are daily active users. Also, assume that each user has 300 friends and follows 250 pages on average. Based on the assumed statistics, let’s look at the traffic, storage, and servers estimation.

Traffic estimation

Assume that each daily active user opens the ...