Requirements of a Distributed Search System's Design

Requirements

Let’s understand the functional and non-functional requirements of a distributed search system.

Functional requirements

The following is a functional requirement of a distributed search system:

• Search: Users should get relevant content based on their search queries.

Non-functional requirements

Here are the non-functional requirements of a distributed search system:

• Availability: The system should be highly available to the users.
• Scalability: The system should have the ability to scale with the increasing amount of data. In other words, it should be able to index a large amount of data.
• Fast search on big data: The user should get the results quickly, no matter how much content they are searching.
• Reduced cost: The overall cost of building a search system should be less.

Resource estimation

Let’s estimate the total number of servers, storage, and bandwidth that is required by the distributed search system. We’ll calculate these numbers using an example of a YouTube search.

Storage estimation

Each video’s metadata is stored in a separate JSON document. Each document is uniquely identified by the video ID. This metadata contains the title of the video, its description, the channel name, and a transcript. We assume the following numbers for estimating the storage required to index one video:

• The size of a single JSON document is 200 KB.
• The number of unique terms or keys extracted from a single JSON document is 1,000.
• The amount of storage space required to add one term into the index table is 100 Bytes.

The following formula is used to compute the storage required to index one video:

$$Tota{l}_{storage\mathrm{/}video}=Storag{e}_{\mathrm{/}doc}$$ ...