Evaluation of Web Crawler's Design
Evaluate the proposed design of the web crawler system based on the fulfillment of its non-functional requirements.
We'll cover the following...
Reviewing design requirements
Let’s evaluate how our design meets the non-functional requirements of the proposed system.
Scalability
Our design states that scaling our system horizontally is vital. Therefore, the proposed design incorporates the following design choices to meet the scalability requirements:
-
The system is scalable to handle the ever-increasing number of URLs. It includes all the required resources, including schedulers, web crawler workers, HTML fetchers, extractors, and blob stores, which are added/removed on demand.
-
In the case of a distributed URL frontier, the system utilizes consistent hashing to distribute the hostnames among various crawling workers, where each worker is running on a server. With this, adding or removing a crawler server isn’t a problem.
Extensibility and modularity
So far, our design is only focusing on a ...