System Design: Web crawler

Introduction

A web crawler is an Internet bot that systematically scoursTo go or move swiftly about, over, or through in search of something the World Wide Web for content, starting its operation from a pool of seed URLsStored URLs serving the purpose of a starting point to a crawler. This process of acquiring content from the WWW is called crawling. It further saves the crawled content in the data stores. This process of efficiently saving data for subsequent use is called storing.

It’s the first step performed by search engines; the stored data is used for indexing and ...