Rate Limiter [backup]

What is a rate limiter?

Rate limiter, as the name suggests, puts a limit on the number of requests a server fulfills. It throttles those requests that cross the predefined limit. For example, a client using a particular service’s API that is configured to allow 500 requests per minute would block further incoming requests for the client if the number of requests the client makes exceeds that limit.

Why do we need a rate-limiter?

A rate limiter is generally used as a defensive layer for services to avoid their excessive usage-whether intended or unintended. It also protects services against abusive behaviors targeting the application layer like Denial-of-service (DOS)DOS attacks and brute-force password attempts.

Following is a list of scenarios where rate limiters can be used to make the service more reliable.

• Preventing resource starvation: Some denial of service incidents are caused by errors in software or configurations in some part of a system which causes resource starvation. Such attacks are referred to as friendly-fire denial of service (DoS). One of the common use cases of rate limiters is to avoid resource starvation caused by such denial of service attacks, whether intentional or unintentional.

• Managing policies and quotas: There is also a need for rate limiters to provide fair and reasonable use of resources’ capacity when they are shared among many users. The policy refers to applying limits on the time duration or quantity allocated (quota).

• Controlling data flow: Rate limiters could also be used in systems where there is a need to process a large amount of data. Rate limiter controls the flow of data to distribute the work evenly among different machines avoiding the burden on a single machine.

• Avoiding excess costs: The rate-limiting can also be used to control the cost of operations. For example, organizations can use rate-limiting to prevent experiments from running out of control and avoid large bills. Some cloud service providers also use this concept by providing freemium services to certain limits, which can be increased on request by charging from users.

Goals and requirements of the system

Our focus in this lesson is to design a rate limiter with the following functional and non-functional requirements.

Functional requirements

• Limit the number of requests a client can send to an API within a time window.

• The limit of requests per window must be configurable.

• The client should get a message (error or notification) whenever the defined threshold is crossed within a single server or combination of servers.

Non-functional requirements

• Availability: Essentially, the rate limiter protects our system; therefore, it should be highly available.

• Low Latency: As all API requests pass through the rate limiter, it should work with a minimum latency without affecting the user experience.

• Scalability: Our design should be highly scalable. It should be able to rate-limit an increasing number of clients’ requests ...