Understanding the Token Bucket Algorithm
When configuring rate limits on an API Gateway, NGINX, or inside application code (using Redis), you'll often encounter the Token Bucket algorithm. It is the industry standard because it allows for short bursts of traffic while enforcing a strict long-term average limit.
The Two Core Parameters
- Refill Rate (Sustained Rate): The number of tokens added to the bucket every second. This dictates the maximum average Requests Per Second (RPS) the client is allowed over a long period.
- Bucket Capacity (Burst Limit): The absolute maximum number of tokens the bucket can hold. This dictates how many requests the client can make instantaneously if they haven't made any requests recently.
Why Capacity Matters
If you set your capacity identical to your refill rate (e.g., Capacity = 100, Refill = 100/s), you are effectively enforcing a strict "1 request every 10 milliseconds" limit. If a web browser attempts to load an HTML page and concurrently fires 10 API requests to fetch data, 9 of them might be immediately rejected with a 429 Too Many Requests because they arrived at the same millisecond, even though the total traffic for the second is well under 100.
By increasing the Bucket Capacity, you allow the user to "save up" their unused tokens to handle concurrent, bursty web traffic elegantly.