The Hidden Danger of Retry Loops
Implementing retries for network calls is a best practice for building resilient microservices. However, misconfigured retries are a primary cause of cascading system outages.
The Timeout Trap
Imagine your API Gateway has a hard timeout of 5000ms. Inside your microservice, you configure an HTTP client to retry a failing downstream service up to 5 times, with an exponential backoff (base delay 500ms, multiplier 2). The delays alone will be 500ms + 1000ms + 2000ms + 4000ms = 7500ms.
By the 3rd retry, your API Gateway will have already timed out and returned a 504 to the user. Your microservice continues to hopelessly retry the downstream service, wasting CPU and network resources on a request that has already been abandoned by the client. This is known as Thread Starvation.
Why Jitter is Mandatory
If a downstream service goes down, 1,000 clients might simultaneously fail and immediately begin their retry loops. Without jitter (randomness), all 1,000 clients will sleep for exactly 500ms, wake up, and hit the recovering service at the exact same millisecond, knocking it back offline. This is called the Thundering Herd Problem.
Adding a 50% Jitter means each client randomly waits between 250ms and 500ms. This spreads the retries across a time window, allowing the downstream service to process the queued requests smoothly.