Debugging Distributed Systems: A War Story

A production incident that looked like a database problem turned out to be a retry storm triggered by a misconfigured timeout three services upstream, each one adding its own retry on top of the last. Nothing was individually wrong. The composition was the disaster.

When the pagers go off for a database CPU spike and connection exhaustion, the immediate instinct is to look at the data layer. The DBA team starts hunting for a missing index, a bad query plan, or a sudden surge in external traffic. During this specific incident, the database metrics were indeed catastrophic. But the database was entirely innocent. It was just the victim of our own microservices.

Here is how a seemingly resilient architecture accidentally executes a massive denial of service attack against itself.

The Anatomy of a Retry Storm In a deeply decoupled architecture, a single user action often triggers a synchronous chain of API calls. Service A calls Service B, which calls Service C, which finally queries the database.

During the incident, the database experienced a minor, transient latency bump. It was nothing critical. But Service C had a tightly configured timeout. It gave up waiting for the database, dropped the connection, and automatically retried the query.

Because Service C took longer than usual to return an error, Service B hit its own timeout limit. Following standard engineering best practices, Service B retried its request to Service C. Service A experienced the same delay, timed out, and retried its call to Service B.

Every single service in the chain was configured to retry three times on failure. Three retries compounded across three network hops resulted in twenty seven identical database queries for every single original user request. A tiny latency bump was exponentially amplified into a traffic tsunami that immediately crushed the primary database cluster.

The Technical Fix The immediate mitigation was shedding load, but the architectural fix required stripping autonomy away from individual microservices.

You cannot let every backend team own their own retry logic. We centralized the retry and timeout policies. Instead of developers writing custom for loops in their application code, we pushed all network resiliency rules into a shared middleware layer. We enforced a strict policy where retries are only permitted at the absolute edge of the architecture. If an internal service fails, it must fail fast and propagate the error upward. We also implemented aggressive circuit breakers to ensure that when a downstream dependency is struggling, we stop sending it traffic entirely until it recovers.

The Organizational Blind Spot The technical fix was straightforward, but the post mortem revealed a much harder organizational lesson.

When we reviewed the codebase, no single engineer had made a mistake. The team owning Service B wrote perfect code for their specific domain. The team owning Service C did the same. They all implemented retries because that is what you are taught to do to handle network blips.

In a distributed system, the most dangerous bugs do not live inside the application code. They live in the gaps between services. When architecture is divided into silos, teams optimize for their local component. If nobody is responsible for looking at the end to end call chain, perfectly written microservices will eventually combine to take down production.