Timeouts, deadlines, retries with backoff and jitter, circuit breakers, bulkheads and load shedding to contain partial failures and measure them.
One Friday, the recommendations service starts taking thirty seconds instead of eighty milliseconds. It is not down: it answers, just late. The checkout page calls it to show "you may also like", its threads sit waiting, the connection pool runs dry and, a few minutes later, nobody can pay. To make it worse, every layer retries whatever fails and multiplies the load on a service that was already struggling. A secondary detail took down the main feature.
Resilience is a system's ability to keep delivering what matters most, possibly degraded, when a part fails or slows down, and to recover without heroics. It is not about avoiding failure but about containing it: deciding how long you wait, how many times you insist, when you stop calling, what you isolate and what you give up.
This article walks through those mechanisms in the order they are usually applied (timeouts and deadlines, retries, circuit breakers, bulkheads, load shedding and degradation), how to test them with chaos engineering, and how to measure them. It leaves two neighbouring topics out: consistency theory lives in CAP, PACELC and consistency, and transaction compensation in sagas, outbox and idempotency.
Library versions are as of October 2026 (GitHub and NuGet). Amazon Builders' Library articles now redirect to builder.aws.com. I did not read the book Release It! directly: what is said about it comes from Martin Fowler's article and from reviews, and is flagged as such.
Inside one process, a function call either returns or throws. In a distributed system there is a third outcome: you do not know what happened. The request may have been lost, executed without the response arriving, or still be running. And the system is rarely all down or all healthy; something is almost always degraded.
The fallacies of distributed computing sum up the assumptions that make you forget this. According to Arnon Rotem-Gal-Oz's paper and the Wikipedia entry, L Peter Deutsch, then at Sun Microsystems, collected seven of them in 1994, building on work by other Sun colleagues, and James Gosling added the eighth around 1997. Each one has a design consequence:
A dead dependency fails fast: the connection is refused and the caller moves on. A slow dependency holds threads, connections and memory while you wait. The cascading failures chapter of Google's SRE book describes this as bimodal latency: a small fraction of requests hanging until a long deadline can tie up most threads and starve everything else. That is why nearly every mechanism in this article aims at the same thing: turn slowness into a fast, bounded failure.
The timeout is the first line of defence and the most often forgotten. Marc Brooker, in his Builders' Library article on timeouts, retries and backoff, recommends setting one on every remote call, even between processes on the same machine, and separating the connection timeout from the request timeout. Do not assume your client ships with a sensible one: the gRPC documentation, for example, warns that by default it sets no deadline at all.
To pick the value, the same article suggests starting from an acceptable false-timeout rate (say 0.1%) and using the matching percentile of the dependency's latency (p99.9 in that case). It adds two caveats:
A per-hop timeout is not enough. If the edge gives the user 2 s and every intermediate service sets 2 s on its own call, work keeps running downstream long after the user got an error. The alternative is a deadline: set once, at the top, with each hop passing on only the time that remains.
sequenceDiagram participant U as User participant E as Edge participant C as Checkout participant P as Pricing U->>E: GET /checkout Note over E: total deadline 2000 ms E->>C: call (1950 ms left) Note over C: 300 ms of own work C->>P: call (1650 ms left) P-->>C: response C-->>E: response E-->>U: 200
The SRE book recommends exactly that: set the deadline high in the stack, subtract the time spent at each hop, and check the remaining time before each stage so you do not do work the client has already abandoned. It adds cancellation propagation: if the request can no longer succeed, cancel the related work across the whole chain instead of letting it run out its deadline. gRPC sends the remaining time as a duration rather than an absolute timestamp, which avoids depending on synchronized clocks; its documentation notes that automatic propagation varies by language.
The shape of the contract is enough to guide the design: the deadline and the cancellation signal travel with the call.
// Indicative shape: the call context carries the deadline and cancellation type CallContext = { readonly deadline: number; // cut-off instant in ms, from the process clock readonly signal: AbortSignal; // fires if the caller gives up }; export declare function remainingMs(ctx: CallContext): number; export declare function getQuote(ctx: CallContext, sku: string): Promise<Quote>;
Retrying fixes transient failures: a dropped packet, a restarting replica, a brief spike. Brooker calls retries selfish: the client spends more of the server's time to improve its own odds. With rare failures it is a good deal; under overload, it makes things worse. Three conditions before you retry:
429
503
Retry-After
Retrying immediately piles load on at the worst moment. Exponential backoff spaces attempts out (say 50 ms, 100 ms, 200 ms…) up to a cap. But if a thousand clients fail together and all wait exactly the same, they come back together. Jitter adds randomness to the wait to break that synchronization.
Brooker compared variants in an AWS Architecture Blog post from March 2015. Equal jitter (keep half the wait and randomize the other half) did worst among the randomized options; full jitter (wait a random value between zero and the exponential cap) and decorrelated jitter came out close, with no clear winner. His conclusion is that jittered backoff should be the standard approach for remote clients.
The real danger shows up when several layers retry. If each of five layers makes up to three attempts, the database at the bottom can see 3⁵ = 243 times the original load: that is Brooker's example. The SRE book shows the same effect with different numbers. Both sources agree on the remedy: retry at a single point in the stack. Brooker suggests a single layer for cheap operations; the SRE book, the layer immediately above the one rejecting the request.
Capping attempts per request is not enough: under high concurrency, three attempts per request is still triple the load. A retry budget caps retries in proportion to traffic:
The whole policy fits in a per-dependency configuration; this is its shape, not recommended values:
# Indicative shape of a per-dependency policy pricing: connectTimeout: 100ms requestTimeout: 300ms # observed p99.9 plus headroom retry: maxAttempts: 3 # includes the first attempt backoff: { base: 50ms, max: 1s, jitter: full } retryOn: [connect-failure, '503', '429'] budgetRatio: 0.1 # retries / requests in the window
Retries insist; a circuit breaker does the opposite. When a dependency fails persistently, it stops calling it for a while and returns an error immediately. Martin Fowler described it on his bliki on 6 March 2014 and credits Michael Nygard and his book Release It! (Pragmatic Bookshelf, first edition 2007, second edition 2018) with popularizing it.
stateDiagram-v2 [*] --> Closed Closed --> Open: failures exceed the threshold Open --> HalfOpen: reset timeout expires HalfOpen --> Closed: trial call succeeds HalfOpen --> Open: trial call fails
What Fowler's article leaves to you as design decisions:
One breaker per dependency, or per instance when the failure affects a single replica, keeps one sick component from cutting traffic to the healthy ones.
Brooker points out that circuit breakers introduce modal behaviour (the system acts differently depending on the state) that is hard to test and can slow recovery, and he prefers the token bucket budget. They are not mutually exclusive, but know which problem each one solves: the budget caps amplification; the breaker also spares the caller from waiting.
The name comes from a ship's bulkheads: watertight compartments so that one breach does not flood the whole hull. It is another of the stability patterns attributed to Release It!. In software it means splitting scarce resources so that a slow dependency can only exhaust its own share: a connection or thread pool per dependency, a semaphore capping concurrent calls to each one, or quotas per client or per tenant (see multi-tenancy).
In the opening example, a bulkhead of twenty concurrent calls to recommendations would have left those twenty stuck and the rest of the pool free for payments. Some tools use other names for the same idea: what Envoy calls circuit breaking is really a set of per-cluster limits on connections, pending requests, in-flight requests and active retries, closer to a bulkhead than to Fowler's breaker.
The mechanisms so far protect the caller. The next two protect the server when more work arrives than it can handle.
Backpressure means the consumer sets the pace for the producer. The Reactive Streams specification (1.0, April 2015) was defined as a standard for asynchronous streams with non-blocking back pressure, and Java 9 adopted its interfaces as java.util.concurrent.Flow: the subscriber requests a number of items and the producer sends no more. How this applies to queues and logs (prefetch, maximum length, lag) is in caching, queues and streaming.
java.util.concurrent.Flow
Load shedding is deliberately rejecting part of the traffic as you approach saturation, so you can keep serving the rest well. The SRE book suggests basing it on CPU, memory or queue length, and mentions two queueing techniques: serving in LIFO order or using CoDel, because requests that have waited a long time are probably no longer wanted. It also describes rejecting lower-criticality requests first. The Builders' Library has a full article on the subject ("Using load shedding to avoid overload").
Rejecting well is design too: answer 503 or 429 (RFC 6585, April 2012) with Retry-After so the client knows how long to wait, and do it early and cheaply, before spending work on the request. Per-client rate limiting, which shares techniques with shedding, is covered in scalability and classic cases.
Between working fully and failing there is a middle ground: do less work per request. The SRE book gives examples such as searching a cached subset of the data or using a cheaper ranking algorithm. In product terms it usually looks like this: hide the recommendations, show a generic catalogue, serve the last known value with its age visible, or accept an order and confirm stock later.
Two warnings from the sources:
The practical rule: a default is acceptable when being wrong is cheap and visible (generic recommendations), and dangerous when the value decides money or permissions (a "default" price or permission). In that second case, fail explicitly.
None of these mechanisms is proven until something really fails. Chaos engineering brings that moment forward. It started at Netflix: the team introduced Chaos Monkey in December 2010, in a post on lessons from moving to AWS, as a service that randomly terminates instances to force teams to build services that survive it; it was open-sourced in July 2012. The Principles of Chaos Engineering site defines it as the discipline of experimenting on a system to build confidence in its capability to withstand turbulent conditions in production; the "Chaos Engineering" paper (Basiri et al., IEEE Software, 2016) uses almost the same definition, applied to distributed systems.
The site lists five principles: build a hypothesis around steady state, meaning a measurable output of the system (throughput, error rates, latency percentiles) rather than its internal attributes; vary real-world events (a server dying, malformed responses, a traffic spike); run experiments in production; automate them to run continuously; and minimize the blast radius. In practice, start small and keep a stop button handy.
Before reaching production, the SRE book suggests load testing each component until it breaks, with both gradual ramps and sudden spikes, checking whether the system returns to normal on its own, and blackholing non-critical dependencies to confirm they cannot drag down critical ones. That last experiment would have caught the recommendations problem before Friday.
Resilience is measured with the vocabulary of quality attributes: SLIs, SLOs and error budgets, using percentiles rather than averages. That article explains how to set them; what matters here is which signals tell you whether the mechanisms work:
Some of this can be checked in CI as a fitness function: for example, a rule that fails if an HTTP client is built without an explicit timeout.
Status as of October 2026, from their repositories and package registries:
If you run a service mesh and a library, decide which of the two layers owns retries: if both retry, you are back to amplification.
400
403
The opening scenario, with illustrative numbers. The checkout page calls four dependencies and the edge grants a total deadline of 1.5 s:
The reasoning, step by step:
Consulted in October 2026. The book Release It! was not read directly: what is said about it comes from Fowler's article, Wikipedia and reviews.
Resilience in distributed systems