One checkout, three different risks
A customer confirms an order in an online store, and the checkout service sends a request to the payment provider. The payment succeeds, but the response never reaches the checkout service. After waiting a few seconds, the application shows an error, and the customer clicks “Try again.”
Because the checkout service does not know the payment’s outcome, a repeated request may execute the same operation again. Meanwhile, slow requests can keep resources occupied, and automatic retries can add more load. A single lost response can therefore create problems for both the payment’s status and the system’s availability.
To understand this scenario, we need to consider three constraints of distributed systems together:
- Transferring data takes time.
- The systems we depend on sometimes become unavailable.
- Resources are limited.
Using this checkout example, we will explore how these constraints affect decisions about timeouts, retries, data freshness, and load management.
Transferring data takes time
In code, a remote call can look like an ordinary method call:
inventoryClient.reserve(productId, quantity);
To execute this call, data must be serialized, transmitted over the network, and processed on the other side. Once the remote service finishes, its response must travel back. Establishing a connection, network congestion, and time spent in a queue can all increase the delay. An HTTP client may hide many of these steps from the code, but it cannot remove the time they take.
Latency also affects when information reaches other components. For example, when a customer buys the last item in stock, the inventory service updates its database, while the search index and product cache receive the change asynchronously. Until the update reaches them, different components may show different information about the same product:
- The inventory service reports that the product is unavailable.
- Search results still show it as available.
- The cached product page still says “In stock.”
Each component works with the information it has received so far. The design must therefore make clear which operations can tolerate stale data. A search result that is briefly out of date may be acceptable; at checkout, we need to make sure the product is actually available.
These two flows have different requirements. A product page can read from a cache for a fast response. Checkout should rely on the inventory service’s authoritative data and use a reservation operation that correctly handles concurrent orders for the last item.
The same product, seen at different times
Before the purchase, every component shows the same state.
Checkout must rely on an inventory operation that handles concurrent reservations correctly, rather than on the cache.Every call uses part of the time budget
Every remote call consumes part of the request’s latency budget, the total time available to return a response. The delays of sequential calls add up, and a slow dependency leaves less time for the remaining steps. Whenever we add a service call, we must decide how long we can wait for it and what to do if it times out.
Clarifying the data-freshness requirement helps make that decision:
Does this operation need the remote service’s latest data right now, or can it use information already available locally?
If local data is sufficient, an extra network call may be unnecessary. If the latest state matters, time for calling that dependency must be allocated within the overall deadline.
A dependency may be unavailable, and the outcome may be unknown
A server failure, a broken network connection, or a configuration change can make a service we depend on unavailable. For the calling application, the difficulty goes beyond not receiving a response: sometimes it cannot tell whether the operation ran on the other side at all.
For example, when a payment request times out, any of these things may have happened:
- The request never reached the payment provider.
- The provider accepted the request but has not finished processing it.
- The payment completed, but the response was lost.
The caller sees only that the expected response did not arrive and may be unable to distinguish these cases. A timeout means the caller stopped waiting. It does not prove that the operation failed. This distinction matters when deciding whether to retry, because a repeated request could execute an already completed payment again.
To prevent this, retries of the same payment must keep the same operation identifier. An idempotency key lets a provider that supports this feature recognize a repeated request and avoid charging the customer twice.
If the result is still unknown, the application can keep the payment pending and check its status with the provider. Reconciliation helps match the local record to the provider’s result. In this situation, it is more appropriate to tell the user that the payment is being checked than to immediately mark it as failed and direct them to make a new payment.
The request timed out. Was the customer charged?
payment-order-1042The same across all attemptsThe checkout service sends a stable idempotency key for the same business operation.
Which mechanism solves which problem?
Timeouts, retries, and other resilience mechanisms serve different purposes. When choosing them, distinguish the failures each one addresses:
| Mechanism | Purpose |
|---|---|
| Timeout | Limits how long the caller waits |
| Retry with backoff and jitter | Allows another attempt after a temporary failure, with increasing and randomized delays between attempts |
| Circuit breaker | Temporarily reduces calls to a dependency that is repeatedly failing |
| Bulkhead | Limits how much of a shared resource one dependency or workload can use |
| Idempotency | Makes repeated attempts safe for operations that support it |
The limits for these mechanisms must work together: the wait for each retry and the delays between attempts must fit within the request’s overall deadline. A fallback must also match the meaning of the operation. Showing cached product information may be acceptable, for example, but reporting a successful payment without confirmation from the provider is not.
Resources are limited
The load a system can accept is constrained by CPU, memory, network bandwidth, database connections, worker threads, and queue capacity. Cloud infrastructure can expand some of these resources, but scaling takes time. Cost, quotas, and downstream system limits also matter.
When requests take longer to complete, more work remains active at the same time. Under approximately steady flow, the relationship can be expressed as:
Average in-flight requests ≈ request arrival rate × average time spent in the system.
If a service receives 200 requests per second and each request takes an average of 100 milliseconds, about 20 requests are active at once. If the average duration rises to two seconds while the arrival rate stays the same, that number reaches about 400. Even without more incoming traffic, increased latency alone can greatly increase the work held in the system.
This calculation does not mean that 400 threads or database connections are required; actual resource use depends on the implementation. However, if waiting requests hold connections, memory, or other limited resources, less capacity remains for new work.
Retries use those same resources. Sending additional attempts to a slowing dependency can increase load and make the queue grow. An unbounded queue postpones the problem: memory use increases, and some requests are no longer useful by the time processing reaches them.
Same traffic. What about the load?
200 requests/s × 0.1 s ≈ 20 in-flight requests
How should you read this calculation?
A simplified calculation based on Little’s Law for a steady average flow. The bar shows the proportion of 400 requests, not an actual capacity limit. This is not a thread or connection count; an indefinitely growing queue does not satisfy the steady-state assumption.What should happen when capacity runs out?
As a system approaches its capacity, the aim is to retain enough resources to finish work already accepted. This requires defining limits and deciding in advance what happens when those limits are reached:
- Bound the queue size and the number of operations running concurrently.
- Limit both the number of retries and their total duration.
- Apply rate limiting and backpressure where appropriate.
- Reject extra work early if accepting it would threaten the system’s overall availability.
- Monitor waiting time and resource saturation alongside CPU use.
Rejecting a new request early is disappointing for the user, but it can prevent every request from waiting a long time and eventually timing out. Which work is accepted and which is deferred should follow the service’s requirements.
How do these three rules amplify one another?
Return to the checkout scenario: as the payment provider slows down, requests remain active longer and shared resources begin to fill up. Callers that time out and retry add more load. A delay in one dependency can spread to other parts of the application that were working normally.
Each mechanism has a distinct role in this flow. A timeout limits the caller’s wait, while a concurrency limit prevents active work from exhausting resources. A retry budget controls the load from repeated attempts; idempotency governs repeated execution of the same business operation. These mechanisms must be chosen together, because a timeout alone does not address extra retry load or the risk of duplicate payments.
Three questions for reviewing a design
Every important operation between services should have a concrete answer to these three questions:
- What happens when data is delayed?
- What happens when a dependency is unavailable or the operation’s outcome is unknown?
- What happens when available capacity is exhausted?
For checkout, the answers become practical behavior: a payment with an unknown outcome is checked, retries use the same idempotency key, and requests that exceed the system’s capacity receive a clear response. The user sees that their payment is still being checked, rather than being told it has failed.
When reviewing architecture decisions, trace these cases alongside the normal flow. Knowing which data may be stale, which operations may be repeated, and which work will stop under increasing load makes both implementation and decisions during a failure easier.