Part IV · Chapter 6 · Several Services · lesson 4

What does this return when the service it needs is gone?

14 min

Builds on The cache is serving data from twenty minutes ago

Checkout calls the fraud service on every order. For the last ninety seconds the fraud service has been refusing connections, and checkout is returning a blank error page to every customer who taps Pay. Nobody chose that behaviour. It is what an unhandled exception does when nothing above it has an opinion.

Four other things could happen instead. Three of them are better here and one of them is better somewhere else.

Predict

The fraud service is unreachable. Which behaviour should checkout have, and why that one?

Refuse the order and tell the customer to try again — fail closedAccept the order without screening it — fail openIt depends on what a wrong answer costs compared with no answer at allAccept the order and screen it later from a queue

The two questions

What does a wrong answer cost, next to no answer at all? If proceeding without the dependency can authorise something harmful — a payment, an access grant, a deletion — then no answer is cheaper than a guessed one. If proceeding without it only makes the page plainer, a guessed answer is nearly free.

Does the caller need the result before it can reply? If it does, you are choosing between refusing, guessing and reusing an old answer. If it does not, there is a fourth option that is better than all three, and it is the one most often skipped because it requires changing the product's promise rather than its configuration.

The four, with the mechanism that implements each

Fail closed. Refuse. This is the correct default for anything that decides whether an action is permitted. In Envoy, the ext_authz filter's failure_mode_allow defaults to false, which means an unreachable authorisation service denies the request — that default is doing you a favour and is worth leaving alone. Open Policy Agent behaves the same way: no policy decision means no. The cost: your availability is now the product of your own and the dependency's, so a control you cannot reach is an outage you own.

Fail open. Proceed without it. Correct when the dependency decorates the answer rather than deciding it — recommendations, experiment assignment, the "customers also bought" strip. Most feature-flag SDKs do this by construction: the LaunchDarkly client takes a default value in the call itself and returns it when it cannot reach the service, so the code path exists whether or not the network does. The cost: the behaviour during an outage is the behaviour of the hardcoded defaults, which nobody has read in a year and which may not be the same as the live configuration.

Serve stale. Return the last value you had. Correct when the value changes slowly compared with the length of the outage, so that an old answer is still a right answer. HTTP has this written into the standard — stale-if-error on a cached response tells a well-behaved cache to serve an expired copy rather than fail — and nginx implements the same idea with proxy_cache_use_stale error timeout updating. For configuration, the equivalent is a last-known-good copy on local disk that the process reads at start-up if the config service is unreachable. The cost: you must now be able to answer "how stale is too stale", and the answer has to be enforced, or serving stale quietly becomes serving wrong.

Queue it. Accept the request, do the dependent work afterwards. Correct when the caller does not need the result to reply and the work must not be lost. The implementation I would reach for first is not a broker: it is an outbox table in the same Postgres transaction as the order write, with a worker draining it, because that gives you exactly-once-ish semantics against a database you already have and no second system to operate. Reach for SQS or Kafka when the volume outgrows a table or when something other than you needs to consume the stream. The cost: you now owe the caller a way to learn the outcome — a status field, a webhook, an email — and a queue with no consumer of its results is how a system silently stops screening orders for three weeks.

Something people say

“Failing open is the resilient choice, because the service stays up.”

Why it spreads: It is true for most dependencies in most systems, and the people who say it are usually thinking of a recommendation widget when they say it. It also sounds like the brave engineering answer next to refusing work, and 'we stayed up' is an easier sentence to say in a review than 'we refused eleven thousand orders'.
The part that is true: Most of a service's dependencies genuinely are enhancements, and failing open on those really does raise availability with no downside worth naming. A system that fails closed on everything has multiplied every dependency's unavailability into its own, which is a worse place to be than failing open on the decorative ones.

Your rate limiter is backed by Redis and the Redis connection times out. The client library fails open. What have you actually configured?

What would you do

Checkout writes the order to Postgres and calls the fraud service synchronously before confirming. The fraud service is unreachable and has been for a few minutes. Fulfilment picks orders from a warehouse queue with a published same-day cutoff. Historically the screen rejects roughly one order in two hundred, and the rejected ones skew heavily towards large baskets.

Optimising for: Keep taking orders during a fraud-service outage without accepting a loss you cannot bound.
You cannot know: How long this outage will last, which decides whether a hold is an inconvenience or a lost day
Whether the rejection rate during an outage resembles the rate outside one
Whether fulfilment can actually honour a hold flag, or whether it reads the orders table directly
What the finance team's tolerance for unscreened loss over one day actually is

Fail closed: refuse new orders until the fraud service returns.Fail open: accept and confirm every order unscreened.Accept the order, write a screening_pending row in the same transaction, and hold fulfilment until a worker drains it.Serve stale: screen against the last rules the service published, cached locally.

Check

Which of these dependencies is the best candidate for serving stale?

The service that publishes shipping rates by destination, updated about once a monthThe service that returns a customer's current account balanceThe inventory service, during a saleThe authorisation service that decides whether this user may see the record

When the dependency holds the only copy

The four behaviours all assume you have something to fall back to: a default, an old value, or a place to put the work.

Take that away and three of them disappear. If the unreachable service holds the only copy of the data — the single source of truth for the customer's address, the only ledger that knows the balance — then there is nothing to serve stale from, because you never had a copy. Failing open means proceeding without a fact you actually need rather than without a decoration. Queueing works only if what you needed was a write; it does nothing for a read, because no amount of patience produces an answer that is not stored anywhere you can reach.

What is left is failing closed, and the honest conclusion is that the choice was made earlier than this lesson. A single-copy dependency on the request path is a decision to have its availability be a ceiling on yours, and the four behaviours cannot undo it. The work is upstream: replicate the data, cache it with a staleness bound somebody has agreed to, or move the dependency off the request path entirely. Notice the shape of that — the fix is to create a second copy, which is the problem the whole of track 2 is about, arriving from the other direction.

Worth remembering

An unhandled exception is a behaviour nobody chose; fail closed, fail open, serve stale and queue it are the four that can be chosen.
Two questions pick between them: what a wrong answer costs next to no answer, and whether the caller needs the result before it can reply.
Fail closed is the right default for anything deciding whether an action is permitted, and it makes the dependency's availability a ceiling on yours.
Fail open is right where a wrong yes is cheap, and during the outage your behaviour is whatever the hardcoded defaults say — which nobody has read recently.
Serving stale is right when the value changes far more slowly than the outage lasts; it works for rules and not for counters, because a stale counter is missing exactly the events you care about.
Queueing is the strongest option when the caller does not need the result, and an outbox table in the same transaction as the write beats adding a broker until volume or a second consumer forces one.
If the unreachable dependency holds the only copy, three of the four behaviours are unavailable and the real decision was made when it went on the request path.