Part VI · Chapter 12 · The Decisions · lesson 2

Design for the load you can measure, plus one order of magnitude

15 min

Builds on Will it fit on one box?, What one request costs you, in cents and From "we have 50,000 users" to requests per second

One product, designed three times.

A booking service: businesses publish free slots, customers reserve them, both sides get an email. At a hundred businesses it is one process and one database. At ten thousand it is the same two things on a bigger box, plus four additions. At ten million customers it is nine components, every one of them with a measurement behind it.

Before the arithmetic: which addition is forced first?

Predict

Starting from one process and one database, which component does this product need first?

A read replica, because reads outnumber writes heavily in a booking productA cache in front of the availability queryA second application server behind a load balancerA background worker, because the email provider is inside the request

The staircase, which is the shape of the whole problem

Before the three designs, the arithmetic that explains why "design for ten million" is a different question from "design for ten thousand", and why the answer is not a smooth curve.

Capacity gets bought in doublings. You move up box sizes, paying a premium at each step, until there is no bigger box — and then the next unit of growth forces a second node, a shard key, a coordination layer and an on-call rota, all at once. The cost does not rise with your user count. It sits flat, and then it steps.

Scale steps

Hypothetical model, not a forecast

Monthly cost against user count, where cost is a staircase: bigger boxes up to a ceiling, then more boxes. Rows sweep the user count, so period is users rather than time.

It assumes

One base box serves twenty thousand of these users, which is a booking product's read-heavy mix.
Capacity is bought in doublings and each doubling costs fifteen per cent more than twice the last.
Four doublings are available, so the largest single box is sixteen times the base one.
Past that ceiling you buy whole copies of the largest box and pay a coordination tax on everything.
The coordination tax is a quarter of the bill, standing in for the pooler, the router and the cross-shard work.
Monthly costs here are illustrative round numbers for the shape of the staircase, not a quote.

It ignores

Engineering time, which is the real cost of the horizontal step
Storage, bandwidth and managed-service charges priced separately from compute
Reserved and spot pricing, which flatten the staircase at the cost of flexibility
That the horizontal step usually arrives as an outage rather than as an invoice

Leave the user count at ten million. Now raise 'users one base box serves' until the whole load is back on a single node, and read the monthly cost.

Users to serve10,000,000

Users one base box serves20,000

Doublings available before the ceiling4

Four doublings means the largest box is sixteen times the base.

Monthly cost at this user count: 223,873
Monthly cost per user: 0.022
Nodes: 32
Users this tier covers: 10,240,000
Users until the next price step: 240,000
Users the biggest single box covers: 320,000
Cost of the biggest single box: 5,597
Cost one user past the ceiling: 13,992
Times the bill multiplies at the ceiling: 2.5
Already past the vertical ceiling: 1

Monthly cost of the tier that covers this load

Flat stretches joined by vertical jumps. The jump at the vertical ceiling is the tall one: the bill multiplies by two and a half for one extra user, and every step after it is a straight line because you are just buying more copies of the same box.
Users served	Monthly cost of the tier that covers this load
0	200
100000	2,433
200000	5,597
300000	5,597
400000	13,992
500000	13,992
600000	13,992
700000	20,988
800000	20,988
900000	20,988
1000000	27,984
1100000	27,984
1200000	27,984
1300000	34,980
1400000	34,980
1500000	34,980
1600000	34,980
1700000	41,976
1800000	41,976
1900000	41,976
2000000	48,972
2100000	48,972
2200000	48,972
2300000	55,968
2400000	55,968
2500000	55,968
2600000	62,964
2700000	62,964
2800000	62,964
2900000	69,960
3000000	69,960
3100000	69,960
3200000	69,960
3300000	76,956
3400000	76,956
3500000	76,956
3600000	83,952
3700000	83,952
3800000	83,952
3900000	90,948
4000000	90,948
4100000	90,948
4200000	97,944
4300000	97,944
4400000	97,944
4500000	104,940
4600000	104,940
4700000	104,940
4800000	104,940
4900000	111,936
5000000	111,936
5100000	111,936
5200000	118,932
5300000	118,932
5400000	118,932
5500000	125,928
5600000	125,928
5700000	125,928
5800000	132,924
5900000	132,924
6000000	132,924
6100000	139,920
6200000	139,920
6300000	139,920
6400000	139,920
6500000	146,917
6600000	146,917
6700000	146,917
6800000	153,913
6900000	153,913
7000000	153,913
7100000	160,909
7200000	160,909
7300000	160,909
7400000	167,905
7500000	167,905
7600000	167,905
7700000	174,901
7800000	174,901
7900000	174,901
8000000	174,901
8100000	181,897
8200000	181,897
8300000	181,897
8400000	188,893
8500000	188,893
8600000	188,893
8700000	195,889
8800000	195,889
8900000	195,889
9000000	202,885
9100000	202,885
9200000	202,885
9300000	209,881
9400000	209,881
9500000	209,881
9600000	209,881
9700000	216,877
9800000	216,877
9900000	216,877
10000000	223,873

Not there yet — keep moving the controls.

Three readings out of that, and the third one is the lesson.

The step at the ceiling is the only upgrade worth planning for in advance. One box short of the ceiling, the bill is about five and a half thousand dollars a month. One user past it, it is about fourteen thousand — two of the largest box plus the coordination tax — and nothing about the product changed. Every other transition on that chart is a bigger instance type and a maintenance window.

Per-user cost does not improve with size. It is about two cents a month at ten thousand users and slightly worse at ten million, because the coordination tax applies to the whole bill once you are horizontal. There is no economy of scale waiting for you. Growth does not make the architecture cheaper per user; it makes it more of a thing to operate.

The cheapest step on the staircase is the one that moves you down it. The challenge above is the point: raising how much one box serves — a missing index, a chatty query pattern, a payload nobody trims, a cache that works — moved ten million users from thirty-two nodes to one and cut the bill by a factor of roughly forty. No architecture diagram changed. That is almost always the first thing to try and almost never the thing on the whiteboard, because it is not a component and nobody gets to name it in a design review.

A hundred businesses

Roughly a thousand customers between them. Peak is a few requests a second.

One application process. One managed relational database. One object store for the logos and the exported reports.
One background worker and a table of jobs. The email and the payment webhook go here. Forced by: the share of the booking endpoint's response time spent waiting on one outbound call, which is measurable on day one with no users at all.
Two application processes behind a load balancer. Forced by: deploys dropping requests, and a single host being a single outage. An availability reason, not a capacity one, and worth saying out loud so nobody writes "for scale" in the document.

That is it. No cache, no replica, no broker, no second service. The database at this size answers an indexed point read in single-digit milliseconds and is bored, and this shape would hold roughly a hundred times the load without changing — which the one-box arithmetic in the first track will confirm for your own numbers in an afternoon.

Ten thousand businesses

Four additions, and each one names the measurement that forced it. Notice that none of them is a service.

A connection pooler. Forced by: connection count, not request rate. Each application process holds a pool, every autoscaling event multiplies it, and the database's connection limit is reached long before its CPU is. The symptom is connection errors under normal load, and the fix is one process in front of the database. This is the single most common first database upgrade and almost nobody predicts it, because the metric everyone watches is query time.

A read replica for reporting. Forced by: the measured effect of the monthly report on unrelated endpoints. Not "reads outnumber writes" — the specific observation that when the report runs, the booking endpoint's upper percentiles move, because one analytical query is competing for the same buffer cache and the same disk. Send that one query elsewhere. Do not send the booking reads anywhere: they are fast, they are consistent, and a replica makes them stale.

Partitioning the slots table by date. Forced by: row count crossing the point where the planner changed its mind, and the realisation that the vast majority of the table is in the past and nothing ever reads it. The honest trigger here is not a size, it is a plan change you noticed in a query plan.

A cache on the public availability page. Forced by: a measured hit ratio. The same handful of rows for a popular venue are read thousands of times a minute by people who are not logged in, and the page tolerates being a minute stale because the booking itself re-checks under a constraint. Both halves of that sentence are required, and the second half is the one that makes it safe.

What is still absent at ten thousand businesses: a message broker, more than one service, container orchestration, a service mesh, a second region, a search cluster. Every one of those has been proposed by somebody, and the answer each time is the same question — which number forced it?

Ten million customers

Now the components arrive, and they arrive with numbers.

A horizontal application tier. Forced by the staircase above: one box is past its ceiling. This is the first upgrade in the whole story that is forced by raw load.
A partitioned or sharded primary, keyed by business. Forced by write throughput against one primary's sustainable rate, and by the table size at which maintenance stops fitting in a window. The shard key is the genuinely one-way decision here, which is why the previous lesson wanted a tenant key on every row from the first migration.
A real broker rather than a job table. Forced by a crossing: there are now four independent consumers of the booking event — email, the search index, the warehouse, partner webhooks — and one needs to replay a week of history after a bug. A job table does one consumer well and four badly.
A search index. Forced by query patterns the relational engine cannot serve at an acceptable response time: free text, geographic radius, facets, ranking.
A content delivery network and a second region. Forced by measured response times on another continent, or by a contract naming where data lives. Both are numbers; one of them is in a PDF.
The on-call rota, the objectives, and the detection the reliability tracks are about. Forced by the only number that matters at this size: how much of the year this may be unavailable.

Nine components, from three. Worth noticing how small the list stays when every entry has to name its trigger — the version drawn on whiteboards usually has twice as many boxes and half as many numbers.

What would you do

You are the ten-thousand-business version. Sales has signed two enterprise pilots and is forecasting a hundredfold increase in customers within two years. Leadership has asked for an architecture that "will not need to be rebuilt", and somebody has drawn the ten-million diagram on a whiteboard.

You have four engineers. The current system is comfortable and nothing is on fire.

Optimising for: Minimise total engineering cost over the next eight quarters, given a sales forecast of hundredfold growth that nobody outside the sales team believes.
You cannot know: Whether the forecast is real, which nobody will know for at least three quarters
Whether growth arrives as a steady climb or as two enterprise launches in one week
Which of the four current additions breaks first under ten times the load, because nobody has load-tested it
Whether the enterprise pilots bring data residency requirements, which would force a region decision for contractual rather than technical reasons

Build the ten-million architecture now, while there is budget and attention for itChange nothing, and keep a written list of triggers with the action beside eachKeep the trigger list, and spend two weeks now only on the things that get dearer with every rowBring in a consultancy to design the target architecture, and build towards it incrementally

The rule, with the number it rests on

Design so the current design survives about ten times the load you can measure, and keep a written list of what you would do at a hundred times.

The factor of ten is not magic and it has two defensible sources. It is roughly the headroom one vertical step on the staircase buys, so it is the growth you absorb without a shape change. And it is roughly the growth a healthy product achieves between a capacity signal appearing on a dashboard and a team finishing the response — which, for anything involving a schema change, is a quarter.

Below ten times you are buying capacity you can observe the need for. Above it you are buying on a forecast, which is what the first track's case file priced: forty per cent of a five-person team, for eighteen months, with nothing a customer could see.

The hundred-times list is what separates this from a slogan. It takes an afternoon, it names the trigger beside each action, and its real output is a list of things to instrument — because a trigger you cannot currently measure is not a trigger, it is a hope.

Check

Which of these is a trigger rather than a forecast?

We will shard the database when we reach a million customersWe will shard when sustained write throughput on the primary exceeds two thirds of what we measured it serving in the load test, for a weekWe will add a cache when the database starts strugglingWe will move to a message broker once we have adopted a microservice architecture

The same product designed by somebody optimising for the design reviewOptional

Worth naming the opposing incentive honestly, because it is strong and it is not stupid.

An architecture with nine components is easier to present than one with three. It demonstrates knowledge of nine things, gives every senior engineer in the room something to have an opinion about, maps onto team boundaries somebody is hoping to create, and is the architecture your preferred candidates have on their CVs. Presenting the three-component version requires you to say "we do not need that yet" nine times to people more senior than you, and to be right, in public, with no evidence yet available.

That asymmetry is why the measurement discipline has to be written down rather than held as a preference. A trigger list makes the refusal impersonal: the answer to "should we add a search cluster?" stops being a judgement about your taste and becomes a lookup. It also commits you in the other direction, which is what makes it fair — when the trigger fires you have pre-agreed to the component, and you do not get to move the threshold because you have grown attached to the simplicity.

The further reading that helps here is not architecture writing. It is your own invoice and dashboards from twelve months ago, read beside the components you added in that period, asking for each one which measurement was on the screen when you decided.

Worth remembering

The first component this product needs is forced by a dependency's response time, not by load: one outbound email or payment call inside the request moves the work to a background worker on day one.
The second is forced by deploys and single-host failure, which is an availability justification — so do not write 'for scale' next to the load balancer.
Capacity is a staircase, not a curve. The bill sits flat and then steps, and the tall step is the vertical ceiling, where it multiplies by about two and a half for one extra user.
Per-user cost does not improve with size. There is no economy of scale coming, because the coordination tax applies to the whole bill once you are horizontal.
The cheapest move on the staircase is the one that moves you down it: making one box serve more users took ten million users from thirty-two nodes to one in the model.
At ten thousand businesses the four additions are a connection pooler, a reporting replica, a partitioned table and one cache — and the pooler is forced by connection count, which nobody predicts because everybody watches query time.
A cache needs both halves of its justification: a measured hit ratio, and a page that tolerates being stale because the write path re-checks.
Build ahead of demand only where cost grows with your data — a tenant key, an idempotency key — because components cost the same whenever you adopt them and later means adopting them against a measurement.
A trigger needs a measurement you already collect, a threshold and a duration. Its most useful output is the list of things you have to instrument.

Where this stops holding

All of it assumes you see the growth coming. One vertical step of headroom is a plan because it buys you a quarter to respond, and that is only true if the growth takes a quarter to arrive.

It often does not. A launch, a press mention, a partner switching their traffic over on a Monday morning, a link on a large aggregator: these arrive in minutes, and against them a tenfold headroom is not a buffer, it is the entire budget, consumed before anybody reads the alert. The provisioning-lag arithmetic in the serving track is the honest version — if autoscaling needs several minutes to add capacity and the ramp arrives in one, the headroom you hold is all the capacity you will have during the event.

So the crossing is: when the time from the first signal to the peak is shorter than the time it takes you to add capacity, headroom stops being a plan and admission control starts being the answer. The tools are the ones from the reliability track rather than from this one — a queue in front of the expensive path, a rate limit per tenant, shedding the cheapest-to-refuse traffic first, and a degraded response that is better than a timeout. Designing for burst is designing what you refuse, and no amount of designing for the load you can measure prepares you for it.