The company that built for ten million and got eight thousand

Eleven services, three Kubernetes clusters, a managed Kafka, a service mesh, and a GitOps pipeline. Five engineers. Eight thousand users, who between them generated about four requests a second.

It all worked. Nothing ever fell over from load, because nothing was ever remotely close to load.

Before the case file: what was the largest cost of building it this way?

Predict

Eleven services, three clusters, a mesh and a message bus, for eight thousand users and five engineers. What cost the most?

Choose one answer

The monthly cloud billThe engineering time — two of the five people were effectively running a platformThe incidents, since more moving parts means more failuresHiring, because this stack needs people who know it

Lock in answer

Case file

◇ Composite, not a real person

A seed-stage B2B workflow product with five engineers, eighteen months after its first customer

One Python application and one Postgres database, deployed by a shell script, serving about four hundred users with no outages worth remembering.

Users

8,000

Peak load

~22 requests a second

Monthly bill

~$4,400

After simplifying

~$800

Engineers

5, of whom ~2 on platform

Change to production

~35 minutes

1. 1Split the application into eleven services, along the boundaries of the pages in the product rather than along data ownership.
2. 2Adopted Kubernetes on a managed control plane, with one cluster per environment: development, staging and production.
3. 3Added a service mesh for mutual TLS between services, to answer a security questionnaire from an enterprise prospect.
4. 4Added a managed Kafka cluster so that services could communicate asynchronously instead of calling each other directly.
5. 5Containerised every service, pinned base images by digest, and built them in CI.
6. 6Wrote every piece of infrastructure as Terraform from the first cluster onward, with no manual console changes permitted.

A correct, stable, observable system costing about $4,400 a month to run, consuming roughly forty per cent of the engineering team's time, in which a one-line change took about thirty-five minutes to reach production. Eighteen months later, two engineers moved it back to two application servers and a managed Postgres in a fortnight, for about $800 a month, and no customer noticed anything except that releases got faster.

What luck did

The simplification happened at all, and it happened because the engineer who had championed the original platform left for reasons that had nothing to do with it. Nobody had to lose an argument in public, which is the usual thing that stops a team from unwinding its own architecture.

Who else tried this

A large number of teams built approximately this stack for approximately this user count, and most of them are still running it today, paying the bill and the attention. Almost none of them will ever record it as a mistake, because nothing visibly failed and the alternative was never measured.

Why it is not a template

Reversing this was a fortnight's work only because the eleven services were one codebase behind eleven deployments, sharing one database, with no other team consuming any of their interfaces. A team with genuinely separate datastores, other teams depending on its interfaces, or a compliance obligation attached to the isolation cannot do this in a fortnight and should not try.

What each piece actually bought

The mesh bought one checkbox on a questionnaire. Mutual TLS between services was a line item from an enterprise prospect that did not sign. The mesh's own first production incident was a certificate rotation that failed inside a sidecar nobody on the team had read the source of, and every service-to-service call failed for about forty minutes while five people read documentation. The service that was down was not the service that broke.

Kafka carried about twelve messages a second. That is a Postgres table with a status column and a worker polling it, which is what they replaced it with, and which nobody has thought about since. Kafka earns its place when you need replay, multiple independent consumers of the same stream, or throughput a database table cannot carry — none of which was true here.

Three clusters bought environment parity, badly. Development and staging were mirrors of production in topology and nothing like it in data, so the things they caught were deployment-layer problems, which were problems the deployment layer had introduced.

Eleven services bought nothing, because they were not independent. Split by page rather than by data ownership, they all read and wrote the same database, so every meaningful change touched several of them and had to be released together. They were one system with eleven deployment units and eleven times the operational surface.

The two decisions that were still right

This is the part the usual version of this story leaves out, and leaving it out is what makes the story useless.

Containerising the application, with base images pinned by digest and built in CI. This survived the migration completely intact. The same images that ran in Kubernetes ran on two virtual machines under a container runtime the following month. It made the build reproducible, it removed an entire category of "works on my machine", and it is the one piece of the original platform nobody has ever proposed removing. If the team had done only this, the eighteen months would have been net positive.

Putting all infrastructure in Terraform from the first day, with no manual changes allowed. This is why the simplification took a fortnight rather than a quarter. The team could read what the old stack actually contained — every security group, every subnet, every parameter — rather than discovering it by breaking things. The new, smaller stack was a reviewable diff against the old one, which also meant the decision could be argued about on its merits with the real list in front of everybody.

Notice the shared property. Both of these are about reproducibility, and neither is about scale. The decisions that aged well were the ones that did not depend on a forecast being right.

Check

Which of these would have made the original architecture the correct choice at eight thousand users?

Choose one answer

A credible forecast of ten million users within two yearsA signed contract requiring network-level isolation between components, with an auditor checking itA team of five who all already knew Kubernetes wellTwo separate products sharing nothing but a login

Check answer

When this stops being the wrong answer

The condition is the honest one, and it is in the premise: when the growth that was forecast actually turns up.

At eight thousand users none of it was needed. At eight million, with thirty engineers in six teams that must deploy without coordinating, most of it becomes the straightforward answer: you need orchestration because you have hundreds of containers, you need a mesh or something like it because you have policy to enforce across teams, and you need a log because you have genuinely independent consumers of the same events.

So the criticism is not of the components. It is of the ordering. Every one of those pieces can be adopted in a fortnight when the need is in front of you, with the numbers to size it, by a team that has grown enough to operate it. The same pieces adopted eighteen months early cost forty per cent of a five-person team for the whole of those eighteen months, and bought nothing that could be pointed at.

Which gives the one rule I would actually hand somebody: adopt the complexity when you can name the number that forced it. Not the forecast — the measurement. "Our deploys are blocking three teams" is a number. "We will have ten million users" is a hope with infrastructure attached.

›Why the bill is the least interesting number in the case fileOptional

Because it is the only cost in the story that somebody is accountable for.

A cloud bill arrives monthly, has a line for each component, and gets reviewed by a finance function that will ask about it. That makes it the cost most likely to be noticed and the most likely to be fixed — which is exactly why teams end up with the same architecture on smaller instances.

The forty per cent of engineering time has none of those properties. It arrives as a day of cluster upgrade here, an afternoon of certificate debugging there, a week of pipeline work that reads as infrastructure investment in a stand-up. Nothing aggregates it, nobody owns it, and it never appears beside the roadmap items it displaced. The only way to see it is to go looking: count the pull requests that touched the deployment layer against the ones that touched the product, over a quarter, and the ratio is usually a surprise to everyone including the people who wrote them.

That count takes an afternoon and it is the most useful architectural measurement a small team can make.

Worth remembering

• Eleven services, three clusters, a mesh and a managed Kafka served eight thousand users generating about four requests a second, for roughly $4,400 a month.
• The largest cost was not the bill: two of five engineers were effectively operating a platform, and that cost arrived a day at a time with nobody aggregating it.
• Eleven services split by page rather than by data ownership all shared one database, so they were one system with eleven deployment units and no independence.
• The mesh answered a questionnaire from a prospect that did not sign, and caused the worst incident in the period through a certificate rotation inside a sidecar nobody had read.
• Kafka carried about twelve messages a second, which a Postgres table with a status column and a polling worker replaced permanently.
• Two decisions were still right: containerised builds with digest-pinned base images, and all infrastructure in Terraform — and both are about reproducibility rather than scale.
• Reversing the whole thing took a fortnight and cost about $800 a month afterwards, and that was only possible because no other team consumed any of the interfaces.
• Adopt the complexity when you can name the measurement that forced it — 'our deploys are blocking three teams' is a number, 'we will have ten million users' is a forecast.