It works for you and falls over for a thousand people

Every program you have written so far has had one user, and that user was you. You clicked, it responded, you clicked again. Requests arrived one after another because there was only one person making them.

Almost everything that is hard about systems comes from removing that assumption.

This lesson has no arithmetic in it. The arithmetic starts in the next chapter and runs for the rest of the subject. What you need first is the mental picture the arithmetic is about, because the numbers are meaningless until you can see what they are counting.

Predict

One request takes a tenth of a second of real work. Ten requests arrive at the same instant on a machine with one processor core. How long does the last one take to finish?

Choose one answer

A tenth of a second — they run at the same timeAbout a second — the work has to happen one piece at a time, and the last one waits for the other nineTen times a tenth of a second, so each takes a full secondIt cannot be answered without knowing the language

Lock in answer

Waiting is a separate thing from working

A beginner optimises service time, because that is the part the code controls and the part a profiler shows. For a system under real load, wait time is usually the larger of the two, and it does not appear anywhere in the code. You cannot find it by reading the program, because it is not in the program — it is in the arrival pattern.

This is why "it was fast on my machine" is such a persistent and honest confusion. On your machine, wait time is zero. It is zero because you are one person, and no amount of careful testing by one person will produce it.

Two numbers, and they are not the same question

The two words you will hear constantly, which beginners regularly swap:

Latency is how long one request takes. Measured per request, in units of time.

Throughput is how many requests get finished per unit of time. Measured per second.

They are related and they are not the same, and the gap between them is where most capacity misunderstandings live. A system can have excellent throughput and terrible latency at the same time: a queue that is drained efficiently can finish a great many jobs per minute while any individual job waits a long while for its turn. Supermarket checkouts are exactly this. One till open all day has decent throughput and a long wait; the shop's number and the shopper's number are different numbers, and the shop usually quotes its own.

The question "can it handle a thousand users" is a throughput question wearing a latency costume. What people actually want to know is whether the thousandth person has an acceptable wait, and that is a question about both.

Check

A service is handling every request it is given and finishing all of them, and users say it feels sluggish. Which number is the problem?

Choose one answer

Throughput — it must not be keeping upLatency — the work is getting done, and each person is waiting too long for theirsBoth, since they always move togetherNeither — 'feels sluggish' is not a measurement

Check answer

What I would actually do, before any tooling

Ask how many people are using it at once, not how many have accounts. A product with many thousands of registered users and a handful of simultaneous ones is a small system. The two numbers get confused constantly, usually by whoever is most keen for the system to be impressive.

Ask what the slowest single request does. One request that waits on something external while holding a worker is how a system with plenty of spare processor becomes unresponsive. The pattern matters more here than any number.

Expect the complaint before the graph. At the point where waiting starts to appear, the machine's own measurements still look unremarkable — the processor is not pinned, memory is fine. Chapter three is entirely about why the trouble arrives well before anything looks full, and it is the single most useful thing in this subject.

When requests genuinely never overlap

The condition under which none of this applies: when only one thing ever happens at a time.

A nightly job, a build step, a script you run by hand, a desktop tool with one user at the keyboard — these have no wait time, because there is nothing to wait behind. For them, service time is response time and making the code faster makes the thing faster, directly and predictably.

That is a real and common category, and it is worth naming because the advice for it is the opposite of the advice for everything else in this subject. If nothing overlaps, profile the code and optimise the slow part. If things overlap, that habit will have you making the wrong part faster, which is the subject of a lesson two chapters from here.

Worth remembering

• One user means zero wait time, which is why 'fast on my machine' is honest and useless.
• Response time is service time plus wait time; the code controls the first and the arrival pattern controls the second.
• Ten simultaneous requests on one core give the first a short wait and the last a long one — the average hides the spread.
• Latency is how long one request takes; throughput is how many finish per unit of time, and they are different questions.
• Throughput can rise while latency gets worse, which is the normal picture near a limit.
• 'Can it handle a thousand users' is really a question about both numbers, and about simultaneous users rather than registered ones.
• When nothing overlaps — a nightly job, a script, a desktop tool — there is no wait time and optimising the slow code genuinely is the answer.