Raise the rate until the completions stop rising

Track one claimed that a queue gets slow long before it gets full, and that the knee arrives at a utilisation that still looks like headroom on a graph. You now have a service whose capacity you can compute to within a few per cent, a generator that does not wait for permission, and four minutes. Go and check whether the claim was true on your hardware.

One command, seven arrival rates, twenty seconds each.

Predict

Your service completes about forty-eight requests a second. You offer it sixty a second for twenty seconds, with the generator giving up on any request that has not answered inside two seconds. What does the completed-per-second column do?

Choose one answer

Flattens at about forty-eight and stays there — the service does what it can and the surplus waitsCollapses to roughly half of capacity, because the queue grows past the two-second budget and the service keeps completing requests nobody is left to receiveKeeps rising to about sixty, because the queue is unbounded so nothing is refusedDrops to near zero, because once the queue is past the timeout nothing can ever be delivered again

Lock in answer

What track one predicted, with your service time in it

Queueing delay

≈ Hypothetical model, not a forecast

How long work waits in a single-server queue as utilisation rises, and the utilisation at which that wait breaks a stated budget.

It assumes

• Arrivals are Poisson, which is what the generator produces from its fixed seed.
• One worker serves one request at a time, in order, with no preemption.
• The queue is unbounded and nothing waiting ever gives up and leaves.
• Service times vary exponentially, which for this handler is badly wrong and is the point.
• The service time used is the median measured by the acceptance check, not the declared cost.

It ignores

• Load shedding, admission control, backpressure and timeouts — a real system refuses work rather than queueing forever, and the infinity at utilisation 1 is never actually reached
• Multiple servers, which change the formula and soften the knee considerably
• Service times more regular than exponential, which roughly halve the wait
• Bursty, correlated arrivals, which make the wait worse at every utilisation
• Retries, which turn an overloaded queue into a faster overloaded queue

Leave the service time alone and raise the arrivals until the model stops quoting a wait and reports that the offered load is at or above capacity. The arrival rate where that happens is the last row of the sweep you are about to run.

Arrivals40/s

Requests reaching the server each second.

Wait budget200 ms

The queueing wait that counts as too long. Service time is not part of it.

Utilisation modelled

0.84

Utilisation offered, before clamping

0.84

Time waiting before service starts

110 ms

Time in the system, wait plus service

131 ms

Requests waiting

4.41

Response time as a multiple of service time

6.25

Utilisation at which the wait breaks the budget

90.498%

Offered load is at or above capacity

0

• Time in the system
• Time spent waiting

◷ Not there yet — keep moving the controls.

Three readings to take from that before you run anything.

At forty arrivals a second the model puts utilisation at eighty-four per cent and the time in the system at a hundred and thirty-one milliseconds — six and a quarter times the work being done. Eighty-four per cent is a number a dashboard renders as comfortable.

The two-hundred-millisecond wait budget breaks at ninety and a half per cent utilisation, which is about forty-three arrivals a second. That is a closed form rather than a search: the budget divided by the budget plus the service time.

And just under fifty arrivals a second the model stops forecasting and starts flagging. There is no steady-state wait to quote above capacity, because nothing settles.

The sweep

docker compose up -d service
docker compose run --rm load --rate 10,20,30,40,45,48,60 --seconds 20

 offered    sent     ok  refused  gaveup  error  ok_per_sec   p50_ms   p99_ms    done    lost  queued
      10     211    211        0       0      0        10.6     21.3     62.7     211       0       2
      20     410    410        0       0      0        20.5     21.5     82.1     410       0       3
      30     599    599        0       0      0        29.9     41.3    143.6     599       0       7
      40     797    797        0       0      0        39.9     61.5    226.2     797       0      11
      45     882    882        0       0      0        44.1     82.1    327.2     882       0      16
      48     931    931        0       0      0        46.5    122.6    387.9     931       0      18
      60    1178    506        0     672      0        25.3   1087.5   1946.2    1070       0     205

The whole lesson is in two columns of that table.

The completions column stops tracking the offered rate between forty-five and forty-eight. Ten offered, ten delivered. Twenty, twenty. Forty, just under forty. Forty-five, forty-four. Forty-eight, forty-six and a half. The curve leaves the diagonal and lies down on the ceiling, and the ceiling is the number you predicted in the last lesson.

The median had already tripled before the completions moved at all. At thirty offered — sixty-three per cent of capacity — the median is forty-one milliseconds against a declared twenty. Nothing is refused, nothing has timed out, every request succeeded, and the service is already taking twice as long as it costs. That is the knee, and it arrived at a utilisation most people would describe as two thirds busy.

At sixty offered, the service does more work and delivers less of it. The done column — requests the service itself completed — reads 1070, higher than any other row. The ok column — requests whose answer was received — reads 506. Five hundred and sixty-four requests were served to callers who had already left. The completions-per-second figure halves while the service works harder than it ever has.

Where the model was right and where it was out by a factor of two

The shape held. The constants did not, and they were wrong in a direction worth knowing.

At forty arrivals a second the model said a hundred and thirty-one milliseconds in the system. The measurement says sixty-one and a half at the median. The model is roughly twice as pessimistic, and that is not a bug in either — it is the model's own fourth assumption failing in the lab's favour.

The queueing model assumes service times vary exponentially. This handler's service time barely varies at all: two milliseconds of spin and eighteen of sleep, every time. A queue with near-constant service time waits about half as long as the exponential case at the same utilisation, which is the classic M/M/1 against M/D/1 result and is named in the model's own list of omissions.

So the correction is not "the model was wrong". It is: this lab's handler is the best case, and your real service is somewhere between this and the model. A handler whose cost varies — a cache that sometimes misses, a query whose plan sometimes changes — moves up towards the model's figure, and a bursty arrival pattern moves past it.

Check the generator before you believe the service

The one failure mode that invalidates a sweep is the generator running out before the service does, and it is silent. The guard is one column.

sent must track the offered rate times the window. Sixty a second for twenty seconds is twelve hundred arrivals, and the table says 1178 — within two per cent, so the schedule held. If sent comes in well under the product, the generator could not keep up and every row at that rate is a measurement of your load generator.

Three things that cause it, in the order they bite:

The generator is sharing a core with the service. Both containers are on one machine in this lab, by design, and that is fine while the service wants a tenth of a core. It stops being fine when you raise the worker count in lesson five and the service starts wanting half of one. Watch sent at every new configuration, not once.

Ephemeral ports. Every arrival opens a fresh connection and closes it, and a closed connection sits in TIME_WAIT for a couple of minutes on most systems. At a few hundred arrivals a second the port table is the next limit along, and it shows up as error climbing rather than as anything sensible.

Scheduling granularity. The generator sleeps between arrivals, and a sleep whose requested interval is shorter than the host's timer resolution is a sleep that overshoots. This is the floor on how precisely any open-loop generator can hold a high rate, and it is why the arrival schedule is drawn from a fixed seed rather than a fixed interval.

What I would use instead of this generator for anything real: wrk2, which exists specifically because the original wrk is closed-loop and therefore cannot show you this plateau at all, or vegeta, whose attack -rate is open-loop by default and whose report separates latency from the requested rate. Both are a single binary. The reason this lab ships its own is that a hundred lines of Python you can read beats a tool whose arrival model you have to take on trust — and the arrival model is the only thing that matters here.

Check

You re-run the sweep and every row below forty-five looks the same as before, but the row at forty-eight now shows sent of 640 against an offered rate of forty-eight for twenty seconds. What do you conclude?

Choose one answer

That row is a measurement of the generator, not the service — discard it and find out what is competing for the machine before trusting anything above forty-fiveThe service refused the extra arrivals, which is why fewer were sentThe service got faster, so fewer requests were neededNothing — a third is within normal run-to-run variation for a twenty-second window

Check answer

Where this stops holding

The sweep above is trustworthy because the generator is comfortably faster than the service. That relationship is a property of this configuration, not of the lab, and lesson five breaks it on purpose.

The crossing to watch for: once the service's own processor demand plus the generator's exceeds one core on your machine, the two start trading cycles and every row becomes a measurement of the pair. In this configuration the service wants about a tenth of a core and there is no contest. At sixteen workers it wants most of one, and at that point a sweep on a laptop needs the generator moved to another machine, or the measurement needs to be read as a floor rather than a reading.

The honest check is the one above and it costs nothing: sent against the offered rate times the window, on every row, every run.

Worth remembering

• The completions column leaves the diagonal and flattens at the ceiling you predicted from the handler, within about three per cent.
• The median had already tripled at sixty-three per cent of capacity, before a single request was refused or timed out — that is the knee, and it arrives at a utilisation a dashboard renders as comfortable.
• Two plateaus exist and they differ: requests the service completes flattens and stays flat, while requests whose answer is received rises, turns over and falls.
• At a quarter above capacity the service completed more requests than in any other row and delivered half as many, because the queue had grown past the caller's two-second budget.
• The queueing model was about twice as pessimistic as the measurement, because this handler's service time barely varies and a near-constant queue waits about half as long as the exponential case at the same utilisation.
• So this lab is the best case. A handler whose cost varies moves towards the model's figure and a bursty arrival pattern moves past it.
• Read the sent column before any latency column: a row where the generator failed to keep its schedule is a measurement of the generator, and it looks exactly like a measurement of the service.
• An open-loop generator is the only kind that can show a plateau at all, which is why a closed-loop tool reports a clean curve that production never reproduces.