Part V · Chapter 8 · Running It · lesson 2
Read the handler, then say what it will do per second
13 min
Builds on How many requests can be inside your service at once
Part V · Chapter 8 · Running It · lesson 2
13 min
Builds on How many requests can be inside your service at once
The service from the last lesson is on your disk and not running. Before you start it, read the four functions that decide its throughput and write a number down. Twenty-odd lines, no framework, nothing hidden — and the commitment is the point. A prediction you made and missed teaches more than the measurement on its own ever will.
def spin(ms):
"""Occupy the processor for ms milliseconds. No sleeping, no waiting."""
end = time.perf_counter() + ms / 1000.0
while time.perf_counter() < end:
pass
def serve_work(conn):
try:
path_of(conn)
spin(SPIN_MS) # 2 milliseconds, holds the processor
time.sleep(WAIT_MS / 1000.0) # 18 milliseconds, gives it up
respond(conn, "200 OK", "done\n")
bump("completed")
except OSError:
bump("abandoned")
finally:
shut(conn)
def worker():
while True:
conn = work.get() # WORKERS threads run this loop
bump("in_service")
try:
serve_work(conn)
finally:
bump("in_service", -1)
work.task_done()
def door(conn):
depth = work.qsize()
if LIMIT > 0 and depth >= LIMIT: # LIMIT is 0, so never
refuse(conn)
return
bump("accepted")
work.put(conn) # unbounded queue, accept everything
Predict
Little's law
Hypothetical model, not a forecast
Ties arrival rate, latency and concurrency together, solving for whichever of the three is left implied.
It assumes
It ignores
Leave the latency at the measured median and raise the requests in flight from one to four — the change you will actually make in lesson five. Read off the arrival rate the identity now allows.
Average time one request spends in the system, entry to exit.
Not there yet — keep moving the controls.
With one request in flight and the measured median from the acceptance check, the identity allows a little under fifty a second. That is the whole prediction, and it took one division.
Note what is not in it. No distribution, no utilisation, no queueing correction. Little's law is an identity about averages: requests in flight equals arrival rate times time in flight, and it holds for any arrival pattern whatsoever. It tells you the ceiling and tells you nothing about the latency on the way to it, which is the next lesson's job.
Two commands. The first brings the service up and leaves it up; the second offers load at two rates well below the ceiling, which is how you confirm the measurement agrees with the arithmetic before you push it anywhere interesting.
docker compose up -d service
docker compose run --rm load --rate 10,20 --seconds 20
offered sent ok refused gaveup error ok_per_sec p50_ms p99_ms done lost queued
10 211 211 0 0 0 10.6 21.3 62.7 211 0 2
20 410 410 0 0 0 20.5 21.5 82.1 410 0 3
Read the columns in this order, every time, for the rest of the track.
sent against offered. Twenty times twenty seconds is four hundred, and
the generator sent four hundred and ten. The generator is keeping its
schedule, so the load axis means what it says. This is the first thing that
breaks when you push the rate up, and the first thing to check when a result
surprises you.ok_per_sec against offered. Ten offered, ten completed. Twenty
offered, twenty completed. Below the ceiling the service hands back exactly
what it is given, which is the only regime in which throughput is a measure
of demand rather than of capacity.p50_ms against the declared cost. Twenty-one against a declared twenty.
Nothing is queueing yet.queued. The deepest the queue got during the window. Two and three.
Essentially at rest.Then push to the ceiling and compare:
docker compose run --rm load --rate 48 --seconds 20
offered sent ok refused gaveup error ok_per_sec p50_ms p99_ms done lost queued
48 931 931 0 0 0 46.5 122.6 387.9 931 0 18
Forty-six and a half completed a second against a predicted forty-eight. The prediction was good to within about three per cent, which is a better result than most capacity arithmetic gets, and it is only that good because the cost of this handler was declared rather than discovered.
The median, though, has gone from twenty-one to a hundred and twenty-three. The throughput prediction was right and said nothing at all about that, which is the honest limit of an identity.
Real handlers do not announce their cost. Three ways to get the two numbers out of one that does not, in the order I would try them.
Time one request with nothing else running, then time its processor use
separately. The wall-clock figure gives you the worker ceiling and the
processor figure gives you the other one. On Linux, /usr/bin/time -v reports
user and system processor seconds beside elapsed time; inside a container,
docker stats --no-stream gives you processor share while a known rate is
offered. Python's own time.process_time() against time.perf_counter() around
the handler is the two-line version and the one I actually use first, because
it needs no tooling and the ratio between them is the whole answer.
Count the awaits, not the lines. In an async handler, every await on a
socket or a database is time the worker gives back and every stretch between
them is time it holds. asyncio's debug mode logs any callback that holds the
loop longer than a threshold you set, which finds the spin you did not know
about — a JSON parse of a large payload, a synchronous driver, a regular
expression over a long string.
Read the pool sizes and work backwards. gunicorn's --workers times
--threads, uvicorn's single loop, Puma's threads setting: these are the
concurrency in Little's law, and they are usually written down somewhere even
when the service time is not. Concurrency divided by your target latency is the
arrival rate the configuration currently allows, and comparing that to peak
traffic is a five-minute capacity review.
Where this gets you wrong: a handler whose cost depends on the request. This lab's handler charges every request the same twenty milliseconds, so one median describes all of it. A search endpoint whose cost scales with result count, or an export endpoint that streams a variable number of rows, has no single service time, and a ceiling computed from the median of a mixed population will be comfortably wrong in whichever direction the expensive requests happen to arrive.
Check
The arithmetic above survives a change of machine, a change of worker count and a change of arrival pattern. It does not survive a handler whose cost depends on its input.
The crossing to watch for: once the slowest decile of requests costs more than about twice the median, a single service time stops describing the workload and the ceiling you compute from the median is a ceiling for a traffic mix you may not get. At that point the honest move is to split the measurement — one service time per endpoint, or per request-size bucket — and compute a ceiling per class. One number for a bimodal workload is not a conservative estimate; it is an estimate whose error changes sign with the traffic.