Part II · Chapter 2 · One Service · lesson 4

It runs fine on your laptop and badly in production

12 min

Builds on It works for you and falls over for a thousand people

The weekly report endpoint returns in well under a tenth of a second on your laptop. In production the same code, same commit, takes about two seconds at the median and about nine at the ninety-fifth percentile.

Nothing is on fire. Nobody can reproduce it. Someone has already suggested a bigger database instance.

Before any of that: which difference between the two machines is doing this?

Predict

Same code, same commit. Fast on a laptop, about twenty times slower in production. Which difference is most likely responsible?

Production has far more rows, so a query that scanned nothing now scans everythingProduction serves many users at once, so everything is queueingThe application and the database are on different machines nowThe production container has a CPU limit and is being throttled

Six differences between a laptop and production, in the order they bite.

Row count. Hundreds locally, tens of millions in production. A plan that was free becomes the response.

Concurrency. Your laptop has one user. Pools, worker slots and locks are all invisible at one.

Network distance. Loopback locally, a cross-zone hop in production. Each round trip costs about a millisecond, so a chatty handler multiplies.

Cache temperature. Your whole dev dataset fits in memory. In production the working set may not, and then reads hit disk.

Resource limits. Container CPU quotas, memory limits and burstable credit balances. Your laptop has none of them, and a quota throttles in a way that looks like slowness rather than load.

Neighbours. Other containers on the node, other tenants, a sidecar proxy.

Only the fifth and sixth are about the hardware being smaller.

What would you do

A Python service, one endpoint, /reports/weekly. Locally it is instant. In production the median is about two seconds and the ninety-fifth percentile is about nine. The endpoint is called from a dashboard that perhaps forty people have open at once during the working day.

Production is a container with two vCPUs and a managed Postgres in a different availability zone. The table the report reads has roughly forty million rows; your development copy has about three hundred. There is no tracing, and the only dashboards are CPU and memory, both of which look unremarkable.

Optimising for: Identifying the cause this week, with evidence somebody else could check, and without changing production before you know what you are changing.
You cannot know: Whether the report query uses an index in production, or scans the table
How many separate queries the handler issues per request — nobody has counted
Whether the container is being CPU-throttled during the working day
Whether the production working set still fits in the database's memory
Whether the median or the tail is the thing users are actually complaining about

Restore a sanitised production snapshot into a local Postgres of the same major version, re-run the endpoint against production-sized data, and read the plan for every statement it issues.Raise the replica count and the connection pool size, and see whether the median improves.Move the database to a larger instance with more memory and faster storage.Put the endpoint's response in Redis with a short expiry and move on.

What I would actually run, in order

One query, before anything else. pg_stat_statements ordered by total_exec_time descending gives you the statements your production database actually spends its life on, ranked, with call counts and mean times. It ships with Postgres, needs the extension created and the library preloaded, and it answers "which query" and "how many times per request" in one look. If a statement you expected once per request has a call count forty times the request count, you have found an N+1 without instrumenting anything.

Then the plan. EXPLAIN (ANALYZE, BUFFERS) on that statement, in production or against the restored snapshot. You are looking for a Seq Scan with a large "Rows Removed by Filter", or an index that exists and is not chosen. auto_explain with a minimum duration logs the real plans of real slow requests, which beats reasoning about the plan of a query you typed by hand.

Rule out throttling with a counter, not a feeling. In the container's cgroup, nr_throttled in cpu.stat is non-zero or it is not; in Prometheus, container_cpu_cfs_throttled_periods_total. This takes a minute and removes the one cause whose symptom is indistinguishable from every other cause.

Count the queries locally. In Django, the debug toolbar or len(connection.queries); in SQLAlchemy, an after_cursor_execute event with a counter. Twelve queries per request on a loopback interface is invisible and across a zone boundary it is a measurable fraction of the response.

The reason this order works is that each step is cheap and each one eliminates a class. The expensive move — resize, rewrite, add a cache — comes after the class is known, not instead of knowing it.

Check

You restore the production-sized snapshot locally and the endpoint is still fast. What have you learned?

Nothing useful — the reproduction failedThat it is concurrency, the network hop, or a resource limit — the three a laptop cannot show youThat the snapshot is not representative, so the test was invalidThat the problem is the managed database service itself

When the laptop is the better machine

The condition under which all of the above inverts: when the production machine is genuinely smaller than the laptop.

A current laptop with ten or so performance cores and local flash storage reading at several gigabytes a second is strictly faster than a container with two vCPUs on network-attached storage with a provisioned IOPS ceiling. When that is the situation, you are not looking for a difference in behaviour, you are looking at a difference in hardware, and the six-item list is the wrong list.

Three things to check, in this order: the instance type and whether it is a burstable class, the credit balance if it is, and the storage IOPS and throughput limits against what the workload asks for. A burstable instance that has exhausted its credits is throttled to a fraction of a core and no profiler will ever tell you that.

And then the answer is often to buy a larger machine, which is the cheapest fix in this entire subject and the one engineers are most reluctant to reach for. If the service needs an hour of work and fifty dollars a month, prefer the fifty dollars.

Worth remembering

Six differences between a laptop and production: row count, concurrency, network distance, cache temperature, resource limits, and neighbours — and only the last two are about smaller hardware.
Row count is the most common cause by a wide margin, because a sequential scan over a few hundred development rows is free and so the missing index never gets written.
A local reproduction against a production-sized snapshot tests three of the six causes; failing to reproduce is informative rather than wasted.
pg_stat_statements ordered by total execution time answers 'which query' and 'how many per request' without any tracing stack.
CPU throttling looks like latency rather than load — check the throttled-periods counter rather than the CPU graph.
A cache or a bigger instance will mask the symptom; both are legitimate holding actions with a stated expiry and neither is a diagnosis.
When the production box is genuinely smaller than the laptop, check instance class, burst credits and storage IOPS first — and consider that buying a bigger machine is the cheapest fix available.