Part VI · Chapter 11 · Choosing · lesson 5

The limit you meet first is not the one in the documentation

14 min

Builds on Find the ceiling in a design you have never seen and One table is 400GB and everything that touches it is slow

A live-scoring feature for a tournament. At the final whistle, clients post roughly eight thousand score events a second, all for one tournament, through an API Gateway endpoint to a Lambda that writes to DynamoDB and mirrors the events into a Kinesis stream for the replay archive.

Someone has done the quota homework and raised everything that looked close. The feature throttles anyway, at about an eighth of the load, and the limit it hit appears in no quota console and in no increase request.

Every published quota, as the team wrote them down

API GATEWAY  (REST, per region, per account)
  steady state                 10,000 requests/s           adjustable
  burst                        5,000 requests               adjustable
LAMBDA  (per region, per account)
  concurrent executions        1,000 default                adjustable
  per-function scaling rate    +1,000 environments / 10 s   not adjustable
  new accounts                 start lower, raised on use   automatic
DYNAMODB
  on-demand table ceiling      40,000 WRU/s default         adjustable
  per partition, write         1,000 WRU/s                  NOT adjustable
  per partition, read          3,000 RRU/s                  NOT adjustable
  new on-demand table          serves 4,000 WRU/s immediately, roughly
                               doubles every 30 min as traffic grows
  item size                    400 KB max
KINESIS DATA STREAMS
  per shard, write             1 MB/s or 1,000 records/s    add shards to grow
  per shard, read              2 MB/s, 5 GetRecords calls/s (shared fan-out)
S3
  per partitioned prefix       3,500 PUT/POST/DELETE per s
                               5,500 GET/HEAD per s

demand at the whistle          ~8,000 events/s, sustained for ~90 s
partition key, dynamodb        tournament_id
partition key, kinesis         tournament_id
archive prefix, s3             s3://replays/2026/10/31/
lambda duration               ~40 ms per event

Four of those six services are quoted per account or per region. Two are quoted per something inside your design, and you chose what that something is.

Predict

Eight thousand events a second for one tournament. Which limit gives first, and at what rate?

Lambda concurrent executions, at a thousand concurrentAPI Gateway's steady-state rate, at ten thousand requests a secondThe DynamoDB per-partition write ceiling, at a thousand writes a second — and the single Kinesis shard, at the same figure, for the same reasonThe DynamoDB table's on-demand throughput ceiling, at forty thousand write units a second

The method, in the order that finds things

The mistake is treating a quota page as a list of ceilings. It is a list of numerators. Each one is divided by something before it reaches you, and the denominator lives in your design.

One: for every quota, write down what it divides by. Per account, per region, per table, per partition key, per shard, per prefix, per connection, per function, per queue, per message group. This single column reorders the whole list, because the per-account ones are usually generous and the ones dividing by something you chose are usually not.

Two: evaluate each quota at the most concentrated value of its denominator, not the average. The question is never "eight thousand events a second across how many partitions" — it is "how many land on the busiest one during the worst minute". For this design the answer is all of them, because the partition key is constant within a tournament. Average load is the number that makes a design look fine; peak concentration is the number that throttles it.

Three: separate the ceiling you get eventually from the ceiling you get now. Several of the figures above are warm-up behaviours rather than limits: a newly created on-demand table serves a fraction of its eventual throughput and grows over tens of minutes, and Lambda adds execution environments in steps rather than all at once. A feature whose load arrives as a step at a known kickoff time meets the warm-up ceiling, not the steady-state one, and no amount of waiting helps because the traffic does not build.

Four: find the limits nobody publishes as limits. Burst credits on older general-purpose volumes and smaller instance families are the classic: a disk or a CPU that performs well for twenty minutes and then performs at its baseline, which is sometimes a fraction of what you measured. The tell is a benchmark that was fine and a production incident at the same load, half an hour in.

Five: test the concentrated case deliberately, and read the throttle counter. Not a load test that spreads traffic evenly — a load test with one key, ramped until the provider rejects something, with the throttled-request metric on screen. Latency is a lagging and ambiguous indicator; a throttle count is the provider telling you which ceiling you found, by name.

Then record each ceiling, the denominator it divides by, where you read it, and the date. A quota you read in a console six months ago is not a quota you know.

The real ceilings for this design, ranked

1,000/s   dynamodb, one partition           hot key: tournament_id   NOT adjustable
1,000/s   kinesis, one shard                hot key: tournament_id   add shards, rekey
3,500/s   s3, one date prefix               hot prefix: date path    rekey the prefix
4,000/s   dynamodb on-demand, first 30 min  warm-up, not a limit     pre-warm
~8,000/s  lambda, 1k concurrent at 40 ms    fine, with margin        adjustable
10,000/s  api gateway steady state          fine                     adjustable
40,000/s  dynamodb table on-demand          unreachable with 1 key   adjustable

The ranking inverts the quota page. The two smallest are the two the team never looked at, because they are not in the account's quota list — they are properties of a key choice made in a schema review.

And the fix is a key change rather than a quota request. Writing to tournament_id#<shard 0-9> spreads the burst across ten partitions and ten times the ceiling, at the cost of every read fanning out across ten keys and merging. The same trick works on the Kinesis partition key and on the S3 prefix, which is why the archive path should start with something high-cardinality rather than a date. None of those three changes needs a vendor's permission; all three are expensive to retrofit, which is the argument for doing step one of the method during design rather than during an incident.

Check

You shard the key ten ways and the throttling stops. Six months later a single very large tournament throttles again, at roughly ten times the old rate. What is the general lesson?

Shard wider — thirty or a hundred ways — since the mechanism is proven and the cost is only read fan-out.A fixed shard count converts a hard ceiling into a higher hard ceiling. The ceiling is still there, it is still a multiple of a per-partition figure, and it still needs a monitored headroom number.The platform's adaptive capacity should have absorbed it, so this is a vendor problem to raise with support.The write path should not be a key-value store at all, since per-partition limits are inherent to it.

What to hand over

The output of this exercise is not a reassurance. It is four lines per constraint, and a reviewer should be able to check each one:

constraint   dynamodb write, per partition
ceiling      1,000 WRU/s per partition key, x10 shards = 10,000/s for one tournament
denominator  tournament_id#shard — most concentrated case is one tournament, 10 keys
read where   service documentation, per-partition limits section
checked      2026-10-27
alert        throttled-request count > 0 for 1 minute, and per-key write rate > 7,000/s
breaks when  a single tournament exceeds roughly 10,000 events/s, or the shard
             count is changed without rebuilding the read path

The alert line is the one teams skip and the one that pays. A quota you have computed and not instrumented is a number in a document; the throttle counter is the provider telling you the truth about your own design, for free, every minute.

Where this stops holding

All of the above is a method for finding a denominator you control and evaluating it at its worst value. It is overhead — real overhead, several hours of design review — and it stops being worth paying the moment demand is genuinely even across the dimension the quota divides by.

Concretely: if your partition key is something with high cardinality and no correlation to load, like a user id in a consumer product with no celebrities, then the table-level and account-level quotas really are your ceilings, the quota console really is the right place to read them, and dividing by a concentrated denominator gives the same answer as dividing by the average. In that regime, steps one and two of the method produce nothing and you should skip them.

Two cautions on claiming you are in that regime. First, evenness is a property of the load distribution rather than of the key's cardinality: a tenant id has high cardinality and one tenant can be ten times the next. Second, evenness expires. The design that was flat for two years becomes concentrated when one customer grows, when a feature introduces a global entity, or when a tournament final exists. The method is cheap to re-run; the retrofit is not.

As of 2026-10-31

Every figure in the two quota blocks is a vendor-published value, which means it is adjustable, revisable and in several cases region-specific. Enumerated, so each can be rechecked on its own. One: Lambda's default limit of a thousand concurrent executions per region per account, adjustable. Two: Lambda's per-function scaling rate of a thousand execution environments every ten seconds, which is the whole of the current scaling model — an earlier model granted a separate regional burst allowance of between five hundred and three thousand, and that quota no longer appears in the documentation, so a design note written against it is reasoning about a Lambda that no longer exists. New accounts also start below these figures and are raised automatically with use. Three: API Gateway's default steady-state rate of ten thousand requests a second per region with a burst allowance of five thousand requests. Four: DynamoDB's default on-demand table ceiling of forty thousand write units a second, adjustable. Five: DynamoDB's per-partition ceilings of a thousand write units and three thousand read units a second, which are not adjustable — the most important claim on the page and the one to verify first. Six: DynamoDB's on-demand warm-up, serving a lower initial throughput and roughly doubling as traffic grows; the specific initial figure and doubling interval are the shakiest numbers in the lesson. Seven: a DynamoDB maximum item size of four hundred kilobytes. Eight: Kinesis per-shard write of one megabyte a second or a thousand records a second, and per-shard read of two megabytes a second across five calls a second in shared fan-out mode. Nine: S3 request rates of three thousand five hundred mutating and five thousand five hundred read requests a second per partitioned prefix. Ten: burst-credit behaviour on older general-purpose volume types and smaller instance families, which differs by volume type and has changed with newer generations. The method itself — find the denominator, evaluate at the concentrated case, separate warm-up from ceiling, test with one key, read the throttle counter — is mechanism and does not expire.

Worth remembering

A quota page lists numerators; your design supplies the denominator, and that is where the real ceiling comes from.
For every quota, write down what it divides by: account, region, table, partition, shard, prefix, connection, function or message group.
Evaluate each quota at the most concentrated value of its denominator during the worst minute, never at the average.
A table-level ceiling is a sum across partitions and is unreachable when one key takes the load.
Per-partition limits are frequently not adjustable at any price, while the account-level ones everybody raises usually were not the constraint.
Separate the ceiling you get eventually from the one you get now: warm-up and burst behaviour bind on load that arrives as a step.
Burst credits on older volume types and smaller instances explain the benchmark that passed and the incident half an hour in.
Test with one key and read the throttled-request counter; latency is lagging and ambiguous, a throttle count names the ceiling.
Sharding a hot key multiplies the ceiling by a constant you chose — it does not remove it, so the multiple belongs in the capacity record with an alert.
Adaptive capacity is built for a sustained hot key rather than a ninety-second spike — the vendor does not document its reaction time, so plan for it not arriving rather than for how fast it does.
Record every ceiling with its denominator, where you read it, the date, and the alert that fires before you reach it.
If load is genuinely even across the dividing dimension, skip the method — but check that evenness is a property of the distribution rather than of the key's cardinality.