Chapter 8
How should web applications respond to scaling pressure?#
- Created
- Updated
Scaling changes a system in response to a specific pressure. Each response should relieve an identified limit while making its new costs and failure modes visible.
8.1 Start with the pressure#
Use the pressure map from the Reality chapter to select a response hypothesis. The table below connects common pressures to the contracts each response adds.
| Pressure or failed assumption | Response pattern worth considering | New obligation and evidence |
|---|---|---|
| Writes and reads compete for one path | Separate the authoritative write path from derived read views | Define freshness, rebuild, and reconciliation; measure write completion and read usefulness |
| A burst contains work that need not finish synchronously | Queue or buffer bounded background work | Expose pending and terminal states, queue age, retry behavior, and worker saturation |
| A request may be repeated after an ambiguous result | Use a stable idempotency key and authoritative deduplication | Store the mutation and reusable result atomically; test a retry after a lost response |
| One tenant or domain instance consumes shared capacity | Partition, quota, or isolate work by a meaningful domain boundary | Define routing, fairness, cross-partition operations, and migration |
| Incoming work exceeds the rate at which it can be completed | Apply backpressure, admission control, rate limits, or load shedding | Make rejection explicit, bound waiting, guide retries, and preserve the essential path |
| Optional work threatens a core operation | Degrade a named capability while preserving the core | Define the degraded user state, dependency boundary, recovery trigger, and completion signal |
| History grows beyond the useful query or retention path | Add a measured index, derived summary, archival, or deletion path | Account for write cost, staleness, rebuild, retention policy, and restored-state checks |
Combine techniques only where the pressure requires them. A queue can duplicate effects without idempotency, and controlled rejection can amplify load if clients immediately retry.
8.1.1 Scaling pressure lab#
The demonstration below starts from a concrete event-tracking pressure: event writes spike, dashboard reads get slower, and the latest-events query has to stay useful without pretending the system needs every scaling technique at once.
Compare scaling lab
Choose scaling moves for one pressure
The important part is the exported trade-off table. Use it to explain why a chosen technique fits the pressure, what it costs, what failure mode it introduces, and which signal would show whether the design is working.
8.2 Name the saturation boundary#
Scaling pressure becomes operationally important when it approaches a limit. The competence-envelope model introduced in the Challenge part calls the boundary condition saturation: the system can no longer absorb the disturbance with its current capabilities (Woods, 2018). Naming the saturated resource prevents "add more servers" from becoming the default response. The table below groups common limits by the evidence that reveals their approach.
| Limit category | Examples | What approaching saturation may look like |
|---|---|---|
| Physical resources | CPU, memory, disk space, storage I/O, network bandwidth | Growing wait time, swapping, failed allocations, full disks, or reduced throughput |
| Virtual resources | Connection and thread pools, file descriptors, bounded queues, quotas, rate limits | Pool wait, rejected work, 429 responses, queue growth, or quota errors |
| Dependency capacity | Databases, identity providers, third-party APIs, provisioning systems | Timeouts, blocked callers, correlated downstream errors, or delayed scaling |
| Operational capacity | On-call attention, deployment speed, diagnostic access, coordination | Slow decisions, unsafe workarounds, competing responders, or recovery blocked by normal process |
Saturation also creates feedback loops. Retries can amplify load, synchronous fan-out can overload an internal service, and blocked threads can make a CPU-based autoscaler misread demand. A resource leak may separate the causal change from the eventual failure by hours or months. These dynamics explain why a locally sensible mechanism can worsen a system-wide incident and why the pressure map must identify both the first limit and what happens after it is reached (Hochstein, 2026).
8.2.1 Saturation boundary lab#
The model below compares one effective event rate with four declared stage limits. Increase demand until a boundary becomes binding, expand that limit, and observe which constraint becomes visible next. Then enable backpressure and compare hidden backlog growth with explicit rejection.
Capacity envelope
Find the first binding limit
This is a teaching model, not a capacity benchmark. It compares one effective demand rate with declared stage limits so the movement of a bottleneck is visible.
Pause & think
Locate the next capacity limit
Each request visits every stage once. Capacities are API 180, database 110, workers 150, and downstream 130 requests/s. Offered load is 150 requests/s, with no retries. Use the smallest stage capacity as the throughput ceiling. Then raise database capacity to 150 requests/s, leaving all other capacities unchanged.
Automatically checked practice. Saved answers are private to your account.
Checking saved answers…
Your answers to earlier versions
Show worked answer
Initially the database limits throughput to 110 requests/s. Increasing its capacity moves the limit to the downstream stage at 130 requests/s. Offered load still exceeds capacity by 20 requests/s.
Treat the rates as a reasoning aid rather than measured capacity. The lesson is structural: removing one bottleneck changes the system's limiting factor, while retries can consume headroom before useful demand changes.
8.3 Use a formal model before choosing a technique#
Formal models help challenge an intuition about the limit. Choose the question before the equation:
| Question | Model and worked treatment |
|---|---|
| Do backlog, throughput, and average time agree? | Little's Law |
| How much speedup can the parallel part provide? | Amdahl's Law |
| How do failures across critical steps combine? | Series reliability |
| Can coordination make added concurrency slower? | Universal Scalability Law |
Appendix E collects the equations, assumptions, and limits.
The project does not require every model. Apply at least one suitable law or formal model to an application-architecture decision and one to the development workflow. Each application should identify the variables, state the assumptions, show the calculation or reasoning, connect the result to a measured or proposed limit, and explain what the model leaves out. A formula added after the design decision is not evidence that the formula guided the decision.
8.3.1 Contention beyond Amdahl's bound#
8.4 Three documented cases of web scaling pressure#
Scaling techniques become useful in combinations rather than as isolated vocabulary. The cases below begin with published production accounts from GitHub, Spotify, and Shopify. They provide observed pressures, interventions, and outcomes that a fictional scenario cannot. They are still company-authored retrospectives rather than controlled experiments: they report what each organization measured in its own system, and several changes may have occurred together. Treat their reported outcomes as evidence about those systems, not as performance promises for another application.
Read each case through the same questions:
- What did users experience?
- Which resource or boundary approached saturation?
- What evidence exposed that boundary?
- What did the organization change?
- What improved, and what new trade-off appeared?
- Which part transfers to the event-tracking project, and which part does not?
The table below previews the evidence and the transfer question in each case.
| Production case | Observed pressure | Intervention described by the source | Reported evidence | Transfer question for the tracker |
|---|---|---|---|---|
| GitHub Issues navigation | Repeated page visits still paid network and rendering costs | Persistent client cache, stale-while-revalidate, controlled preheating | More React navigations became instant, with measured cache divergence | Which repeated read can tolerate controlled staleness? |
| Spotify podcast-video ingestion | Transcoding capacity saturated and queued publication work aged | More capacity, work prioritization, earlier alerts, planned backpressure | Publication moved from minutes to hours during the incident | What does acceptance mean, and how will users see pending work? |
| Shopify flash-sale request processing | Slow requests and browser refreshes consumed application-worker time | Detect and discard requests whose clients had already disconnected | Order-of-magnitude spikes, full request queues, and 499 responses exposed waste |
Which work should be refused, cancelled, or shed before adding hosts? |
Sustainability angle
Scaling techniques have resource profiles. Caches, queues, replicas, CDNs, shards, and multi-region deployments can improve user experience, but each adds infrastructure and operational cost. Ask which pressure justifies the extra resource use.
8.5 Orchestration maintains declared capacity#
Horizontal scaling is not merely the instruction to "run more servers". An orchestrator repeatedly compares a declared desired state with the state it observes and takes action to close the gap. A Deployment can declare a replica count and pod template, while a Service gives clients a stable route as individual pods change.
8.5.1 Orchestration control-loop lab#
The lab below runs Webernetes, a partial browser-based port of Kubernetes intended for interactive teaching. Change the desired replica count, send requests through a service, terminate a pod, and perform a rolling update. Watch for the distinction between the operator's instruction, the temporary mismatch, and the controller action that restores the declared state.
Browser cluster lab · Webernetes
Watch the control loop close
Debrief the mechanism and its limits
Name the desired state, the observed state, and the controller action after each experiment. Then identify what remains outside this model.
- Replicas add service instances; they do not remove database limits.
- Replacement demonstrates recovery, not uninterrupted availability.
- This simulator omits real images, resource limits, persistent volumes, and production networking.
This is a mechanism demonstration, not a capacity benchmark or a production Kubernetes environment. The browser runs compatible control logic and a simulated network, but it does not execute real container images or model persistent volumes, resource limits, autoscaling, and the full production networking stack. Ask which component noticed the mismatch, what action it took, and which scaling problem remains unsolved.
Replica orchestration cannot make every component horizontally scalable. If each application instance depends on one saturated database, holds incompatible local state, or performs the same expensive downstream work, adding pods may move or amplify the bottleneck. Treat the replica count as a response hypothesis and measure whether useful capacity or availability improves.
8.6 Latency and geography#
Geography becomes a scaling pressure when physical distance, regional failure, data residency, or a geographically distributed user base approaches an application requirement. The fastest handler still waits for network propagation and every sequential round trip consumes more of the latency budget.
Static assets and public cacheable responses can often move closer to users through a CDN or edge cache without moving the authoritative write path. Personalized data and mutations are harder: executing code at the edge does not remove the distance to a single authoritative database. Moving or replicating that authority introduces routing, failover, consistency, conflict, and residency decisions. Measure which bytes and round trips dominate before treating multi-region deployment as the inevitable final stage of scaling.
8.6.1 Partitions force an operational choice#
The CAP result is narrower than a general slogan about balancing consistency and availability. For a distributed read-write object in an asynchronous network, a network partition can prevent a design from guaranteeing both atomic consistency and an available response from every non-failing node (Gilbert & Lynch, 2002). Here atomic consistency means operations appear in one current order, while availability means every request to a non-failing node eventually receives a non-error response. The partition is the condition that makes those guarantees collide; it is not a feature that an application can disable after communication has already failed.
A system that requires a majority quorum may preserve one history by rejecting or delaying a request that reaches an isolated minority. A system that accepts writes on both sides may keep responding, but the replicas can diverge and later need domain-specific reconciliation. Neither label chooses the right policy on its own. The decision depends on which operations may be stale, rejected, delayed, merged, compensated, or repaired without violating the domain.
CAP partition labSplit one region from a two-region majority and apply the same two work-item transitions under both policies. Inspect the replies before healing the network: reconciliation can change stored state, but it cannot retract a success that a user already observed.
Distributed data fault table
Decide what happens across a partition
Helsinki becomes isolated from the Frankfurt–Dublin majority. Alice and Bob then try to start the same work item from opposite sides of the split.
The model holds workload, topology, and failure constant so the policy is visible. Real systems define quorums, timeouts, leases, conflict resolution, and consistency guarantees more precisely.
The lab models one write invariant and a fixed topology. Real systems need explicit definitions for reads as well as writes, failure detection and timeouts, quorum membership, leases, conflict resolution, and the consistency guarantee exposed to clients.
In some project themes, geography may matter very little. In others, a platform serving users or devices in different regions may need to consider latency, data residency, or regional failure. Justify multi-region architecture against the project's latency, availability, and operating requirements.
8.6.2 Latency budget lab#
The demonstration below treats latency as a budget. Some of the budget is bounded by physics: a signal cannot cross distance instantly, and real networks add routing and protocol overhead. Some of the budget is architectural: every sequential database round trip adds another wait.
Latency budget lab
Count distance and database waits
The point is not to memorize exact numbers. Real latency depends on providers, routing, congestion, TLS, database load, and query plans. The useful lesson is the shape of the problem: distance creates a lower bound, and repeated sequential database access multiplies that bound.
8.7 Trade-offs#
For each mechanism, explain what becomes simpler and what becomes harder:
- Caching reduces repeated computation but adds freshness and invalidation decisions.
- Queues shorten synchronous work but add backlog, retries, idempotency, and delayed completion.
- Partitioning reduces shared-resource pressure but adds routing, migration, and cross-partition queries.
The “waterbed” metaphor describes this movement of complexity (Randal et al., 2003); it is a review question, not a law that complexity must remain constant. Ask what pressure justifies the mechanism, why a simpler design is insufficient, and which new failure would make you reconsider.
Sustainability angle
Retaining raw data, precomputing summaries, and reserving peak capacity consume resources even when they protect a useful goal. State that goal and measure the recurring cost.
8.8 Implement the smallest justified response#
A target diagram shows that a group can reason about a possible architecture. It does not yet show that the response works in the selected application, preserves required behavior, or moves the expected limit. Complete one bounded implementation experiment before finalizing the target design.
8.8.1 Field note — Thesis Journey Tracker#
Thesis Journey Tracker serves only two people, so runtime scale was not the problem. The goal was to deliver the project quickly while keeping it easy to maintain and adjust as the supervision workflow changed. I deliberately chose a lightweight, server-rendered interface instead of a conventional single-page application. The first agent-generated version rendered complete pages, but less than two hours later the agent introduced a narrow HTML-fragment endpoint for the student panel. A small browser script replaced that panel without a full refresh and retained ordinary navigation as a fallback. This provided responsiveness where it mattered without requiring a client-side application architecture for the whole system.
Select implementation A, B, or C and one pressure for which the group has runtime evidence or a credibly projected limit that the available environment can exercise. Preserve the accepted revision and its workload result, then implement the smallest response that addresses that limit. Suitable experiments include a measured query index, a bounded cache, an idempotency boundary, admission control, a queue with an explicit pending state, tenant isolation, or another technique justified by the evidence. Distributed infrastructure is not required.
Keep the workload, fixture, correctness checks, and measurement procedure unchanged where the response is intended to affect them. After the change:
- rerun the common acceptance scenarios and inspect for behavior displaced by the response;
- rerun the selected pressure and report every repetition rather than only the best result;
- compare latency, throughput, errors, correctness, and the chosen resource-efficiency proxy before and after;
- exercise one failure or recovery obligation introduced by the response, such as stale data, a duplicate retry, a full queue, a rejected request, or an unavailable worker;
- state whether the result supported, challenged, or left the original hypothesis unresolved.
An unsuccessful intervention remains useful evidence when the baseline, change, and failure are preserved. Do not replace a negative result silently with a more elaborate design. The target architecture can retain the response, revise it, or reject it, but it should distinguish this implemented evidence from its remaining proposals.
8.9 Prepare to be surprised#
Design can expand the competence envelope, but finite systems always retain another boundary. Graceful degradation keeps selected functions useful when capacity is constrained. Graceful extensibility is a complementary idea: when an unanticipated condition exceeds the designed envelope, the sociotechnical system can adapt its capabilities and recover (Woods, 2018).
That ability requires optionality prepared before the incident:
- dynamic configuration and feature flags that expose understood controls;
- fast, practiced paths to restart, scale, shed work, roll back, or fix forward;
- diagnostic access and telemetry that remain useful under pressure;
- people who understand the system and have authority to act;
- exercises that keep rarely used recovery paths discoverable and safe.
More controls are not automatically better. An undocumented switch or unsafe manual path creates another failure mode. The goal is to balance work that prevents known failures with work that improves the ability to respond when the design encounters a surprise.
8.10 Project artifact#
Produce three connected deliverables:
- Three-way comparison. Apply the same pressure set to implementations A, B, and C. Mark each result as observed in running code or proposed.
- Implemented response. Preserve the before-and-after revisions, unchanged workload and acceptance checks, observed result, new failure or recovery obligation, and conclusion.
- Target design. Justify the selected responses and explain why simpler alternatives are insufficient.
Across the package:
- connect at least one pressure and proposed response to a documented production account;
- separate what the source reports, your interpretation, and the transfer you propose for the project; do not assume that the source's scale, result, or trade-off carries over;
- apply one formal model to an application decision and one to the development workflow.
Use one trade-off table row for each response in the target design. Record:
- the pressure and approaching limit;
- the technique, intended benefit, cost, and new failure mode;
- the operational signal and at least one prepared recovery action;
- where complexity moved, such as into clients, data freshness, background operations, routing, migration, or incident response;
- why a simpler alternative was not enough.
If the target design contains asynchronous multi-step work that can perform an external side effect, include one crash-after-side-effect audit. Record:
- the last durable coordination fact and the external effect that may already have occurred;
- the timeout and retry rule;
- the retry-safety or deduplication mechanism, its owner, and any stable key it uses;
- the terminal-state behavior and evidence available to an operator.
If no component at the effect boundary can deduplicate, state that duplicate delivery remains possible.
8.11 Summary#
Choose a scaling response from the pressure and first binding limit. Use a suitable model to bound expectations, implement the smallest justified change, and repeat the workload through the real deployment. Judge both the improvement and the consistency, recovery, resource, and ownership costs it introduces.
8.12 References#
- Woods, D. D. (2018). The Theory of Graceful Extensibility: Basic Rules that Govern Adaptive Systems. Environment Systems and Decisions, 38(4), 433–457. https://doi.org/10.1007/s10669-018-9708-3
- Hochstein, L. (2026, July). Saturation: How Your Software Will Fail at Scale. Software Should Work conference presentation. https://www.youtube.com/watch?v=PHYCRubnmSM
- Randal, A., Sugalski, D., & Tötsch, L. (2003). Perl 6 Essentials. O’Reilly Media. https://www.oreilly.com/library/view/perl-6-essentials/0596004990/
- Gilbert, S., & Lynch, N. A. (2002). Brewer’s Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services. SIGACT News, 33(2), 51–59. https://doi.org/10.1145/564585.564601