Chapter 8
How should web applications respond to scaling pressure?#
- Created
- Updated
Scaling changes a system in response to a specific pressure. Each response should relieve an identified limit while making its new costs and failure modes visible.
8.1 Start with the pressure#
Use the primary-pressure and bounded correctness or recovery evidence from Challenge to select a response hypothesis. The table below connects common pressures to the contracts each response adds.
| Pressure or failed assumption | Response pattern worth considering | New obligation and evidence |
|---|---|---|
| Writes and reads compete for one path | Separate the authoritative write path from derived read views | Define freshness, rebuild, and reconciliation; measure write completion and read usefulness |
| A burst contains work that need not finish synchronously | Queue or buffer bounded background work | Expose pending and terminal states, queue age, retry behavior, and worker saturation |
| A request may be repeated after an ambiguous result | Use a stable idempotency key and authoritative deduplication | Store the mutation and reusable result atomically; test a retry after a lost response |
| One tenant or domain instance consumes shared capacity | Partition, quota, or isolate work by a meaningful domain boundary | Define routing, fairness, cross-partition operations, and migration |
| Incoming work exceeds the rate at which it can be completed | Apply backpressure, admission control, rate limits, or load shedding | Make rejection explicit, bound waiting, guide retries, and preserve the essential path |
| Optional work threatens a core operation | Degrade a named capability while preserving the core | Define the degraded user state, dependency boundary, recovery trigger, and completion signal |
| History grows beyond the useful query or retention path | Add a measured index, derived summary, archival, or deletion path | Account for write cost, staleness, rebuild, retention policy, and restored-state checks |
Combine techniques only where the pressure requires them. A queue can duplicate effects without idempotency, and controlled rejection can amplify load if clients immediately retry.
8.1.1 Prepare one scaling decision#
Use the response patterns above to frame the existing before-and-after project experiment. Copy these prompts into the specification or decision record you maintain:
Observed limit, revision and supporting record:
One response to test and expected observable benefit:
Simpler alternative and why it is insufficient:
Same-workload comparison and acceptance checks:
New failure, freshness or recovery obligation:
Result, uncertainty and reversal condition:
For example, a slow workspace feed may justify testing an index that matches its filter and ordering. Inspect the query plan and repeat the declared workload; also check result ordering and the added write or storage cost. The existence of an index is not evidence of useful improvement.
Each accepted work-item creation or transition must preserve its state change and matching activity event together.
Queueing may defer derived summaries after that commit; a faster acknowledgment must not conceal a pending core operation.
For optional practice with a particular mechanism, the course workspace contains http-cache/ and persisted-retry/ with fixed checks.
Choose one when it answers your question; completing extra exercises is not an additional grading requirement.
8.2 Name the saturation boundary#
Scaling pressure becomes operationally important when it approaches a limit. The competence-envelope model introduced in the Challenge part calls the boundary condition saturation: the system can no longer absorb the disturbance with its current capabilities (Woods, 2018). Naming the saturated resource prevents "add more servers" from becoming the default response. The table below groups common limits by the evidence that reveals their approach.
| Limit category | Examples | What approaching saturation may look like |
|---|---|---|
| Physical resources | CPU, memory, disk space, storage I/O, network bandwidth | Growing wait time, swapping, failed allocations, full disks, or reduced throughput |
| Virtual resources | Connection and thread pools, file descriptors, bounded queues, quotas, rate limits | Pool wait, rejected work, 429 responses, queue growth, or quota errors |
| Dependency capacity | Databases, identity providers, third-party APIs, provisioning systems | Timeouts, blocked callers, correlated downstream errors, or delayed scaling |
| Operational capacity | On-call attention, deployment speed, diagnostic access, coordination | Slow decisions, unsafe workarounds, competing responders, or recovery blocked by normal process |
Saturation can create reinforcing feedback loops: overload causes timeouts, timeouts cause retries, and retries add load. Synchronous fan-out can overload an internal service, and blocked threads can make a CPU-based autoscaler misread demand. A resource leak may separate the causal change from the eventual failure by hours or months. These dynamics explain why a locally sensible mechanism can worsen a system-wide incident and why the existing pressure record should identify both the first limit and what happens after it is reached (Hochstein, 2026).
8.2.1 Saturation boundary lab#
The model below follows offered, admitted and completed rates through four declared stage limits. Increase demand until a boundary becomes binding, expand that limit, and observe which constraint becomes visible next. Then enable backpressure and distinguish the excess that queues from the excess explicitly rejected. Name a real observation that could confirm the limiting stage; the model's entered capacities do not establish it.
Capacity limit lab
Find the first binding limit
A steady-rate model with one visit per stage, successful work and one buffer before the pipeline. Completion is the minimum of admitted load and all declared capacities. Admission limiting rejects excess immediately; without it the buffer grows. Bars compare admitted rate with limits, not measured stage utilization. There are no latency, transient queue or production capacity predictions.
Pause & think
Locate the next capacity limit
Each request visits every stage once. Capacities are API 180, database 110, workers 150, and downstream 130 requests/s. Offered load is 150 requests/s, with no retries. Use the smallest stage capacity as the throughput ceiling. Then raise database capacity to 150 requests/s, leaving all other capacities unchanged.
Automatically checked practice. Saved answers are private to your account.
Preview the exercise context for the assistant
This prepares an editable message. Review it in the assistant before sending. Your notebook notes are not included.
Checking saved answers…
Your answers to earlier versions
Show worked answer
Initially the database limits throughput to 110 requests/s. Increasing its capacity moves the limit to the downstream stage at 130 requests/s. Offered load still exceeds capacity by 20 requests/s.
Treat the rates as a reasoning aid rather than measured capacity. The lesson is structural: removing one bottleneck changes the system's limiting factor, while retries can consume headroom before useful demand changes.
8.3 Use a formal model before choosing a technique#
Formal models help challenge an intuition about the limit. Choose the question before the equation:
| Question | Model and worked treatment |
|---|---|
| Do backlog, throughput, and average time agree? | Little's Law |
| How much speedup can the parallel part provide? | Amdahl's Law |
| How do failures across critical steps combine? | Series reliability |
| Can coordination make added concurrency slower? | Universal Scalability Law |
Appendix E collects the equations, assumptions, and limits.
Before opening the concept references, choose between Little's Law and Amdahl's Law for each question below. Explain your choice before checking the arithmetic.
Pause & think
Choose the question before the equation
You have two questions. First, a stable event pipeline completes 40 events/s with average time 0.25 s inside one declared boundary: what average work in progress is consistent with those observations? Second, a fixed development task has an 80% parallelizable share and a 20% sequential share: what is the ideal speedup with four workers and no added coordination cost? Choose a law for each question, then calculate its result.
Automatically checked practice. Saved answers are private to your account.
Preview the exercise context for the assistant
This prepares an editable message. Review it in the assistant before sending. Your notebook notes are not included.
Checking saved answers…
Your answers to earlier versions
Show worked answer
Little's Law answers the flow-consistency question: 40 × 0.25 = 10 events. Amdahl's Law answers the fixed-workload speedup question: 1 / (0.2 + 0.8/4) = 2.5. Matching averages do not identify a bottleneck or prove acceptable latency. Ideal speedup does not guarantee accepted-change throughput: integration overhead, variable tasks, and correctness can change the result. Explain which measurement or assumption you would check before recommending more workers.
What would you still need to measure before recommending more workers? Connect the answer to throughput, latency, and the assumptions of the chosen model.
The project does not require every model. Apply at least one suitable law or formal model to an application-architecture decision and one to the development workflow. Each application should identify the variables, state the assumptions, show the calculation or reasoning, connect the result to a measured or proposed limit, and explain what the model leaves out. A formula added after the design decision is not evidence that the formula guided the decision.
8.3.1 Contention beyond Amdahl's bound#
8.4 Three documented cases of web scaling pressure#
Scaling techniques become useful in combinations rather than as isolated vocabulary. The cases below begin with published production accounts from GitHub, Spotify, and Shopify. They provide observed pressures, interventions, and outcomes that a fictional scenario cannot. They are still company-authored retrospectives rather than controlled experiments: they report what each organization measured in its own system, and several changes may have occurred together. Treat their reported outcomes as evidence about those systems, not as performance promises for another application.
Read each case through the same questions:
- What did users experience?
- Which resource or boundary approached saturation?
- What evidence exposed that boundary?
- What did the organization change?
- What improved, and what new trade-off appeared?
- Which part transfers to the event-tracking project, and which part does not?
The table below previews the evidence and the transfer question in each case.
| Production case | Observed pressure | Intervention described by the source | Reported evidence | Transfer question for the tracker |
|---|---|---|---|---|
| GitHub Issues navigation | Repeated page visits still paid network and rendering costs | Persistent client cache, stale-while-revalidate, controlled preheating | More React navigations became instant, with measured cache divergence | Which repeated read can tolerate controlled staleness? |
| Spotify podcast-video ingestion | Transcoding capacity saturated and queued publication work aged | More capacity, work prioritization, earlier alerts, planned backpressure | Publication moved from minutes to hours during the incident | What does acceptance mean, and how will users see pending work? |
| Shopify flash-sale request processing | Slow requests and browser refreshes consumed application-worker time | Detect and discard requests whose clients had already disconnected | Order-of-magnitude spikes, full request queues, and 499 responses exposed waste |
Which work should be refused, cancelled, or shed before adding hosts? |
Sustainability angle
Scaling techniques have resource profiles. Caches, queues, replicas, CDNs, shards, and multi-region deployments can improve user experience, but each adds infrastructure and operational cost. Ask which pressure justifies the extra resource use. For an AI-assisted optimization, compare its development energy with its lifetime execution savings.
8.5 Orchestration maintains declared capacity#
Horizontal scaling is not merely the instruction to "run more servers". An orchestrator repeatedly compares a declared desired state with the state it observes and takes action to close the gap. A Deployment can declare a replica count and pod template, while a Service gives clients a stable route as individual pods change.
8.5.1 Orchestration control-loop lab#
The lab below runs Webernetes, a partial browser-based port of Kubernetes intended for interactive teaching. Change the desired replica count, send requests through a service, terminate a pod, and inspect a rolling update. Watch for the distinction between the operator's instruction, the temporary mismatch, and the controller action that restores the declared state. The deliberately broken release adds a second question: can the replica count converge while application requests fail? Inspect the response evidence and roll back, keeping running replicas separate from accepted application outcomes.
Browser cluster lab · Webernetes
Watch controllers restore the desired state
Debrief the mechanism and its limits
Roll out v2, send requests, then roll back and repeat. Name the desired state, the observed state, and the controller action after each experiment. Then identify what remains outside this model.
- Replicas add service instances; they do not remove database limits.
- Replacement demonstrates recovery, not uninterrupted availability.
- This simulator omits real images, resource limits, persistent volumes, and production networking.
This is a mechanism demonstration, not a capacity benchmark or a production Kubernetes environment. The browser runs compatible control logic and a simulated network, but it does not execute real container images or model persistent volumes, resource limits, autoscaling, and the full production networking stack. Ask which component noticed the mismatch, what action it took, and which scaling problem remains unsolved.
Replica orchestration cannot make every component horizontally scalable. If each application instance depends on one saturated database, holds incompatible local state, or performs the same expensive downstream work, adding pods may move or amplify the bottleneck. Treat the replica count as a response hypothesis and measure whether useful capacity or availability improves.
8.6 Latency and geography#
Geography becomes a scaling pressure when physical distance, regional failure, data residency, or a geographically distributed user base approaches an application requirement. The fastest handler still waits for network propagation and every sequential round trip consumes more of the latency budget.
Static assets and public cacheable responses can often move closer to users through a CDN or edge cache without moving the authoritative write path. Personalized data and mutations are harder: executing code at the edge does not remove the distance to a single authoritative database. Moving or replicating that authority introduces routing, failover, consistency, conflict, and residency decisions. Measure which bytes and round trips dominate before treating multi-region deployment as the inevitable final stage of scaling.
8.6.1 Partitions force an operational choice#
For a distributed read-write object in an asynchronous network, a network partition can prevent a design from guaranteeing both linearizability and an available response from every non-failing node (Gilbert & Lynch, 2002). Linearizability requires one operation order that respects real time: a read starting after a completed write must see that value or a later write. Availability requires every request to a non-failing node eventually to return a response allowed by the object's contract; it sets no latency bound.
For the tracker, suppose Alice and Bob send different start commands for the same initial work item to regions that cannot communicate. If both regions accept the transition locally, both clients receive success even though the lifecycle permits only one initial-to-active transition. Reconciliation can repair stored state, but it cannot retract the success already observed by either client. A majority-quorum design can keep one accepted history by making the isolated minority delay or reject the request.
Choose which operations may be stale, rejected, delayed, merged, compensated, or repaired without violating the domain. Appendix C's CAP proof and partition lab develop the reasoning and compare the two policies.
In some project themes, geography may matter very little. In others, a platform serving users or devices in different regions may need to consider latency, data residency, or regional failure. Justify multi-region architecture against the project's latency, availability, and operating requirements.
8.6.2 Latency budget lab#
The demonstration below treats latency as a budget. Some of the budget is bounded by physics: a signal cannot cross distance instantly, and real networks add routing and protocol overhead. Some of the budget is architectural: every sequential database round trip adds another wait. Choose distances independently of the server and database execution times; each timing is a stated hypothetical input unless you supply a sourced observation. Distance alone does not determine a primary/replica design or a hardware class.
Latency budget lab
Count distance and database waits
The point is not to memorize exact numbers. Real latency depends on providers, routing, congestion, TLS, database load, and query plans. The useful lesson is the shape of the problem: distance creates a lower bound, and repeated sequential database access multiplies that bound.
8.7 Trade-offs#
For each mechanism, explain what becomes simpler and what becomes harder:
- Caching reduces repeated computation but adds freshness and invalidation decisions.
- Queues shorten synchronous work but add backlog, retries, idempotency, and delayed completion.
- Partitioning reduces shared-resource pressure but adds routing, migration, and cross-partition queries.
The “waterbed” metaphor describes this movement of complexity (Randal et al., 2003); it is a review question, not a law that complexity must remain constant. Ask what pressure justifies the mechanism, why a simpler design is insufficient, and which new failure would make you reconsider.
Sustainability angle
Retaining raw data, precomputing summaries, and reserving peak capacity consume resources even when they protect a useful goal. State that goal and measure the recurring cost.
8.8 Implement the smallest justified response#
A target diagram shows that a group can reason about a possible architecture. It does not yet show that the response works in the selected application, preserves required behavior, or moves the expected limit. Complete one bounded implementation experiment before finalizing the target design.
8.8.1 Field note — Thesis Journey Tracker#
Thesis Journey Tracker serves only two people, so runtime scale was not the problem. The goal was to deliver the project quickly while keeping it easy to maintain and adjust as the supervision workflow changed. I deliberately chose a lightweight, server-rendered interface instead of a conventional single-page application. The first agent-generated version rendered complete pages, but less than two hours later the agent introduced a narrow HTML-fragment endpoint for the student panel. A small browser script replaced that panel without a full refresh and retained ordinary navigation as a fallback. This provided responsiveness where it mattered without requiring a client-side application architecture for the whole system.
Select implementation A, B, or C and one pressure for which the group has runtime evidence or a credibly projected limit that the available environment can exercise. Preserve the accepted revision and its workload result, then implement the smallest response that addresses that limit. Suitable experiments include a measured query index, a bounded cache, an idempotency boundary, admission control, a queue with an explicit pending state, tenant isolation, or another technique justified by the evidence. Distributed infrastructure is not required.
Keep the workload, fixture, correctness checks, and measurement procedure unchanged where the response is intended to affect them. After the change:
- rerun the common acceptance scenarios and inspect for behavior displaced by the response;
- rerun the selected pressure and report every repetition rather than only the best result;
- compare latency, throughput, errors, correctness, and the chosen resource-efficiency proxy before and after;
- exercise one failure or recovery obligation introduced by the response, such as stale data, a duplicate retry, a full queue, a rejected request, or an unavailable worker;
- state whether the result supported, challenged, or left the original hypothesis unresolved.
An unsuccessful intervention remains useful evidence when the baseline, change, and failure are preserved. Do not replace a negative result silently with a more elaborate design. The target architecture can retain the response, revise it, or reject it, but it should distinguish this implemented evidence from its remaining proposals.
8.9 Prepare to be surprised#
Design can expand the competence envelope, but finite systems always retain another boundary. Graceful degradation keeps selected functions useful when capacity is constrained. Graceful extensibility is a complementary idea: when an unanticipated condition exceeds the designed envelope, the sociotechnical system can adapt its capabilities and recover (Woods, 2018).
That ability requires optionality prepared before the incident:
- dynamic configuration and feature flags that expose understood controls;
- fast, practiced paths to restart, scale, shed work, roll back, or fix forward;
- diagnostic access and telemetry that remain useful under pressure;
- people who understand the system and have authority to act;
- exercises that keep rarely used recovery paths discoverable and safe.
More controls are not automatically better. An undocumented switch or unsafe manual path creates another failure mode. The goal is to balance work that prevents known failures with work that improves the ability to respond when the design encounters a surprise.
8.10 Project artifact#
Connect these three elements in the existing Compare evidence index:
- Three-way comparison. Compare the shared primary pressure across implementations A, B, and C using the same workload and procedure where practical. Link the measured results and B's bounded correctness or recovery case from the existing pressure record. Further scenarios are optional; distinguish their measurements and projections from the required evidence, and make missing measurements explicit.
- Implemented response. In one selected implementation, preserve the before-and-after revisions, unchanged workload and acceptance checks, measured result, new failure or recovery obligation, and conclusion. A proposed improvement does not replace this experiment; an unsuccessful measured response remains useful evidence.
- Target design. Justify the selected responses and explain why simpler alternatives are insufficient.
Across the package:
- connect at least one pressure and proposed response to a documented production account;
- separate what the source reports, your interpretation, and the transfer you propose for the project; do not assume that the source's scale, result, or trade-off carries over;
- apply one formal model to an application decision and one to the development workflow.
Use one trade-off table row for each response in the target design. Record:
- the pressure and approaching limit;
- the technique, intended benefit, cost, and new failure mode;
- the operational signal and at least one prepared recovery action;
- where complexity moved, such as into clients, data freshness, background operations, routing, migration, or incident response;
- why a simpler alternative was not enough.
If the target design contains asynchronous multi-step work that can perform an external side effect, include one crash-after-side-effect audit. Record:
- the last durable coordination fact and the external effect that may already have occurred;
- the timeout and retry rule;
- the retry-safety or deduplication mechanism, its owner, and any stable key it uses;
- the terminal-state behavior and evidence available to an operator.
If no component at the effect boundary can deduplicate, state that duplicate delivery remains possible.
8.11 Summary#
Choose a scaling response from the pressure and first binding limit. Use a suitable model to bound expectations, implement the smallest justified change, and repeat the workload through the real deployment. Judge both the improvement and the consistency, recovery, resource, and ownership costs it introduces.
8.12 References#
- Woods, D. D. (2018). The Theory of Graceful Extensibility: Basic Rules that Govern Adaptive Systems. Environment Systems and Decisions, 38(4), 433–457. https://doi.org/10.1007/s10669-018-9708-3
- Hochstein, L. (2026, July). Saturation: How Your Software Will Fail at Scale. Software Should Work conference presentation. https://www.youtube.com/watch?v=PHYCRubnmSM
- Randal, A., Sugalski, D., & Tötsch, L. (2003). Perl 6 Essentials. O’Reilly Media. https://www.oreilly.com/library/view/perl-6-essentials/0596004990/
- Gilbert, S., & Lynch, N. A. (2002). Brewer’s Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services. SIGACT News, 33(2), 51–59. https://doi.org/10.1145/564585.564601