Chapter 6
Which web application assumptions fail under pressure?#
- Created
- Updated
A web application that works locally has shown that one implementation can satisfy one workload in one controlled environment. Production changes the environment: browser applications and integrations retry, networks delay or lose responses, live data accumulates, shared dependencies slow down, deployments change running systems, and operators have to recover without discarding authoritative state.
Production failures expose architectural assumptions that do not hold under actual operating conditions. Reconstruct the failure chain, locate the earliest useful signal, and identify a path back to useful service.
6.1 Local correctness leaves production assumptions untested#
Production is not simply a larger version of a developer's laptop. It contains independently changing consumers, intermediaries, services, data stores, providers, and operational tools. A failure can cross several of these boundaries before a user sees an error.
Consider five assumptions that local development can easily hide:
- Consumer behavior: a browser, integration, or agent may retry after an ambiguous timeout or reconnect in a synchronized burst.
- Network behavior: a request can reach the authority even when its response never reaches the consumer.
- Dependency behavior: a slow identity, storage, payment, or messaging service can hold resources and spread waiting through otherwise healthy components.
- Data behavior: replicas can lag, writes can conflict, queues can age, and a restored endpoint can still return stale or incomplete state.
- Operational behavior: the dashboard, deployment system, credentials, status page, or rollback mechanism may depend on the service being repaired.
These are architectural assumptions because they determine where authority, waiting, retries, and recovery live. Testing the happy path proves none of them.
Sustainability angle
Production pressure often exposes waste that local development hides: retry storms, polling, failed jobs, duplicated events, abandoned data, and oversized logs. Backoff, deduplication, retention, and load shedding can reduce both failure amplification and unnecessary resource use.
6.1.1 Field note — When caching complexity became an availability risk#
In a commercial Next.js project, GitHub effectively served as the backend for most of the site's meaningful data. Two sections used Next.js caching incorrectly, causing production page requests to repeatedly reach the GitHub API. This remained invisible during local development because our traffic never approached the API quota.
Under high production load, the repeated requests exhausted the quota. Because the site could not present useful content without GitHub, a rate limit on one external dependency became a site-wide outage. During the incident, I debugged the behavior with LLM assistance and found one faulty caching path. After the quota reset, traffic exposed a second path with the same problem, causing another period of downtime.
We eventually removed the GitHub dependency because it had also become unstable for our needs. In retrospect, load testing should have revealed that site traffic was being amplified into API traffic, while monitoring should have warned us before the quota became an availability boundary.
The incident also reinforced my preference for simpler web primitives. For this kind of site, I consider Next.js's interacting rendering and caching mechanisms more complex than the product warranted. If rebuilding it today, I would use a multi-page application with hypermedia-driven fragments and View Transitions for the interactive portions.
That alternative would still need caching, monitoring, fallbacks, and bounded external dependencies. My hypothesis is that explicit page, fragment, and data-access paths would make this site's behavior easier to explain and operate. The caching incident does not test that counterfactual. I would compare a small implementation of the same journey under the same cache and dependency-failure checks, asking an unfamiliar maintainer to trace a response and diagnose a stale result. If diagnosis remained equally difficult or required more coordination, that would challenge my preference.
6.2 What documented web incidents reveal#
The four accounts below were published by the organizations that experienced the incidents. They provide concrete timelines, observations, and recovery choices, but they are not controlled experiments or independent audits. Treat each reported number as evidence about that system and use the transfer question to decide what, if anything, applies to the event tracker.
| Incident | Failure chain | Assumption exposed | First useful signal and recovery | Transfer question for the tracker |
|---|---|---|---|---|
| Discord authentication outage | Zonal maintenance, a degraded disk, cold row caches, limited capacity, timeouts, and retries compounded | Zonal redundancy and a successful short test provided enough margin to continue | HTTP success collapsed; downstream-service time and saturation led to load shedding while capacity returned | Which optional actions can stop before retries overwhelm a shared dependency? |
| GitHub database partition | A network partition triggered failover, leaving writes in two data-center histories | Automated database failover would produce a topology the application could safely use | Replication topology and faults exposed divergence; GitHub degraded features, failed forward, rebuilt, and drained queues | Which state must remain authoritative even if availability has to be reduced? |
| Fastly global outage | A valid customer configuration activated a latent software bug across the edge network | A valid configuration change could not trigger a platform-wide failure | Monitoring detected global disruption within one minute; disabling the triggering configuration restored most service | How small is the blast radius of one valid configuration or deployment change? |
| Cloudflare Workers KV incident | Deployment automation selected the wrong build; KV and tools depending on KV failed together | Tested rollback, authentication, and deployment tooling would remain usable during loss | Immediate alerting identified failure, but recovery required a break-glass route to the previous known-good build | Can the application be recovered when its ordinary control path is unavailable? |
Use one incident to identify the earliest point where a prepared control could have interrupted propagation. Then test that reasoning against a smaller event-tracker failure.
A systematic study complements these company accounts. Yuan et al. (2014) examined 198 severe, developer-confirmed failures in five distributed data systems, including 48 classified as catastrophic. Incorrect handling of explicitly signaled errors featured in 92% of that catastrophic subset. That is not a claim that unit tests prevent 92% of all outages: the study separately analyzes preventability, and its selected systems and issue reports limit generalization. In the existing pressure probe, follow one dependency error through the handler, retry or fallback, durable state, and user-visible result. Checking that an error was detected establishes less than checking how the application recovered from it.
6.3 Trace one failure across the web boundary#
The Architecture chapter separated consumer, representation, execution, and authority. A production failure becomes understandable when the same roles are traced over time rather than inspected as a static diagram.
Consider a deliberately small event-tracker scenario:
- A browser sends
POST /eventswithout a stable idempotency key. - The origin validates the request and commits the event to the authoritative store.
- The response is lost after the commit, so the browser sees a timeout.
- Client application logic retries because it cannot tell whether the action completed.
- Another application instance accepts the retry and records a second event.
- A cached activity view briefly shows neither event and later shows both.
- Request-success metrics look healthy even though one user intent produced duplicate durable state.
No individual observation explains the whole failure. The timeout is a consumer symptom, the lost response is a network event, the retry is a policy, the duplicate is an authority failure, and the stale view is a derived-state delay. Useful evidence has to connect them through a stable command identifier, request trace, commit result, cache status, and user-visible outcome.
Idempotency is a property of the intended effect: repeating one logical operation must have the same effect at the declared boundary as performing it once. First name every outcome the application contract promises, then check whether repeating the operation preserves them together. An unchanged work-item status is insufficient when the same command also promises one activity event for that transition. The retry exercise asks you to inspect both; Appendix C develops the specific HTTP method semantics.
The corrective design is similarly cross-boundary. The consumer reuses one idempotency key, the authority stores the mutation and reusable result atomically, and the interface makes a pending or confirmed outcome visible. Retries may then recover from transport ambiguity without inventing a second domain action.
6.4 From failed assumptions to useful signals#
A system has a competence envelope: the range of conditions it can handle with its current resources, architecture, procedures, tools, and expertise. Ordinary disturbances move the system around inside that envelope. Saturation occurs when pressure reaches a binding limit and pushes the system to or beyond the boundary of what it can currently handle (Woods, 2018).
The limit is not necessarily a server at 100 percent CPU. It may be physical, such as memory or network bandwidth; deliberately imposed, such as a bounded queue, connection pool, quota, or rate limit; or organizational, such as responders unable to diagnose several interacting failures quickly enough. Software, infrastructure, operators, deployment paths, and operational tools all affect where the boundary lies (Hochstein, 2026).
A useful incident view needs several kinds of evidence:
- User outcome: success rate, error state, latency, duplication, stale data, or unavailable capability.
- Binding limit: CPU, disk, connection pool, database quorum, dependency quota, queue capacity, or responder attention.
- Amplification: retries, reconnects, fan-out, cache misses, synchronized jobs, or backlog replay.
- Integrity: authoritative write result, replication position, invariant violation, or reconciliation status.
- Recovery: rollback state, restored capacity, backlog age, replica lag, and whether user-visible behavior has settled.
This framing turns “Will it scale?” into four more useful questions:
- Which limit is the pressure approaching?
- Which signal reveals that the remaining margin is shrinking?
- What failure or feedback loop begins at saturation?
- Which prepared recovery actions can return the system to a useful state?
6.5 Connect failed assumptions to response patterns#
For the lost-response example, the response is an idempotent command: preserve one operation identity and return its recorded result on retry. The new obligation is atomic storage of the state change and reusable result. Other pressures call for different responses; Chapter 8's response guide connects them to the operating contracts they introduce. First check whether the measurements describe a coherent failure.
The Little's Law relationship introduced in the Benchmarking chapter can check one queue or in-flight boundary.
If a stable event-processing stage contains an average of 120 waiting or active events and completes an average of 8 events per second, the implied average time in the stage is W = L / λ = 120 / 8 = 15 seconds.
That calculation does not explain the delay, but it can expose incompatible measurements or show that a claimed user-visible completion time cannot fit the observed backlog and throughput.
If arrivals and completions are not approximately balanced during the observation window, record the system as changing or overloaded instead of forcing the averages into the law.
For implementations A and B, select one queue, pool, request path, or other conserved-work boundary.
State its boundary, estimate or measure average work in progress, completion rate, and average time in the system over one consistent window, then explain whether L = λW is approximately coherent and what the relationship leaves unexplained.
6.6 Recovery is part of the architecture#
Recovery is not a generic command executed after diagnosis. The available response depends on what has already become externally visible or durably authoritative.
| Recovery mode | What it protects | Required preparation | Main risk |
|---|---|---|---|
| Reject or shed load | Remaining capacity and core operations | Admission rules, retry guidance, and an explicit non-critical boundary | Rejected work may be retried immediately or lost silently |
| Degrade functionality | Data integrity or one essential user path | Features that can be paused independently and clear user states | Hidden dependency means the supposedly optional work is still required |
| Roll back | A known-good code or configuration path | Compatible state, retained artifact, health gate, and reachable control | Durable writes or migrations may make reversal unsafe |
| Fix forward | Accepted state that cannot safely be discarded | Reconciliation plan, bounded change, and strong observation | Recovery takes longer while the system remains degraded |
| Rebuild and replay | Authoritative history and deferred effects | Backups, logs, replayable work, ordering, and deduplication | Backlog load can trigger another saturation cycle |
| Break glass | Recovery when the ordinary control plane is unavailable | Independent access, narrow authority, audit, and regular testing | Emergency power can bypass normal safeguards |
Two boundaries deserve particular attention. First, recovery tooling should not depend entirely on the component it is meant to repair. Second, service restored and recovery complete are different states. Traffic may succeed while replicas lag, caches remain stale, queues grow, notifications wait, and user-visible inconsistencies still require reconciliation.
6.6.1 Follow a fix through affected deployments#
An accepted patch and a successful release report progress at different boundaries from verified remediation of affected deployments. The practitioner critique Everybody's Lost Their Minds argues that getting affected systems updated can remain the limiting step even when vulnerability discovery and patch preparation become faster. Use that argument to ask where a fix waits in your own delivery path and who owns the remaining work.
Where would a successful release leave repair work unfinished? Give your agent the remediation starting point, or inspect the runtime inventory yourself.
6.6.2 Investigate an incident evidence packet#
| Time | Record | Boundary and unresolved question |
|---|---|---|
09:02 |
Offered demand rose from 75 to 140 events/s; completed throughput flattened near 110 events/s. | Are these the same population and observation window? |
09:02 |
API CPU is 44%; database pool occupancy is 100%; p95 pool wait is 180 ms. | Work waits for a connection; is the pool too small, or is slow downstream work holding connections? |
09:08 |
Four logical check-in IDs appear in fourteen physical attempts; the effect ledger contains fourteen applied effects for those four IDs. | Ten effects are duplicates; which receiver operation failed to preserve logical identity? |
09:10 |
After atomic receiver deduplication is enabled, twelve new attempts with four new logical IDs produce four effects. | No new duplicate is seen in this cohort; the ten earlier duplicate effects still exist and need reconciliation. |
09:14 |
Importer work is 94 s old; interactive work is 12 s old and rising; global worker CPU is 58%. | Per-class waiting differs; what service-rate or scheduler evidence would distinguish unfair scheduling from a shared downstream limit? |
Choose one point in the packet and write four short statements in your existing notes: a hypothesis, its supporting record, a plausible alternative, and the next observation that would distinguish them. Then propose one bounded response and name its new obligation. For example, limiting admitted work makes some rejection explicit; it does not create capacity or establish which requests should receive priority. Deduplication can prevent another duplicate under its key and transaction contract; it does not undo damage already committed.
For optional execution, use persisted-retry/README.md in the course workspace to inspect a failure after the effect but before the worker records completion.
That experiment addresses the retry boundary; it does not reproduce the invented service's load or tenant scheduling.
Pause & think
The status is unchanged. Is the operation idempotent?
Before choosing, predict whether an unchanged final status is enough to make an operation idempotent. Write a brief reason naming the effects you would inspect.
A work item starts in progress with no completion activity event. Its completion contract promises to set the status to Done and record exactly one activity event for that transition. The first attempt commits both, but its response is lost. A retry on another instance leaves the status at Done and appends a second completion activity event. Both commits are durable, and there is no deduplication.
Count the stored completion activity events, assess the whole operation against its contract, and select the mechanism that would prevent the duplicate event for retries of this logical command.
Automatically checked practice. Saved answers are private to your account.
Preview the exercise context for the assistant
This prepares an editable message. Review it in the assistant before sending. Your notebook notes are not included.
Checking saved answers…
Your answers to earlier versions
Show worked answer
There are two completion activity events. The status assignment is idempotent in isolation, but the whole operation is not: its promised effect includes exactly one activity event for the transition. Idempotency is this property of the intended effect, rather than the name of a retry mechanism. One implementation reuses a stable command key and atomically enforces that key while recording the state change, activity event, and reusable result in authoritative storage. A retry then returns the recorded outcome without appending another event. A per-instance cache cannot coordinate retries routed elsewhere, and a longer timeout cannot guarantee that a response arrives. The caller's uncertainty does not change the stored facts given in this scenario. Explain why a transaction that commits both writes together still needs a rule for repeated commands.
Before revealing the worked answer, explain which promised effect your verdict depends on. Then change one assumption: suppose the second record is only a diagnostic request log, while the contract's activity event is recorded once. Would your verdict change, and what makes those records different?
6.7 Challenge assumptions without changing the common core#
Implementation B should not be framed as “make the app bigger” or as an extension of implementation A. Repeat A's primary pressure against B using the same workload and procedure where practical, and investigate one bounded correctness or recovery case. Keep the required behavior stable across the two independent implementations. The broader catalog offers optional investigations within the phase budget:
- many concurrent tenants, projects, groups, locations, or other domain instances,
- tens or hundreds of thousands of users,
- high and bursty event volume,
- client retries and dependency failure,
- eventual correctness where immediate accuracy is unnecessary,
- and a deployment or configuration change that must be stopped or reversed.
Choose one documented case and reconstruct it as a chain of trigger, latent condition, propagation, user effect, first useful signal, recovery choice, and residual work. Then transfer only the structure of that chain to the event tracker. Do not claim that the project has reproduced the source company's scale, infrastructure, or result.
Explain which assumptions differ between implementations A and B and which remain shared. A good reconstruction makes failure and recovery obligations visible without trying to solve every possible future problem. Use that diagnosis to choose and test a response in the Scaling chapter.
6.8 Activity and project evidence#
In the chapter 6 activity or announced equivalent, briefly connect one documented incident to implementation B: failed assumption, observable signal, response and recovery obligation. Link the source and distinguish reported facts, interpretation and proposed transfer. Keep this in the activity entry, assessed through the existing activity and diary criteria, and link it from the project index. An additional detailed incident pressure map or reproduction is not required.
In the existing A–B project pressure record, repeat the primary pressure and investigate one bounded correctness or recovery case, reusing the engineering session where appropriate. Apply Little's Law to one declared conserved-work boundary and connect a failed assumption or limit to a justified response and its new obligation. Use the same required acceptance core; further pressure scenarios are optional explorations.
6.9 Summary#
Trace a failure from trigger and latent condition through propagation, user effect, and recovery. Use the first useful signal to locate the broken assumption and evaluate a response with its new obligations. Recovery must protect authoritative state and remain possible when ordinary tools fail; successful requests can resume before reconciliation and deferred work are complete.
6.10 References#
- Woods, D. D. (2018). The Theory of Graceful Extensibility: Basic Rules that Govern Adaptive Systems. Environment Systems and Decisions, 38(4), 433–457. https://doi.org/10.1007/s10669-018-9708-3
- Hochstein, L. (2026, July). Saturation: How Your Software Will Fail at Scale. Software Should Work conference presentation. https://www.youtube.com/watch?v=PHYCRubnmSM
- Yuan, D., Luo, Y., Zhuang, X., Rodrigues, G. R., Zhao, X., Zhang, Y., Jain, P. U., & Stumm, M. (2014). Simple Testing Can Prevent Most Critical Failures: An Analysis of Production Failures in Distributed Data-Intensive Systems. 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14), 249–265. https://www.usenix.org/conference/osdi14/technical-sessions/presentation/yuan