On 1 September, in a Google datacenter, an engineer unplugged, one after another, 100% of the fiber paths serving part of the us-central1-b zone. It took thirteen minutes. The interesting part of the report is not the mistake: it is the paragraph Google writes just before it, describing an impeccable redundant architecture.
We already wrote about Google's previous outage here: that one was power and cooling, a facility failure. This one is nothing like it, and that is what makes it worth reading. Nothing broke here. All the hardware was healthy; what was disconnected, somebody disconnected, by hand, with an approved work order.
What was correctly designed
Before describing the mistake, the preliminary report spends a paragraph explaining how each datacenter network is built. It is worth reading in full, because it is exactly what any of us would put in a proposal:
"The network architecture for each datacenter is designed with redundancy across multiple routing devices. The system is designed to be resilient to all single device or fiber-path failures, and most double- or triple-failures do not affect customer traffic. To ensure this, the devices and fiber paths are physically separated in each datacenter, with diverse power sources."
All of that is true, and all of it was in place. And it made absolutely no difference, because the next sentence in the report says that "a procedural error meant that the physical maintenance action sequentially unplugged 100% of fiber paths across all devices within 13 minutes".
Redundancy is a masking machine
This was explained a long time ago —twenty-three years ago. In 2003, three Berkeley researchers —David Oppenheimer, Archana Ganapathi and David Patterson— analysed why three large internet services fail and published the results at USENIX. The conclusion everyone quotes is that operator error was the largest single cause of failures in two of the three services. The one almost nobody quotes is the one that actually explains 1 September, and it is a caveat the authors themselves take the trouble to write down:
"It is therefore not the case that operator error is more frequent than hardware or software problems, just that it is less frequently masked and therefore more often results in a service failure."
Masked. That is the precise word for what redundancy does and for what it does not do. A dying disk, a burnt power supply, a cut path: these are independent events, each on its own, and redundancy exists precisely to cover them before anyone notices. That is why the guarantee in Google's paragraph is worded the way it is: resilient to all single device failures, and to most double or triple ones. Those are statements about improbable coincidences. Physical separation and diverse power exist so that a leak, a fire or a switchboard cannot reach two paths at once.
A procedure is not a coincidence. A procedure traverses: element by element, in order, on purpose, until the list is done. And if the list is your redundant paths, walking it end to end means reaching 100% by definition. There is no mask available, because there is nothing left behind to hide with. That is the literal meaning of "less frequently masked".
And there is a detail in the paper that fits this incident better than anyone would like. The authors write that "operator errors arose when operators were making changes to the system, e.g., scaling or replacing hardware, or deploying or upgrading software", and add that most of them "arose during normal maintenance". The task on 1 September, in Google's words, was "a scheduled capacity upgrade". Scaling hardware. Normal maintenance. Twenty-three years later, in the same box.
With one honest difference worth stating: in those three services, operator errors were mostly configuration errors —over 50% in all three cases— rather than physical procedural errors, which the paper itself places in the minority. This one is a rare kind. What does not change is the mechanism: somebody making a planned change, in a planned window, reaches at once everything that was separated precisely so it would not be reached together.
Your list is shorter than you think
This is where it stops being a hyperscaler anecdote. Your redundancy also carries an assumption nobody wrote down: that nothing touches every copy. And every company has a short list of things that do touch them all, one after another, with credentials and without asking: the patching script, the backup agent, the remote management tool, the firmware update rolled out in batches, the work order that says "both switches". These are not attacks or breakdowns. They are routine, approved, well-intentioned tasks, and they are the only things capable of walking your redundancy end to end. That list fits on half a page and almost nobody has it written down until you ask for it.
This is not the first time we have run into an independence assumption that turns out to be false. In August we wrote that redundancy is not route diversity: two different carriers, two different contracts and the same trench. That was a geographic correlation nobody chose. This one is more uncomfortable, because the correlation is not accidental: it is introduced, every time, by somebody on your own team doing their job properly.
The network came back at 09:19. The incident ended at 11:52
The report gives a duration of 4 hours and 11 minutes, from 07:41 to 11:52 Pacific time. But if you follow the status updates Google published through the morning, that number splits into two halves that look nothing alike. At 09:19, an hour and a half after the first cable, the notice says that "following network-level mitigation efforts, the underlying network infrastructure has fully recovered" and that product teams "are currently validating the recovery status of individual services". The fiber was already back in. Two and a half hours of incident remained.
That second half is the part almost nobody budgets for. Detection, incidentally, was instantaneous: the report says the issue "was detected immediately by automated network loss monitoring systems as well as proactive probes". Finding out cost nothing. Getting back up cost everything else, and most of that time was not plugging fiber back in: it was waiting for the services above to put themselves back together. It is the same distinction we described in the continuity plan nobody has rehearsed, where the problem was that the document said "restore" without anyone having timed the verb.
What the provider can move for you is traffic, not state
The per-service impact breakdown reads well next to the timeline, and what comes out of it is not what you would expect. Compute Engine virtual machines suffered "inability to access virtual machines externally, and inability for VMs to reach remote resources": nobody moved them, they stayed where they were until someone plugged the fiber back in. Managed databases —Cloud SQL, AlloyDB, Spanner, Bigtable, Filestore— had "data access and connectivity severed for instances localized strictly to the impacted infrastructure": the ones living in a single place went down. And Kubernetes had "unreachable regional/zonal clusters and nodes within the affected zone, prompting delayed cluster recovery times": a high layer, and nobody evacuated it either.
The only ones that moved on their own were Cloud Run and App Engine: "temporary latency spikes and pending queue aborts as backend workloads automatically evacuated and shifted to healthy capacity". Without anyone calling anyone. And now the part that throws you: they were also the last to come back. In the 11:36 notice, when everything else is already listed as recovered, the only two products Google is still naming are Cloud Run and App Engine; and the report says their degradation lasted "until 11:52". Which means the official end time of the incident is theirs, not the network's.
The easy reading would be "the higher the layer, the better". That is not what the report says. What it says is that the provider can move traffic without asking you, and that this has a price of its own: automatic failover has its own recovery too, and in this case it was the longest of all. Lifting your virtual machines into a cloud does not buy you failover; it buys you a zone. And picking a layer that does fail over on its own does not buy you instant. Those are two different things and both need budgeting.
The mitigation they published was a decision made months earlier
In the middle of the outage, the status notice published its workaround section. It said two things. One: "customers that are severely affected and have services available in alternate zones can failover to alternate zones". And two: "customers with service layer retries may see successful completions as only a portion of the zone is affected".
The two mitigations your provider can offer you in the middle of a four-hour outage are things you either already had in place, or that do not exist for you. Nobody buys a second zone at 07:41 with their machines mute, or adds retries to an application while it is falling over. That is literally what a recovery plan is: options bought in advance. Nothing is decided on the day of the incident; on the day of the incident you only spend what you already had.
Two easy readings we are not going to make
The first is "this is why we do not trust the cloud". No. Google published a report with the start time, the end time, the root cause, an explicit admission that the error was procedural, and a commitment to say later what changes. The uncomfortable question is not for them, it is for everyone else: does your last outage have a document like that? With the exact time, what was being touched, and who approved it. If that piece of paper does not exist, the same change will happen again, and the second time you still will not know why.
The second easy reading is laughing at the engineer, which is what most of the week's headlines did. It is the worse of the two. The most important sentence in the whole report is the one immediately after the root cause: "the nature of the error, combined with the speed of the action, prevented warnings of incorrect action reaching the engineer before complete disconnection". There were warnings. They did not arrive in time.
A procedure where one person can reach 100% of the paths, and where the alarms are slower than the hands, is a procedure design problem. Not a person problem. Blaming whoever pulled the cable is the most effective way to guarantee the next one repeats it, because it leaves untouched the only thing that could have been fixed.
And that Google read the incident this way is shown by its own immediate action, in the last notice of that day: "maintenance in the region has been halted while proactive audits are carried out". They did not stop a device or a version. They stopped the procedure, across the whole region, until it had been reviewed. When what fails is the way things are done, the first serious mitigation is to stop doing them.
What you can do on Monday, and when you need not do anything
First comes the list from earlier: which of your procedures can touch more than one redundant element. Written down, not from memory. Second is a number almost nobody has measured: how long undoing takes —not detecting—; and if the answer includes "someone has to go there", add the drive, and add it on a Sunday. Our last full recovery drill, stopwatch in hand, was 14 minutes: an internal figure and a test, not a contractual promise, and what matters is not our number but that you have one you measured rather than one you estimated. Third is to look at Google's two workarounds and ask yourself which of them you would have in place today.
And on the metal, the cheapest measure of all: work on one path and work on the other must not be signed off for the same night. It is the first thing we ask for in writing whenever a customer's equipment is physically touched in a colocation room, and not out of distrust: because separating the windows is the only thing that turns a slip of the hands into a single-path incident. It costs no money. It costs writing it into the work order.
Now the part that sells us less and needs saying anyway: if your company can take four hours down once every two years without it showing up in the accounts, do not build multi-zone. The cure will cost you more than the disease. For a shop with online orders, four hours on a Tuesday in September is hard cash; for an accountancy firm, it is an odd morning of working locally and grumbling. Those are two different decisions and both are legitimate. The only position that is not consists of never having done the arithmetic and discovering the figure on the day it happens.
What we are not claiming
- ✗The report is preliminary. Google states literally that a final incident report with preventative actions will be posted once the investigation is complete. We do not know what they will change, and anything we wrote about that would be invented.
- ✗We do not know what the "procedural error" consisted of. Whether it was a badly written work order, mislabelling, the wrong rack or steps in the wrong order, the report does not say. Our reading about the sequential nature rests on the word Google uses, not on a reconstruction of the mistake.
- ✗We reconstructed the second half of the incident from the status updates, not from a Google explanation. That the network was recovered by 09:19 and the incident closed at 11:52 are two published facts; why Cloud Run and App Engine took longer, nobody says, and neither do we.
- ✗We know there were warnings and that they did not arrive in time, but not which system issues them or how long it normally takes. That the alarms were slower than the hands is what the report says; why, it does not.
- ✗Only part of the
us-central1-bzone was affected, and the notice repeats that "the remaining zones in the region are unaffected". Containment at the region level worked. Saying so is part of telling the incident properly, even if it is the part that makes the worst headline.
Redundancy masks what happens by accident. It does not mask what happens on purpose, in order, with the list in hand. If nobody has ever looked at which of your procedures can traverse all of your paths, your high-availability architecture has an unverified assumption at its foundation — and those assumptions get checked exactly once, on their own, at a time of their choosing. Failure is inevitable. The breakdown, once again, is a design decision.
Sources (verified 7 Sep 2026): Google Cloud incident J5ia5t9p3g9Q5Wi7r8Ev —preliminary report published on 3 Sep with dates, the 4 h 11 min duration, the network architecture paragraph, the root cause ("sequentially unplugged 100% of fiber paths across all devices within 13 minutes"), the sentence about warnings not reaching the engineer, the immediate detection and the per-service impact breakdown; plus the status updates from 1 Sep itself, which give the network recovery at 09:19, the state of Cloud Run and App Engine at 11:36, the two workarounds, "the remaining zones in the region are unaffected" and the halt of maintenance across the region—: status.cloud.google.com. Masking of operator error, changes during "scaling or replacing hardware" and the proportion of configuration errors: Oppenheimer, Ganapathi and Patterson, "Why Do Internet Services Fail, and What Can Be Done About It?", USENIX USITS 2003 (quotes from the full text). The 14-minute drill is an internal everyWAN figure: our own measurement, not a service commitment.
Which of your changes can touch every copy at once?
At everyWAN we design recovery with the arithmetic on the table: which procedures can traverse all of your paths, how long going back really takes —stopwatch in hand, not estimated— and how much of that is worth budgeting for in your case. If you have never measured that second number, that is where we start.
Talk to everyWAN
Microsoft 365
ES 
CAT 
EN