Back to Blog

The sensor was watching the live side of an open breaker

Engine of a standby generator set installed in the plant room of a data centre

On 4 November the final report landed on the outage that left Google Cloud's europe-west2-b zone in London dark. Inside it is one sentence worth an entire audit: "The PLC undervoltage sensing inputs were wired upstream (line side) of the isolated utility feed breakers." Meaning: the controllers that decide whether the standby generators start were sensing voltage before the utility feed breaker, not after it. And feed B's breaker was open and locked out, to repair what a load test had found six days earlier. When feed A went down, the controllers looked at B, saw perfect voltage on a path that could not deliver a single amp, and never issued the start command. The generators were there. The diesel was there. The command was not.

On 21 October, in the heat of it, we wrote about this same outage when only the what was known: that an entire zone is a physical failure domain, not a label in a console. A correction while we are here: that day we said "almost eight hours" and 06:08 UTC, which was what there was; the final report puts the outage at 6 hours and 8 minutes and the cut at 06:53 London time. What was not known then was the why, and the why is considerably more uncomfortable than the what, because it is not about a broken cable —cables break— but about a sensor-wiring decision probably made at commissioning, years earlier, and a maintenance window nobody counted as risk.

What happened, with the report's own timestamps

Every timestamp below is the report's own, in London time. At 06:53 on 20 October the primary utility feed A fails because of a fault in an underground cable on the utility's network. At 07:00 impact begins: the UPS batteries are exhausted and the data halls lose power, taking hosts, local storage and the zone's network hardware with them. At 07:04 the alerts fire. At 07:09 automatic failover to neighbouring zones kicks in. At 07:54 power returns via the standby utility connection. At 08:20 the halls are energised again and the automatic start controllers have been reset and tested. At 12:11 services are available again, and at 13:08 the report marks "majority workload recovery", with 99.5 % of virtual machines operational. The official duration of the incident, counted from the 07:00 start of impact, is 6 hours and 8 minutes.

Look at that first stretch: between the power going away and the halls going dark, seven minutes pass. The report states the designed runtime itself —"Uninterruptible Power Supply (UPS) battery systems provide approximately five minutes of autonomous runtime"— so those seven minutes are the real depletion of batteries sized for five. Out of which comes what strikes us as the expensive lesson here: UPS runtime is not a cushion, it is the maximum window the generator start has to happen in. And that is not our reading: among the corrective actions, Google itself commits to testing automatic generator start "within the backup battery timer". It calls it a timer. That is exactly what it is.

Where the sensor was tapped

The installation is textbook. The 2N is the two complete utility power paths —feeds A and B— with the definition the report writes alongside it: "the facility has complete duplicate power paths for every critical load". On top of that there are generators and a standby utility connection which, worth holding on to, is "configured for manual campus-wide transfer": it is thrown by hand. Automatic generator start is decided by programmable controllers watching for undervoltage, and the problem is where those sensing inputs were tapped: on the line side, upstream of each feed's breaker. The report finishes the job: "Utility feed B remained energized upstream while its incomer feed was in open position." Feed B was still live above, with its breaker open below.

Translated into a sentence that fits in a meeting: voltage present does not mean power deliverable. A sensor tapped before an open breaker reads exactly the same as it would if everything were fine. The controller's logic did not take breaker position into account, only voltage, so its conclusion —"supply is present, do not start"— was flawless given the data it had. And the data was correct. What was not correct was the question.

The report adds the detail that turns a mistake into a design flaw, and it is the report saying it, not us: "the active and standby master PLCs, which were running identical control logic, sensed live voltage from feed B through shared upstream sensors on the disconnected path". There were two controllers, one active and one standby, running the same logic and looking through the same sensors. Two devices carrying the same error are a copy, not redundancy. You do not fix that by buying a third. And it is worth bringing in here, as an analogy rather than proof —the study is about internet services, not electrical plant—, the classic finding of Oppenheimer and colleagues in 2003: in serious outages, configuration and operations dominate, and hardware accounts for a minority. Here the hardware did its job: the cable broke —that is a failure, and failures are inevitable—, the generators were in place, the UPS held for as long as it was meant to. What turned the failure into an outage was a decision made on a drawing, with a marker pen, long before anyone switched on a server.

The quarterly test did its job. The window it opened did not

This is the part of the whole report that interests us most. On 14 October a quarterly generator load test was run. The test found problems. Fixing them required opening and locking out feed B's breaker, and the facility moved into what the report calls "a temporary single-feed (1N) bridged maintenance posture". Six days later, the feed that was left went down.

None of which should read as an argument against testing: quite the opposite. The test did exactly what it exists for, which is to find a problem on a Wednesday morning rather than at seven in the morning the following Tuesday, with Lloyds, HSBC, Royal Mail and Revolut hanging off it. We argue for breaking things on purpose, during office hours, for precisely this reason. What the report exposes is something subtler, and almost never written into any contract: the redundancy you buy is not a state, it is a percentage of time. A 2N facility that spends, say, four weeks a year in maintenance is 2N for 92 % of the time. The rest is 1N. And 1N does not announce itself.

Which leads to the question that actually changes a decision: do they tell you when the facility enters a degraded posture? Because if your provider opens a breaker on Tuesday and you have a change window on Thursday to patch the hypervisors, that information changes your Thursday. If it never reaches you, you do not change it, and you end up stacking your maintenance on top of theirs without knowing.

What the brochure says and what to ask instead

This is the table we use when we look at a facility, whether to house our own hardware or to recommend it to a client. None of the questions on the right is aggressive: a well-run facility answers them in ten minutes, with documents. The reaction to the question is, in itself, information.

What the brochure says What to ask instead
"2N power, two independent utility feeds" How many hours a year are you in 1N for maintenance, and do you notify me when it happens?
"Generators with N hours of fuel autonomy" What decides that they start? Where is the sensing, and does breaker position enter the logic?
"UPS with X minutes of runtime" Those minutes are the start deadline, not a cushion. How long did the transfer actually take in the last test?
"Quarterly load tests" Does the test transfer real load with the utility cut? And what posture is the facility left in while whatever the test finds gets repaired?
"Redundant control PLCs" Do they run the same logic? Two boxes with the same bug are one box. Who reviews that code, and how often?

That last row is not our own invention: the report's corrective actions include reviewing the controller code together with the facility vendor teams, "validate global PLC configurations at sites with similar topologies and PLC manufacturers", freezing planned maintenance at those sites in the meantime, and a follow-up review in which "a second Google team will then validate this within a quarter". That last one says the most: if the operator running some of the largest data centres on the planet has decided it needs a second reading of the code that starts the generators, the problem is not one unlucky facility. It is an entire class of installations.

Six hours eat the whole year

The arithmetic is the part a board understands without argument. 99.9 % annual availability leaves a budget of 8 hours and 46 minutes of downtime a year. This outage consumed 6 hours and 8 minutes: 70 % of the annual budget in a single Tuesday morning, with two months of the year still to go. If what you had promised was 99.99 %, the entire year's budget is 52 minutes, so this outage ate it seven times over. The arithmetic is ours, but the two starting numbers are not debatable: they come out of a multiplication.

And here is the point we repeat until people tire of it: failure is inevitable, an outage is a design decision. The underground cable was going to break some day; nobody decides that. What does get decided is whether that break switches your company off. The difference between the two is not in the provider's brochure: it is in how many sites you have, and in whether you have ever tried losing one.

When you do not need two facilities

We are not going to finish by saying everyone needs to double up on facilities, because it is not true and because whoever says it is selling something. If your business can take half a day down once every three years without losing customers, without breaching a contract and without anyone having to explain themselves to a large client, paying for two facilities is over-engineering, and we will tell you so. There are companies for which the right answer is one good facility, copies somewhere else, and a written plan for what happens in the meantime.

What is not optional is having decided it. The difference between resilience and luck is that one is written down and the other is discovered on a Tuesday at seven in the morning. And the uncomfortable caveat, which applies to us too: a second facility saves you from nothing if both hang off the same transit provider, if DNS lives in a single place, or if the copy you keep at the other end has never actually been restored. We already laid out the numbers on how long a real recovery takes versus what the paperwork promised, and no colocation contract on earth covers the gap between those two things.

Two sentences from the report nobody quoted

The first explains where the hour went. Between the 06:53 cut and power returning at 07:54, sixty-one minutes pass, and that is because the standby utility connection is "configured for manual campus-wide transfer": someone has to walk over and throw it. It is a perfectly reasonable decision —a manual transfer does not trip on its own because of a transient— but it is worth knowing it is there, because it changes the answer to "how long before you get power back?" depending on who is on site and what time it is.

The second is for anyone who spreads load across zones and sleeps soundly. The report acknowledges that "some customers running workloads in unimpacted zones (europe-west2-a and europe-west2-c)… also experienced unexpected impact". The zones that did not go down gave trouble too, because the automatic failover from the one that did landed on top of them. Spreading across zones is still the right thing to do; what is not true is that it comes free on the day one of them goes entirely. If your contingency plan assumes the neighbouring zone soaks up all your traffic without breaking a sweat, that plan has not been tested with the neighbouring zone full.

At everyWAN we house our own hardware in data centres, and we do it for the same reasons anyone does: because the facility is run by people who do this full time. That does not remove the work of asking questions, it moves it. If you are choosing where to put your machines, or reviewing what you already signed, that conversation is colocation properly understood: not rack units and amps, but what happens the day the facility is left on a single leg. And what cannot live inside that facility —the copies, the plan, the rehearsal— is disaster recovery, which stays yours no matter what you sign. As for the other pressure coming at these facilities, the electricity bill and the regulation, we already covered that when we looked at Spain's draft royal decree on data centres.

Sources (verified): everything factual in this post comes from a single primary source — the Google Cloud incident report, europe-west2, updated 4 November 2026: the timestamps, the 6 h 08 min duration counted from the start of impact, the underground cable fault, the quarterly load test of 14 October, the 2N architecture and the manually transferred standby utility connection, the five minutes of UPS runtime, the verbatim sentences about the sensing and the two controllers, the temporary 1N posture, the collateral impact on europe-west2-a and -c, and the corrective actions. The report's times are in US/Pacific with their British-time equivalents; London time is used throughout here. The only contribution from The Register (5 November 2026) is the list of affected organisations. Ours is the arithmetic on those figures: the seven minutes between cut and impact, the sixty-one minutes to the standby feed, the 70 % of a 99.9 % budget and the seven times over for 99.99 %. That configuration and operations outweigh hardware in serious internet-service outages — Oppenheimer, Ganapathi and Patterson, USENIX 2003, a study of internet services, not of electrical plant: we bring it as an analogy, not as proof. The questions in the table are ours, from reading facility contracts.

How many hours a year is your facility on a single leg?

If you do not know the answer, you do not know what availability you actually have. We will go through where your hardware sits, what happens the day that facility is down to one feed, and which part of the plan is still yours.

Talk to everyWAN

Tags:

Share:

Subscribe to our newsletter

To receive IT stories, everyWAN news and exclusive subscriber offers, sign up to our mailing list

Minorisa de Sistemas Informaticos y Gestión S.L. © 2026
everyWAN
everyWAN