Three milliseconds and 44 degrees: the cloud outage that had nothing to do with software

Three milliseconds. That is how long the voltage dip on the utility feed lasted — the one that, on 15 July, took three services in Google Cloud's europe-west4-a zone out of service for 14 hours and 55 minutes. Three milliseconds is less than the time a 7,200 rpm hard drive takes to spin half a turn. There was no CVE, no botched deployment, no BGP route leaked. There was electricity, and there was heat: 44°C in the data hall, and machines powering down before they cooked themselves.

We run our own hardware in datacenters and provide colocation services, so we read these reports with a distinctly non-academic interest: what happened to Google in the Netherlands can happen to anyone with a datacenter, and the list of things that failed that day is exactly the list of things almost nobody checks. Google's official incident report is published and it is honest to the point of discomfort. It is worth reading in full, but if you are not going to, here is what matters: the full chain of events, why the redundancy that existed on paper did not work, and the concrete questions worth asking — of your provider and of yourself — before it is your turn.

The chain, in order

Almost every step that follows is a protection mechanism doing exactly its job. The problem is how they fit together. Reconstructed from the report:

The report's timestamps are in Pacific time; the intervals are relative to the start of the electrical fault. The total incident lasted 14 h 55 min, and per-service impact was 9 h 24 min for Google Cloud VMware Engine (24 private clouds across 20 customers), 12 h 57 min for Bare Metal Solution (9 customers) and 8 h 31 min for NetApp Volumes.

Two feeds, two DRUPS units, and still one row in the dark

On paper, that datacenter had everything you ask of a serious facility: dual utility feeds, diesel rotary backup on each side, cooling with a redundant source. On an audit questionnaire, every box ticked. And still an entire row lost power and a hall hit 44°C.

The lesson is not that redundancy is useless. It is that redundancy only counts if the branches are genuinely independent and all of them are available at the same time. Here it failed on both counts. The triggering event was common to both feeds — a fault upstream, on the utility grid, does not distinguish between your side A and your side B — and one of the cooling branches was out for construction work. An N+1 setup with one branch under maintenance is not N+1: it is N, and worse, it comes with the feeling of having headroom.

The most instructive detail, though, is a different one: row 3 did not go down for lack of available power, but because pushing its entire load through a single feed tripped an overload breaker. It is the classic load-sharing failure: each branch handles its half, but not both halves. You find out on the day of the cut, never before — unless someone has done the exercise of adding up real per-branch consumption instead of trusting the original design. If nobody in your datacenter has redone that sum since the last five racks went in, that sum is overdue.

The link nobody audits: who powers the thing that starts the cooling

If we had to keep one sentence from the report, it would be this: the chiller controller dropped offline during the transient and failed to signal the pumps to restart. The chiller did not break. The pumps did not break. What went down was the controller that tells them to start.

This is enormously common and almost never shows up on a resilience diagram. When someone draws the cooling chain, they draw chillers, pumps, pipework and CRAC units. What almost never gets drawn is the control electronics: the PLCs, the sensors, the variable-speed drives, the little control panels that orchestrate all of the above. Their consumption is trivial — a few hundred watts — next to the megawatts they govern, and precisely because it is trivial they slip outside the protected power supply. The result is a system that survives the cut perfectly well but does not know how to come back on its own.

The proof that this was the diagnosis is in the response itself: among the facility team's first actions was wheeling a portable UPS over to the chiller controllers. An emergency UPS, dragged there in the small hours, European time, to power a few control panels that govern the cooling of a hyperscaler zone. Read like that it sounds almost domestic, and that is exactly the part to hold on to: critical infrastructure rests on small pieces nobody looks at until they fail.

44 degrees: the shutdown was not the failure

Worth saying, because quick takes on the incident tend to imply the opposite: shutting the machines down was the right call. At 44°C ambient, the alternative to an orderly shutdown is not "keep serving traffic"; it is cooking disks, power supplies and network electronics, and turning a fifteen-hour incident into a multi-week hardware replacement programme. The bill for reaching that temperature had already been run up: four SAN storage nodes failed from heat before the shutdown began.

What is debatable is the margin. A little over two hours passed between the cut and 44°C. That is all a full data hall gives you when the cooling stops, and it is rather less than people assume, because density per rack has climbed much faster than a hall's ability to absorb heat. If your contingency plan assumes "there is time to react", put a number on that "time". Google, with all its instrumentation and an on-site team, had two hours.

It is, incidentally, a textbook case for why temperature and cooling status belong on the same monitoring console as everything else — with thresholds that warn before the point of no return, not once there is nothing left to do. We devoted a whole post to this idea: an alert that arrives when you can no longer act is expensive noise.

The small print: a zone is not a building

Here is the part of the whole incident that interests us most, and the one that has been discussed least. The three affected services were VMware Engine, Bare Metal Solution and NetApp Volumes. That is no coincidence: they are precisely the services where the customer has dedicated hardware — hypervisors, physical servers, storage arrays — rather than an abstraction spread across several halls by software. And those three services ran out of one specific building inside the zone.

The standard advice from any hyperscaler is "deploy across multiple zones". It is good advice. But it hides an assumption almost nobody verifies: that your service behaves the way the zone model says it does. Specialised managed services often hang off a single datacenter within the zone, and that does not show up on the pretty diagram in the documentation. The result is a real architectural dependency many teams do not know they have, until that particular building loses its cooling.

Put another way, with no intention of twisting the knife: when you move your VMware into someone else's cloud, you change who owns the rack, not the physics. The watts and the degrees are still there, only now in a building you cannot walk into and whose construction schedule you do not set. Sometimes that trade is well worth making. But what you buy is an operational and financial advantage, not thermal immunity, and it is worth not confusing the two when writing the continuity plan.

What this incident does NOT prove

It does not prove that the cloud is fragile or that everything should come out of it. And we say that as people who make a living building and maintaining infrastructure outside the hyperscalers, and who would have the obvious incentive to say the opposite.

That datacenter had dual utility feeds, rotary backup, redundant cooling, thermal telemetry that made it possible to order a shutdown in time, and an on-site facility team that solved the problem with a portable UPS. The cupboard in an office storeroom — with its 1,500 VA UPS bought in 2019, its domestic air conditioner and its socket shared with the coffee machine — has none of that, and there the same three-millisecond transient would not have produced a fifteen-hour incident: it would have produced a room cooking itself with no alarm reaching anyone until the next morning. A hyperscaler having this happen is not an argument for doing it yourself at home; if anything it is the opposite.

The useful conclusion is not where, but what you ask and what you assume. We already did the five-year math between colocation and public cloud and the honest answer was "it depends, and here are the numbers". This incident does not change that math. It changes the list of questions to ask before signing it.

The seven questions actually worth asking

They work for the datacenter hosting your colocation, for the cloud provider you buy dedicated-hardware services from, and for your own server room if you still have one. They get answered by looking at logs and calendars, not by ticking boxes.

1. Is there construction work under way right now? And if so, which redundancy branch is degraded while it lasts, and until when? It is the highest-yield question on the list and almost nobody asks it. Work on the cooling loop turns your N+1 contract into an N for months, without a single line of the contract changing.
2. Are the cooling controllers on protected power? Not the chillers: the controllers. And the pumps, and the drives. It is the question this incident puts on the table, and the one that separates whoever has thought about the problem from whoever has bought equipment.
3. When was the last transfer test under real load? With a date. Running the generators unloaded on a Tuesday morning proves nothing; what needs testing is the transfer under today's load, not the load on design day.
4. If one branch takes the entire load, does it hold? This is what took down row 3. Ask for the per-branch consumption split and compare it against the breaker ratings. If nobody has redone that sum since the last growth spurt, it is out of date by definition.
5. How long do I have between cooling stopping and shutdown? A number in minutes, measured, not estimated. It is your real reaction window and it determines whether a manual plan makes sense or whether everything has to be automatic.
6. Are my systems genuinely spread out, or does it just look that way? In cloud: is the service I use multi-hall, or does it hang off one building? In colocation: are my two "redundant nodes" in different halls, on different feeds, or are they two adjacent racks fed from the same panel?
7. Who tells me, when and through which channel? And in what detail. Google published a report with the full chain, timings to the minute and a list of corrective actions with target dates. That level of detail is what lets you decide whether to stay. A provider who only gives you "issue resolved" is asking you to trust them without data.

And even with all seven answered

Even if your provider answers all seven well, there remains a real chance that on any given Tuesday an entire hall goes dark for fifteen hours. That does not get fixed with a better contract: it gets fixed by having the service somewhere else and having tested it. It is the same conclusion we reached with this month's AWS outage, and repetition has not improved it: recovery you have not rehearsed is not a plan, it is an intention.

In practice, for most mid-sized companies that means fairly mundane things: backups living well away from the infrastructure they back up, a second location with enough capacity for what matters — not for everything, for what matters — and someone having actually timed how long it really takes to bring the critical parts up there. We spread our own Proxmox and Ceph infrastructure across several datacenters precisely for this reason, and the part that really teaches you something is not building it: it is timing the way back.

Three milliseconds. A good reminder that, underneath all the layers of software, there is still a building, a utility feed and a chilled water loop. And that real resilience is not bought: it is verified.

Sources (verified): the 3 ms voltage dip and the utility breaker protective action, the side-A DRUPS failure due to electrical component failures, the successful transfer of rows 1 and 2 and the overload trip on row 3, the chiller controller dropping offline and the pumps that did not restart, the redundant cooling source being unavailable due to construction work, the 44 °C in the data hall and the crossing of the safe operating threshold, the shutdown of servers, storage and switches initiated by the engineering teams, the four NetApp SAN nodes lost to heat before that shutdown, the interim portable UPS for the controllers, the total duration of 14 h 55 min, the per-service impact (9 h 24 min for GCVE covering 24 private clouds across 20 customers; 12 h 57 min for BMS across 9 customers; 8 h 31 min for GCNV) and the corrective actions with target dates staggered through October 2026 — Google Cloud official incident report. The note that these services ran from one specific datacenter inside the zone, and the transparency gap that creates: The Register (21 July 2026) and DataCenterDynamics. The reading on redundancy degraded by construction work, control electronics outside protected power, the thermal window as a measured number, and the seven questions are ours. Cover image: interior of a technical room (royalty-free image bank).