The zone that went down in London was a power panel

Yesterday morning, at 06:08 UTC, a zone went down. Just one: europe-west2-b, one of the three in Google Cloud's London region. The cause the status dashboard published was not a botched deployment, an expired certificate or a Friday configuration change: it was a "power event". Eight hours later, the sentence most often used to explain why some fell over and others did not — "we are spread across two zones" — had turned out to be a sentence about labels, not about places.

What Google said, word for word

The status note, with the cut marked: "A power event caused an outage within europe-west2-b zone. The power event is mitigated. We are seeing the recovery of services and are prioritizing the recovery of existing workloads. […] Customers may not be able to yet create new workloads." That last sentence is worth more than it looks, and we come back to it in the checklist.

The Register listed seven affected products: BigQuery Data Transfer Service, Cloud Filestore, Cloud Run, App Engine, Google Cloud NetApp Volumes, Google Kubernetes Engine and the managed Apache Kafka service. The status dashboard ended up listing considerably more, including regional products such as Cloud Storage and Compute Engine. That detail matters, and we come back to it at the end. On the receiving end: Lloyds, HSBC, Revolut, Royal Mail, Rightmove, DPD, BT and Screwfix, among others; payments, transfers, parcel delivery, property portals and part of the video hearings system for courts in England and Wales. Google said at 11:58 UTC that full recovery was expected "tentatively" by 13:00, and marked normal service for the majority at 14:03 UTC.

At the time of writing there is no published incident report. Google says there will be one: "We will publish an analysis of this incident once we have completed our internal investigation." Until then, what exactly failed — the feed, a distribution board, the transfer to generators, the batteries — we do not know, and neither does anyone asserting it on social media. What we do have is the label on the cause and the blast radius.

That diagnosis contains no software. It contains electricity. And electricity does not appear on any architecture diagram anybody has ever shown us in a meeting.

"We are spread across two zones" is not an answer

Google's own documentation says it plainly: "Zones and regions are logical abstractions of underlying physical resources", and "A zone is a deployment area within a region". A logical abstraction. A deployment area. Not one word about rooms, switchboards, feeds or generators.

AWS's documentation goes further. Verbatim: "the us-east-1a for your AWS account might not be the same physical location as the us-east-1a for another AWS account". In other words: my zone "a" and your zone "a" may not be the same building. It was not a mistake: it was a deliberate choice to spread load, because otherwise everybody picks the first one on the list. To coordinate across accounts there are AZ IDs, stable identifiers such as use1-az1, which do point — again verbatim — to "the same physical location in every AWS account".

That independent mapping is no longer general. Every Region introduced after November 2012 uses a uniform mapping, and from November 2025 new accounts get the uniform mapping in the old Regions too. But it still applies to accounts created before that date in nine Regions, and the list includes three that show up constantly in Spanish architectures: Ireland, N. Virginia and Oregon, plus N. California, Singapore, Sydney, Tokyo, São Paulo and GovCloud (US-West). And do not conflate providers: this is AWS, not Google; Google documents no per-project randomisation, and we are not going to imply there is any.

What the two providers share is what matters: a zone name is a distribution label. It does not tell you which room it is in, which distribution board it hangs off, whether it shares a feed with the zone next door, or how many metres away that is. What you buy is a commitment to independence, which is a serious thing and is normally honoured. What you do not buy is a floor plan. That a zone is not a building we already wrote in July, when Google lost europe-west4-a to voltage and cooling (we covered it in July with europe-west4-a); today's angle is the next step: what you can audit yourself. And from the opposite end, when what went down was an entire region: your cloud survives losing a zone, not a region.

The oldest failure domain is the one least often drawn

Redundancy is designed in layers, and almost everyone stops at the first two. Disk layer: mirroring. Machine layer: dual power supplies, because the whole box dies — bad memory, a hang. Room layer: a data centre offline from a power cut, a fire, a flood. Power sits below all three, and it is the layer that gets drawn worst for a basic reason: it is not a component, it is the precondition for components existing at all. On a logical diagram nothing is plugged into anything.

Public cloud adds a trap of its own. Resources are classified as zonal, regional or global, and that classification is set by the provider product by product, not by your intuition. A zonal persistent disk, a single-zone Kubernetes cluster or a file volume lives in one zone, even if your console says europe-west2 and even if the word "region" appears three times in the deck that got the budget approved. The architecture looks regional. The resource is not.

Ceph named this years ago, and the name is "pdu"

For anyone running their own hardware this has had a name for years. CRUSH's default hierarchy — CRUSH being the algorithm Ceph uses to decide where each copy of each object goes — contains this list of bucket types, and has for many releases: osd, host, chassis, rack, row, pdu, pod, room, datacenter, zone, region, root.

Between row and pod there is a bucket type called power distribution unit. It is there because whoever designed this knew exactly what happened in London yesterday: that power is a failure domain in its own right, that it does not line up with the rack or the row, and that two machines twenty metres apart can die together if they hang off the same strip. The documentation puts it plainly: "By reflecting the underlying physical organization of the installation, CRUSH can model (and thereby address) the potential for correlated device failures."

Ceph's default failure domain is host: three replicas on three different machines. That protects you from losing a machine, which is the most common failure, and it is a well-chosen default. But if those three machines sit in the same rack, and that rack hangs off one switchboard, then against a power event you have three copies and one failure domain: as far as the board is concerned, one copy. Ceph will report HEALTH_OK throughout, because it is honouring precisely the rule you wrote. The system is not lying; nobody asked it about electricity.

The difference between a cloud zone and a CRUSH bucket is not technological: one can be printed and the other cannot. A ceph osd tree gives you the whole tree — which disk hangs off which machine, which rack, which row — on one screen, and you can lay it next to your room's electrical schematic and argue about it with the electrician. London's "zone b" cannot be printed. Not because it is hidden from you, but because to you it is not a place: it is a label with a commitment behind it. That does not make the cloud bad; it makes the question different. We went through this in full when we wrote about how a three-node Ceph does not heal itself: it holds on.

When NOT to raise your failure domain

The natural reaction to all of the above is "fine, I will set it to rack and be done". We are going to say the opposite, because this is the one we have watched break. If your pool keeps three copies and you ask the algorithm for a separate rack per copy, you need three racks with space and disks in them. With two, CRUSH cannot place the third: placement groups sit undersized and, depending on your min_size, the pool stops accepting writes the moment anything else goes wrong.

What you have done there is swap a durability risk for an availability problem. And the second one arrives sooner: the distribution board may fail once every several years, but a cluster that will not accept writes shows up next Tuesday. Declaring room when you have one room is not resilience, it is a stalled cluster with excellent intentions. On a three-node cluster the sensible domain is still, almost always, host.

A distribution board is not fixed with a command. It is fixed with electrician work and inventory work — two feeds, two boards, a UPS somebody actually load-tests — and we wrote that one up in full when it came round: both power cords on your server go to the same strip. What Ceph adds is that, once that work is done, you can tell the system about it and the system will honour it.

The missing map: one row per replica

We fill this map in standing in front of the rack, with a torch, following each cable to its board. Not from the inventory, not from the spreadsheet left over from the last audit, not from the memory of the colleague who built the thing. One row per copy of your data.

If two rows share a distribution board, you have two copies, not three. If all three share a UPS, you have one. Whatever the dashboard says. The arithmetic is not done by the storage system; it is done by the electrician.

If yours lives in the cloud the map is different, but the exercise is the same and it fits in an afternoon:

1. Write down the identifier, not the name. On AWS, aws ec2 describe-availability-zones --query "AvailabilityZones[].{Name:ZoneName,ID:ZoneId}" --output table gives you both columns. If you spread across accounts or organisations, the ID column is the one that counts.
2. Mark which resources are zonal. One by one, from the product documentation rather than the folder name. The disk, the cluster, the file volume, the queue. Anything living in a single zone gets underlined.
3. Ask about the control plane. Not only "does my application survive losing a zone?" but "can I launch new things while that zone is down?". They are two different questions, and the second one decides how long the bad morning lasts.
4. Put a date on the test. Deliberately switch off what lives in one zone, during office hours, with everyone watching. If the answer to "can we do it Thursday at eleven?" is "absolutely not", you already have the result without switching anything off.

The bill London leaves behind

From 06:08 to 14:03 is seven hours and fifty-five minutes. A 99.9% availability commitment allows 8 hours and 46 minutes of downtime across the whole year. Which means yesterday's incident, on its own, consumed 90% of the annual budget of a 99.9% target. And it is October.

It is also worth not confusing your provider's commitment with your own: what you signed with your customers and what the provider refunds in service credits are two different things, and credits do not pay for the day the warehouse shipped nothing. We wrote about that confusion in full a few days ago: 99.9% does not tell you how long the outage lasts.

One thing does not fit, and it is the one that gave us the most to think about: zone isolation did not entirely hold. Google traced the cause to europe-west2-b, but its own dashboard summary says "some services in europe-west2 region might experienced elevated errors and intermittent network issues", and regional products appear in the list. Which means being spread out was not enough yesterday. The zone boundary did not fall where the diagram said it would — which is, once again, this post's argument. That said, the region stayed up: Google lost one zone of three and did not lose London.

Which board does the second copy hang off

That is the question, and "we are replicated" is not an answer to it. If nobody in the room can answer it, what you have is not two copies: it is two places holding the same one. We build distributed storage with Ceph precisely when the spread has to be something you can point at, and the rest of the infrastructure with the electrical schematic on the table from day one. If you want us to look at yours, the first conversation is filling in that table: get in touch.

Sources. Google Cloud status note on the power event in europe-west2-b and the 20 October 2026 timeline, as reported by The Register; the list of affected organisations from the same coverage and the UK press. Google Cloud documentation, "Regions and zones" (zones and regions as logical abstractions; a zone as a deployment area). AWS documentation, "AZ IDs" (independent per-account mapping, use1-az1 identifiers, affected Regions and the November 2025 change). Ceph documentation, "CRUSH Maps" (default hierarchy bucket types, correlated failures, ceph osd tree) and erasure-code profiles (crush-failure-domain defaulting to host). Incident summary and timeline from Google Cloud's public status dashboard, consulted on 21-10-2026. The nines calculation is our own arithmetic. At the time of publication there is no public incident report from the provider; Google has said it will publish one once its internal investigation is complete.