On 30 September 2026, a five-hour-forty-five-minute incident on Azure. Of the five services Microsoft lists as impacted, four are the road to your infrastructure and the fifth is Azure VMware Solution. Inside: what the official 7Q30-010 record does and does not say, the real detection clock, why the region figure going round is not in the source, and what happens to your contingency procedure when the thing that fails is the console.
On 12 June 2025 Google Cloud went down hard. 40 minutes in, the mitigation had been rolled out and regions started to recover, the smaller ones first; us-central1 was not fully resolved until 2 h 40 min after the incident began. What stretched that region was not the bug: it was the retries. The public incident report says it plainly —"Service Control did not have the appropriate randomized exponential backoff implemented to avoid this"— and describes a herd effect on the Spanner table everything depended on. Here we cover where that same mechanism lives in an ordinary company (the cron job that takes longer than its interval, the overlapping backup, the sixty desktops booting at once when the power comes back, the monitoring that pushes hardest when it hurts), why three retries across three layers mean 64 attempts against your database, the four numbers you need written down, and when NOT to touch anything.
The Windows licensing question comes up in every migration meeting, and the short answer disappoints everyone: changing hypervisor recalculates nothing. Something else changes the bill, and Microsoft has it written in a single sentence: "License Mobility across Server Farms is not available for Windows Server, so each server must be licensed for peak capacity at all times." Each server, peak capacity, at all times. The licence does not follow the virtual machine, so the number depends on how many nodes it could ever start on: 32 core licences or 96, for exactly the same installation. What changed in SQL Server 2022, why the AVMA myth does not apply to you if you are coming from vSphere, and why a Proxmox affinity rule is a useful control but not a contractual boundary.
A dead disk in a three-node cluster fixes itself provided another disk remains on that same server: the CRUSH rule descends to another OSD on the same host. A dead HOST does not. With replica 3 and the failure domain set to "host" — what Proxmox ships by default — there is no fourth place to put the third copy, so placement groups sit at active+undersized+degraded until the node comes back. And the cluster keeps serving, which is exactly why nobody looks at it. Ceph's real timers (20 seconds, 10 minutes, 15 minutes), what you cannot touch while the degraded window lasts, why recovery can cost you corosync quorum, and why the number that matters is not how many nodes you have but how many distinct places.
When a node goes down, the Proxmox VE scheduler decides where your HA-managed machines come back up. On a freshly installed cluster it does that by counting active guests: it has no idea that 101 is the second leg of 100. Proxmox VE 9.0 brought affinity rules so you can tell it, with an asymmetry worth knowing — the rule that pins a guest to a node is a preference, the rule that keeps two guests apart is an order — and an arithmetic that decides whether your VM comes back degraded or does not come back at all.
On 1 September, a scheduled capacity upgrade sequentially unplugged 100% of the fiber paths serving part of Google Cloud's us-central1-b zone, in thirteen minutes. Just before that, the report describes a correct architecture: physically separated paths, diverse power, tolerance to double and triple failures. A 2003 Berkeley study already explained why none of it helped: operator error is not more frequent than hardware failure, it is just less frequently masked.
The time a Proxmox cluster takes to re-form membership after losing a node is not fixed: it grows by 650 ms for every node you add, and on factory values a 29-node cluster reaches 45.21 seconds — exactly where the documentation itself asks you to fix it, before the 60-second watchdog starts rebooting healthy nodes. Proxmox VE 9.2 lowered it to 125 ms, but only when the cluster is created: clusters upgraded from 8 to 9 keep the old value.
A single OSD above 95% stops writes across the whole cluster even while "ceph df" still shows dozens of free terabytes. The three default thresholds (0.85 / 0.90 / 0.95), why the mechanism that repairs switches off five points before the one that serves, and the capacity calculation with one node missing that almost nobody runs: with four nodes the ceiling is 71 points, and 67 if you want the rebuild to actually finish.
On Monday 24 August, at 03:38, a denial-of-service attack began against Norway's shared government digital platform. Ten public services went down and several had nothing wrong with them: the door everyone goes through had jammed. Digdir writes on its status page that <code>eSignering</code> was unavailable "because of the limitations in ID-porten", and also that the services were "stable with the limitations that have been put in place" — part of the outage was put there by the defenders. Why single sign-on is still the right call, what changes when the door becomes infrastructure, and the three questions that reclassify it.
Proxmox VE's own documentation sets the ceiling: about 2 minutes of error detection and failover, and no more than 99.999% availability. What really happens when a node dies (a cold start, not a live migration), why a healthy node reboots itself 60 seconds after losing quorum, the requirements everybody skips, and when we do not deploy HA at all.
Deep technical analysis of Rosalía's LUX Tour ticketing system. 400,000 users, 142,000 tickets, virtual queues, Redis, WebSockets, Circuit Breakers and the architecture that made selling 8 concerts in 2 hours possible.
To receive IT stories, everyWAN news and exclusive subscriber offers, sign up to our mailing list
Microsoft 365
ES 
CAT 
EN