The control plane fails first, and your plan B needs it

A year ago, on a Monday at 08:48 local time, a DNS record was left with no addresses in it and dragged half the internet along. What barely made it into the coverage at the time, and is on the record in the official report, is what kept working: the machines already running. Not one went down. What you could not do was launch another.

What happened, on our clock

AWS manages the DNS records of its endpoints with two automated pieces: a DNS Planner, which watches load balancer health and writes a plan, and a DNS Enactor, which applies that plan from three independent Availability Zones. That night one Enactor fell unusually behind retrying its updates. Meanwhile a second Enactor applied a newer plan and triggered cleanup of the old ones. When the first finally finished, it overwrote the newer plan with its own, by then stale, and the cleanup swept that obsolete plan away: the record dynamodb.us-east-1.amazonaws.com was left with no addresses in it at all. The check that should have prevented this did exist. In AWS's words: "the check that was made at the start of the plan application process, which ensures that the plan is newer than the previously applied plan, was stale by this time due to the unusually high delays in Enactor processing".

The report gives times in Pacific time. Translated to Spanish time, so it reads the way it was lived here: the failure started on Monday 20 October 2025 at 08:48, just as offices were opening. By 09:38 AWS knew the cause lay in DynamoDB's DNS state. By 10:15 there were temporary mitigations so some of its internal services could reconnect. By 11:25 the record was restored and by 11:40 the caches had expired. Two hours and fifty-two minutes. That is the number that made the headlines, and it is the most misleading one.

The ones already running never noticed

The report says it plainly: EC2 instances already running remained healthy and unaffected. EC2's data plane held: the machines already up, their processes and their connections. DynamoDB's did not, because nobody could resolve its name, which is why a perfectly healthy application was returning errors. What blew up outright was the control plane: the part of the system you use to create, modify and destroy things.

Run through the list of what was down and read it as if it were your recovery procedure, because it looks a lot like one: launching new instances, creating or updating Lambda functions, bringing up containers on ECS, EKS and Fargate, replacing a Redshift cluster, requesting temporary credentials from STS, signing in to the console and getting into the Support Console to open a case. All of that is control plane. And all of that is, almost literally, the script somebody runs at nine in the morning when they decide it is time to move elsewhere.

The support console, incidentally, was unreachable from 08:48 to 11:40. Exactly the same window. The channel for asking what is going on was part of what was going on.

The long tail: a 2 h 52 min headline, 28 h 17 min in reality

When DNS came back, the second act began. The subsystem that assigns work to EC2's physical servers (the DropletWorkflow Manager) tried to renew leases for every physical server in the region all at once and, in the report's own words, "had entered a state of congestive collapse and was unable to make forward progress in recovering droplet leases". They fixed it by hand, throttling at 13:14 and selectively restarting hosts until 14:28. It is exactly the pattern we described in the second outage is the one you cause: the system does not collapse from the outage, it collapses from the recovery stampede.

Then came the third act, the nastiest one: newly launched instances were slow to get network connectivity, the health check subsystem brought them online prematurely, and the network load balancer cycled them in and out — removing healthy capacity. AWS ended up disabling automatic failover at 18:36. On local time, the picture looks like this:

A long day and a half between the first error and the last thing AWS declared restored. (Container impact started three minutes before minute zero, at 08:45; the rest of the column counts from 08:48.) If you signed an agreement with an annual availability percentage, that percentage told you none of this, and it is the argument we develop in an annual percentage does not say how long one outage lasts. What the business noticed was not 99-point-whatever: it was that on Tuesday at midday there were still clusters that would not answer a query because the automatic replacement had stalled halfway.

Why this is a plan B problem

Almost every recovery plan we read is written in verbs of creation. "Bring up the machines in the secondary region." "Scale the group to X nodes." "Issue a certificate." "Restore the backup onto a new instance." "Open a case with the provider." These are verbs that assume that, on the worst day of the year, the provider will let you create. And the worst day of the year is precisely the day when the thing that fails is the ability to create.

This is not an AWS quirk. It is how any large platform is built — theirs, the one next door, and the one you build yourself. Two weeks ago we wrote the same thing about another provider and another symptom — the portal would not load and management operations failed during the Azure incident of 30 September; the difference is that here the provider puts it in writing, with names and timestamps. Separating control plane from data plane is good engineering — it is why the running machines held — and at the same time it is the reason the piece your recovery depends on is the fragile one. If you read the design perimeter your provider has already written down, this is the second half of that idea: knowing what blast radius your provider withstands is not enough; you need to know which operations it lets you perform while it withstands it.

The six questions we ask of a recovery plan

This is what we mark in red when we sit down with a continuity procedure in front of us.

• Count the verbs of creation. Underline every step in your procedure that needs somebody outside to let you create something: launch, scale, issue, provision, request an IP, open a ticket. The number you get is the share of your plan you do not control.
• Look at where your operational identity lives. If deploying in Frankfurt means authenticating against a global service that physically lives somewhere else, that somewhere else is in your plan B even if it is not on the diagram. That Monday STS returned errors and latency from 08:51 to 18:59 our time, and AWS writes that customers using root credentials or identity federation against signin.aws.amazon.com hit errors signing in to the console "in regions outside of" Northern Virginia.
• Keep capacity already created, not pending creation. A machine sitting powered on and half empty at site B costs money 365 days a year and is worth exactly what it is worth on the day nobody will let you launch one.
• Throttle your retries before you need to. Growing backoff and a bit of jitter in every client that retries. Nobody fixes this at nine in the morning: either it is written into the code and the proxies beforehand, or your recovery is pushing the wrong way.
• Have a support route that is not the console. A phone number, a named contact, anything. If the only way to raise a case is the provider's web and the provider's web is inside the incident, you have no channel.
• Keep the procedure outside whatever is falling over. A recovery runbook that lives in the same service you are recovering is an expensive joke. Same for the inventory: we keep ours in NetBox as the source of truth, precisely so as not to depend on anybody's memory on a Monday morning.

Where this reading does not hold

We declare the interest before the uncomfortable part: everyWAN sells infrastructure and cloud, colocation on our own hardware and disaster recovery. We have an obvious incentive for this story to end in "come and build it with us". It does not.

Building it on your own hardware does not get you out of the problem: it brings it home. Your hypervisor has its own control plane, and if cluster quorum breaks you launch no new machines even though the ones already running carry on happily; we run Proxmox VE with Ceph in production and we know that scene. Same for the directory, the internal DNS, the authority that issues your certificates. The real difference is not cloud versus iron: it is whether the piece that lets you create sits in one pair of hands, in two, or in none you can reach. And in fairness, an outage of your own control plane gives you something a hyperscaler does not: the phone number of whoever can fix it.

The report you can read is still the one from a year ago

AWS's page of public summaries of serious incidents starts in 2011. There are eighteen entries. The most recent, as of 19 October 2026, is still the one from that Monday: none from 2026 has made it onto that page yet. We do not report this as a reproach — that document is good, detailed and far more than others publish — but as a measure of how rare a technical account of a large outage is. For almost every outage that will affect you there will never be a report. There will be a line on a status page that disappears, and whatever you wrote down while it was happening.

The usual axis, applied here: failure is inevitable, an outage is a design decision. A provider emptying a DNS record at twelve minutes to nine on a Monday is not in your hands. Whether that means your company cannot work for fourteen hours is, and it is decided months earlier, when somebody writes plan B in verbs of creation without asking who has to grant permission.

Sources and method: the whole account of the incident, the verbatim quotes and the times come from AWS's official "Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region" — aws.amazon.com/message/101925. The times in that document are Pacific time; the conversion to Spanish time (nine hours ahead, both zones on summer time on that date) and the duration arithmetic (2 h 52 min, 14 h 2 min, 14 h 21 min, 14 h 32 min and 28 h 17 min from 23:48 PDT on 19 October) are ours, and anybody can redo them from the report's own timestamps. The count of eighteen published summaries and the fact that the most recent is the one from 19 October 2025 come from AWS's post-event summaries page, consulted on 19 October 2026 — aws.amazon.com/premiumsupport/technology/pes. We had no customer affected by this incident and no telemetry of our own from that day: what we report about the case is a reading of a public document. What we say about how we work — Proxmox VE with Ceph in production, NetBox as the inventory's source of truth — is ours and carries no figures: we do not publish internal metrics nobody can check.