Kaspersky's incident response team has published the breakdown of an attack that ran for ten days and took out four ESXi hosts, every virtual machine on them, both vCenter Server Appliances and all connected backup infrastructure. The most telling part is not the scope: it is the order. Antivirus on the backup environment was disabled days in advance; then backup services were stopped, the recovery infrastructure was encrypted and administrative access was disrupted. All of that happened before anything was done to production. The attacker did not go after your data first. They went after your ability to get it back.
There was no exploit either. The report: "…no virtualization layer vulnerabilities were exploited: no guest-to-host escapes, no hypervisor memory corruption. Every action taken against ESXi was executed through legitimate VMware management interfaces." No hypervisor vulnerability. No exotic exploit. They reached the vCenter appliances with credentials obtained from "non-virtualization systems inside the organization's perimeter", perimeter systems with nothing to do with virtualisation, and from there used the management tooling exactly as designed. There is no patch to apply against that; there is an architecture decision that either got made or did not.
The order is not incidental: it is a diagnosis
The fact that someone could stop your backup services and encrypt your repositories before touching a single production host means that from where they stood, both were reachable with the same set of permissions. Your recovery infrastructure and your production were a single failure domain with two names. That is the lesson, and it is about design: no backup product fixes it for you.
Read one after another, the report's recommendations are a list of separations rather than of purchases: treat vCenter as Tier 0; separate virtualisation administrative identities from enterprise domain administration; restrict direct ESXi management paths and monitor TSM-SSH enablement changes; send logging to a central place "outside the virtualization trust domain"; and, verbatim, "…design backup as a separate security boundary with independent administrative identities, minimize domain dependencies". Not one of the five comes with an invoice attached: all five get decided on a whiteboard.
We already wrote about the older version of this same mistake when Veeam published its advisory on the backup server joined to the domain it is supposed to restore: a textbook circular dependency, the box that saves you authenticating against the thing that fell over. The loop here is identical, except the axis is not Active Directory but the virtualisation management plane. And the question that dismantles it is the usual one: which credential can delete your backup? If the answer is "one that also administers production", you already know the order in which things are going to happen.
Forty-three seconds
The attackers authenticated as root over SSH on the appliance and, in the analysis's words, "forty-three seconds later, they fetched and deployed MeshAgent through a shell script". Forty-three seconds between getting in and planting their own remote access tooling: the endpoint component of MeshCentral, legitimate management software.
Work out what fits into forty-three seconds: not an alert someone reads, triages and escalates. Which is why you protect a backup by putting it where that credential cannot reach. In this case antivirus on the backup environment had been off for days and the chain ran to completion anyway. Turning protection off before encrypting is nothing new —we saw it when Deadlock made it the first step of its playbook—. what changes here is that the target of the switch-off was the recovery environment, not production.
And if the one that loses its management plane is your provider
The first two weeks of November bring the same sequence seen from the provider's side. Overnight on 1–2 November, Beget, one of Russia's largest registrars and hosts, was hit in an attack where the attackers reached "systems used to manage the company's infrastructure" and ran scripts to erase data. VPS data, cloud storage and Kubernetes went with it. And here is the sentence anyone renting machines should read twice: all binary backups of its virtual servers were lost. File-level backups were left standing, with a caveat that got worse over the following days: the company initially described them as intact, and the reporting described them as potentially usable to recover some information, and later coverage cites an update of its own from 6 November in which those file backups would reach only about half of its VPS customers. The company itself put the outstanding work at "more than 1,000 servers must be manually inspected" and summed up its position without decoration: "We are not giving up and are continuing to look for ways to recover this data."
Six days later, on 8 November, Russia's ESTT had its cloud infrastructure disrupted by file-encrypting malware. And on 10 November, Belarus's ActiveCloud disclosed that part of its own had been encrypted: down went its E-Cloud hosting platform, the E-Cloud Protected service, the SmartMail corporate mail service and, note this, its customer support channels. No links between the three attacks have been publicly confirmed. What they share with the Kaspersky case is not who attacked, but what broke first.
The support channels are the part almost nobody accounts for. Your recovery plan, if you have hosted machines, usually ends in a sentence along the lines of "we raise a ticket with the provider and restore from their panel". The panel and the ticket are part of the failure domain. The day the provider is the victim, your recovery path is inside the incident. ActiveCloud put it this way, "these have been difficult days", and declared its main services restored by the Friday of that same week; the problem is not how they handled the crisis; it is that this dependency of yours is rarely written down anywhere until the day you need it. We covered this in October, when another provider had someone come in through the portal login and leave with the hypervisor: the provider itself ended up advising in writing that each customer prepare a separate environment and rebuild from their own copy.
The backup you think you have
This is the table we use when we review somebody else's backup strategy. The left column is what we get told; the right column is what we ask. None of the questions is a trap: a properly built backup answers all of them in ten minutes.
| What we get told | What we ask |
|---|---|
| "We back up every VM daily" | Which account does the backup server authenticate with, and does that account also administer production? |
| "The backups are on separate storage" | Can that storage be unmounted or wiped from the same console that manages the hosts? If yes, they share a fate. |
| "We follow the 3-2-1 rule" | How many of the three are administered with the same key? Three paths and one credential is a 1-1-1 with three invoices. |
| "EDR watches the backup server" | Who gets told when that protection is disabled, and how fast? In this case it had been off for days. |
| "The provider keeps the backup" | Can you start a restore without their panel and without their support desk? And if both are down at once? |
| "We have never had a problem" | When was the last timed full restore, and how long did it take? An untested backup is not a backup: it is a lucky charm. |
The 3-2-1 row is the one that starts the most arguments, so let us be explicit: the rule talks about copies, media and locations, and says nothing about credentials. Credentials are the axis an attacker who is already inside moves along, so counting places falls short for us and we count keys instead. Two copies on two continents governed by the same admin account go away with a single command.
We already wrote about counting keys back in August, off the joint FBI and CISA advisory in which one victim lost the backups at both of its data centres. What the Kaspersky case adds is the other half of the problem, and it is a matter of timing: not only how many keys exist, but the order in which they get used. When the attacker's first stop is your recovery environment, the rest of the attack is already decided.
Three days against eight hours and forty-six minutes
The arithmetic goes like this. 99.9% annual availability leaves a budget of 8 hours and 46 minutes of downtime for the whole year. ActiveCloud went public on Tuesday the 10th and declared its main services restored by the Friday of that same week: assume —and the assumption is ours, nobody else's— roughly 72 hours of disruption for an average customer. That is more than eight times the entire annual budget of a 99.9%. And if what you promised your own customers was 99.99%, 52.6 minutes a year, those same 72 hours are the budget of eighty-two years. It is not an accounting exercise, because availability does not carry over from one year to the next; it is a way of seeing the size of the hole.
The only number we can set against that is one measured at home: our last full recovery drill took 14 minutes. It is an internal figure from one specific test, not a contractual guarantee and not a number that will repeat with your inventory, because it depends on volume, on the destination and on what has to come up first. We bring it because the difference between 14 minutes and three days is set almost always by the rehearsal beforehand, with a stopwatch, and almost never by the tool.
When none of this separation is necessary
We are not going to finish by saying everyone needs a second management plane, dedicated virtualisation identities and an immutable repository outside the domain, because for plenty of companies that is over-engineering and we will say so. If your business can sit still for three days without losing customers or breaching a contract, and your data fits on a disk somebody takes home on Friday, building a two-domain architecture will cost you more in complexity than it saves you in risk.
What is non-negotiable, whatever your case costs, are two small things. One: that there exists at least one copy that cannot be deleted with the credential that administers production. It does not matter whether that is repository immutability, a destination with its own account, or a disconnected disk in a safe; what matters is that the key is a different one. And two: that this copy has been restored at least once, in full and against a clock. Because on the day of the incident you do not find out whether your backup exists; you find out whether you knew how to use it. Failure is inevitable; the outage, your company stopping, is still a design decision.
If you are reading this for your own shop, the conversation has two halves. The "where the copy lives, under which key, and how long it takes to come back" half is disaster recovery, and it is the one we put on the table before any other. The "why did a credential from some random perimeter system reach the vCenter" half is zero trust applied where it actually hurts: the management plane. If you want us to look at both with your inventory in front of us, get in touch and we will go through it, no strings.
Sources (verified): the virtualisation case —the ten days, the four ESXi hosts, both vCenter Server Appliances, the connected backup infrastructure, the antivirus on the backup environment disabled days in advance, backup services stopped and recovery infrastructure encrypted before detonation, the verbatim sentences stating that no hypervisor vulnerability was exploited and that credentials came from non-virtualisation systems, the forty-three seconds to MeshAgent and the five recommendations— comes from the Kaspersky response team (GERT) analysis published on Securelist; the ransom notes in that incident attribute the attack to the BlackNevas group. The dates for the three hosting providers (Beget overnight on 1–2 November, ESTT on the 8th and ActiveCloud disclosed on Tuesday the 10th), the quoted statements from both companies, the affected services, the loss of the support channels, the restoration of main services by the Friday and the absence of confirmed links between the attacks come from The Record on Beget and The Record on ActiveCloud. The one second-hand figure in the article, flagged as such in the body too, is Beget's 6 November update about file backups and half of its VPS customers: it appears in several trade write-ups, but we could not verify it in the company's own channel. Ours are the assumption and the arithmetic on the 72 hours of disruption and its comparison with the budgets of 99.9% (8 h 46 min/year) and 99.99% (52.6 min/year). The 14-minute drill is an internal everyWAN figure: a measured test, not a promise.
Which credential can delete your backup?
If it is the same one that administers production, you do not have two copies: you have one with two names. We will go through which key reaches what, where the chain breaks, and how long a full restore actually takes.
Talk to everyWAN